---
title: Should you block AI crawlers?
description: "**It depends on your goals.** - You may want to block them if you have proprietary content, don’t want AI training on your work, or are worried about AI-generated competition. - You may want to allow them if you want…"
url: "https://ssh-assistant.com/docs/should-you-block-ai-crawlers/"
updated: "2026-01-29"
category: Site Load Tester / Crawler Control
---

# Should you block AI crawlers?

**It depends on your goals.**

- You may want to block them if you have proprietary content, don’t want AI training on your work, or are worried about AI-generated competition.
- You may want to allow them if you want AI assistants to reference your site, link back to your content, or improve visibility in AI-powered search tools.

**AI training crawlers vs AI assistant crawlers**

- AI training crawlers download content to train models. Your content becomes part of the model’s knowledge.
- AI assistant crawlers fetch content on demand when users ask questions. These usually show links and citations back to your site.

**Can you block some crawlers and allow others?**

Yes. robots.txt works per user-agent, so you can block AI training bots while still allowing Google and Bing.