What Is AI Bot Blocking?
Written byFlavio Amiel Founder, Roborank
AI bot blocking is the practice of using robots.txt directives, and sometimes server-level rules, to stop AI crawlers from fetching a site’s pages. It is applied per crawler by user-agent token, letting an operator refuse some AI bots while allowing others, depending on whether the goal is to block model training, search indexing, or both.
- AI bots are blocked in robots.txt by naming each user-agent token —
User-agent: GPTBot,User-agent: CCBot, and so on — followed byDisallow: /; there is no single directive that covers them all. - Blocking is per purpose, not per company: OpenAI’s GPTBot (training) and OAI-SearchBot (ChatGPT search) obey different tokens, so you can block one and allow the other.
- The core tradeoff is training exposure versus AI-search visibility — blocking search crawlers protects content but removes you from AI answers that increasingly sit above organic results.
- An April 2026 study of the top 100 US and UK news sites found 79% block at least one AI training bot, but only 14% block every AI bot examined.
How AI Bot Blocking Works
AI bot blocking starts in one plain text file: robots.txt, at the root of your domain. A compliant AI crawler reads that file before fetching pages and obeys any rule addressed to its user-agent token. To block one, you write its token and disallow it — for OpenAI’s training crawler, User-agent: GPTBot on one line and Disallow: / on the next. There is no directive that means “all AI bots,” so each crawler you want to stop has to be named on its own.
The reason this matters is that the tokens map to purposes, not just companies. OpenAI alone runs three: GPTBot for model training, OAI-SearchBot for ChatGPT search indexing, and ChatGPT-User for user-triggered fetches. Add CCBot for Common Crawl’s public dataset, ClaudeBot for Anthropic, Google-Extended for Gemini training, and PerplexityBot, and the picture is a dozen independent switches, each governing a different use of your content.
Robots.txt is voluntary, which is its limitation. Well-behaved crawlers honor it, but a scraper can ignore the file or spoof a trusted user-agent string. When compliance is not enough, operators escalate to server-level blocking — matching the connecting IP against a crawler’s published range list (OpenAI, Common Crawl, and Google all publish theirs) and refusing requests that do not verify. A cleaner, forward-looking option for stating intent is llms.txt, a proposed file for declaring how AI systems may use a site.
The tradeoff: training exposure vs AI-search visibility
Every blocking decision sits on a tension. Block a training crawler and you keep your words out of the next model, but you give up nothing in search. Block a search crawler and you protect content while also disappearing from the AI answers — ChatGPT search results, Google’s AI Overviews — that increasingly sit above the organic links. The two goals pull in opposite directions, and the token split is what lets you hold both positions: refuse GPTBot’s training use while still allowing OAI-SearchBot to cite you. Reaching for a blanket Disallow: / collapses that nuance and quietly costs you the AI visibility you may have wanted to keep.
Example of AI Bot Blocking
A BuzzStream analysis published in April 2026 measured exactly how the top 100 news sites in the US and UK — ranked by SimilarWeb traffic — handle this in their robots.txt, checking disallow rules across 11 AI-related crawlers. The headline: 79% block at least one AI training bot, confirming that among major publishers, refusing AI scraping is now the norm rather than the exception.
The per-bot breakdown is where the tradeoff shows up. CCBot was the most-blocked at 75%, followed by Anthropic’s legacy anthropic-ai at 72% and ClaudeBot at 69%; GPTBot sat at 62% and Google-Extended was the least-blocked training bot at 46% — plausibly because publishers are wary of any collateral effect on Google. Crucially, the choices were not all-or-nothing: 71% block at least one live search or retrieval bot, yet only 14% block every AI bot examined, and 18% block none at all.
The distribution tells the real story. If AI bot blocking were a single on/off decision, you would expect sites clustered at “block all” or “block none.” Instead most land in the middle — blocking the training crawlers they object to while keeping enough search crawlers open to stay citable. That is the practice working as designed: not a wall, but a set of individual doors, each opened or closed on its own reasoning.
The mistake I watch teams make is reaching for User-agent: * / Disallow: / because they read one alarming headline about AI scraping. That wildcard does not say "no AI" — it says "no anyone," and in practice it quietly evicts the search crawlers that decide whether ChatGPT and Google’s AI answers can cite you at all. Blocking is not one decision; it is a decision per bot. The sophisticated move is to separate the two questions — do I mind my content training a model, and do I mind being invisible in AI search? — and answer them independently, token by token. Most people only meant to answer the first one.
Frequently Asked Questions
How do I block AI bots in robots.txt?
User-agent: GPTBot then Disallow: /, repeated for CCBot, ClaudeBot, Google-Extended, and others. There is no wildcard that targets only AI bots, so compliant crawlers must be named individually to be blocked.Does blocking AI crawlers hurt my SEO?
Do AI bots actually obey robots.txt?
Should I block AI bots or not?
The Bottom Line
AI bot blocking is a per-crawler control, not a switch: you name each user-agent token in robots.txt and decide, bot by bot, whether to refuse it. The real question underneath is a tradeoff — keeping content out of model training on one side, staying visible in AI-generated answers on the other — and the token split is what lets you take both positions at once.
Sources
Rank & Cash — the weekly SEO breakdown
One practical teardown a week on ranking in search and getting cited by AI. No fluff.