What Is ClaudeBot?

How ClaudeBot Works

ClaudeBot is Anthropic’s contribution to the growing population of AI crawlers — automated agents that download public web pages so a company can build training data for its models. Functionally it behaves like any well-mannered search crawler: it requests pages over HTTP, identifies itself in the User-Agent header, and follows the Robots Exclusion Protocol. The difference is intent. Where Googlebot crawls to build a search index, ClaudeBot crawls to help teach Claude, Anthropic’s family of language models.

The token it presents is simply ClaudeBot, and its full user-agent string embeds a contact address: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com). That self-identification is what lets a site owner recognize it in server logs and decide, per token, whether to allow it. Anthropic publishes the IP ranges its bots crawl from so operators can verify that a request claiming to be ClaudeBot genuinely originates from Anthropic rather than an impostor spoofing the string.

Crucially, ClaudeBot is not the only bot Anthropic runs, and treating “ClaudeBot” as a synonym for “Anthropic” is where most blocking mistakes begin.

Anthropic’s Bots

Anthropic’s own documentation splits its web access into distinct user agents, each with a narrow job and its own robots.txt token:

• ClaudeBot — the broad crawler that collects public content for model training and development.
• Claude-User — the on-demand fetcher. When a Claude user asks something that needs live web content, this token retrieves the specific pages required to answer.
• Claude-SearchBot — the indexer that gathers content to improve the quality of search results Claude surfaces.

Because each is a separate token, a rule that disallows ClaudeBot leaves the other two untouched. Site owners who want a complete opt-out have to write a block for each. This is the same per-bot logic that governs GPTBot versus OAI-SearchBot on the OpenAI side, and it is why a serious AI bot blocking policy is a list of tokens, not a single line. Some publishers also declare their crawl preferences in an llms.txt file as a complementary signal.

Example of ClaudeBot

The clearest documented illustration of ClaudeBot’s real-world impact is the iFixit incident of July 2024. As reported by 404 Media, iFixit CEO Kyle Wiens said ClaudeBot had hit the repair site’s servers roughly one million times in a 24-hour window, consuming devops resources and pulling content at a pace Wiens publicly objected to on social media.

The resolution is the instructive part. iFixit had a broad crawl-delay and disallow policy but had not named Anthropic’s specific token. Once the site added an explicit User-agent: ClaudeBot disallow rule, Anthropic confirmed the crawler respected that signal and stopped. Nothing about the bot changed — it honored robots.txt the whole time — but it only honored the directive addressed to its token. The episode became a reference point for two lessons at once: that AI crawlers can generate real infrastructure load, and that opting out is token-specific. A generic anti-scraping rule that never mentions ClaudeBot will not stop ClaudeBot.

Frequently Asked Questions

The Bottom Line

ClaudeBot is Anthropic’s training-data crawler — the automated agent that reads public pages to teach Claude. It is only one member of Anthropic’s bot family, it obeys robots.txt when you name its token, and controlling it is a per-token exercise: block ClaudeBot to opt out of training, but address Claude-User and Claude-SearchBot separately if you want to govern real-time and search access too.