What Is CCBot (Common Crawl)?

Q: How do I block CCBot?

Add `User-agent: CCBot` followed by `Disallow: /` to your robots.txt. CCBot documents that it respects robots.txt directives. Blocking it prevents future pages from being added to Common Crawl's public archive, though it cannot remove content already captured in earlier releases.

Written byFlavio Amiel Founder, Roborank

Updated July 15, 2026

CCBot is the web crawler operated by Common Crawl, a nonprofit that publishes a free, openly licensed archive of the web. Identified by the user-agent token CCBot, it gathers pages into Common Crawl’s public dataset, which has been used as training data for many large language models, including OpenAI’s GPT-3.

Key Takeaways

CCBot’s full user-agent string is CCBot/2.0 (https://commoncrawl.org/faq/); it is blocked with a User-agent: CCBot / Disallow: / rule in robots.txt.
Common Crawl is a foundational training corpus: the filtered Common Crawl dataset supplied about 60% of the tokens used to train OpenAI’s GPT-3.
Because the archive is public, blocking CCBot restricts a huge, redistributable class of AI training use in a single directive rather than per-company.
Common Crawl publishes CCBot’s authentic IP ranges at index.commoncrawl.org/ccbot.json so operators can distinguish real traffic from spoofed user agents.

How CCBot Works

CCBot is the AI crawler run by Common Crawl, a nonprofit whose entire purpose is to maintain a free, openly licensed snapshot of the web that anyone can download and analyze. Where a search engine crawls to build its own private index, Common Crawl crawls to publish — releasing regular archives of raw page data that researchers, startups, and AI labs pull from without having to run their own web-scale crawl.

CCBot identifies itself with the user-agent string CCBot/2.0 (https://commoncrawl.org/faq/), and it honors robots.txt. A site that wants out adds two lines — User-agent: CCBot and Disallow: / — and future crawls skip it. Because impostors sometimes borrow the CCBot name, Common Crawl publishes the crawler’s genuine IP ranges at index.commoncrawl.org/ccbot.json, letting an operator confirm real traffic by reverse-DNS and CIDR rather than trusting the header alone.

What makes CCBot distinctive is not the crawler but where its output goes. Common Crawl data is the raw material for a large share of publicly built language models, so a single page captured by CCBot can end up replicated across many downstream training sets. That is why CCBot draws blocking attention out of proportion to its traffic: it is a one-to-many pipe.

Example of CCBot

The most consequential documented use of Common Crawl data is OpenAI’s GPT-3. In the 2020 paper Language Models are Few-Shot Learners, OpenAI details its training mixture, and a filtered version of Common Crawl is by far the largest ingredient: roughly 410 billion tokens, weighted at about 60% of the tokens seen during training. In other words, most of what GPT-3 learned language from came through the pipeline CCBot feeds.

That single fact reframes the robots.txt decision. When a publisher blocks CCBot, they are not opting out of an obscure research project — they are declining to contribute to the exact class of dataset that seeded the modern LLM era, and that continues to underpin many open and commercial models. It also explains the blocking numbers: an April 2026 analysis of the top 100 US and UK news sites found CCBot was the most-blocked AI-related crawler, disallowed by 75% of them, ahead of even GPTBot at 62%.

The limit of the tactic is just as important. Blocking CCBot today does nothing about archives Common Crawl already released, which remain public and mirrored, and it does nothing about crawlers that never touched Common Crawl in the first place. It is a precise instrument for one uniquely redistributable corpus — not a general-purpose AI off switch. For the broader picture of which directives cover which bots, see AI bot blocking; to publish machine-readable guidance about how your content may be used, see llms.txt.

The thing people get wrong

People assume blocking CCBot is a clean, complete way to keep their content out of AI. It is partial, and it is retroactive-proof only going forward. Common Crawl archives are already published, openly licensed, and mirrored in countless downstream training sets; a fresh disallow stops future capture but cannot recall what is already in the corpus. And CCBot is only one door — GPTBot, ClaudeBot, Google-Extended and others crawl on their own tokens entirely. I treat blocking CCBot as one specific decision about one specific, unusually redistributable dataset, not as a master switch for "no AI." The web does not have a master switch.

Frequently Asked Questions

What is CCBot?

CCBot is Common Crawl’s web crawler. Common Crawl is a nonprofit that archives billions of web pages and releases the data for free under an open license. That archive is a common ingredient in the training datasets behind many large language models.

How do I block CCBot?

Add User-agent: CCBot followed by Disallow: / to your robots.txt. CCBot documents that it respects robots.txt directives. Blocking it prevents future pages from being added to Common Crawl’s public archive, though it cannot remove content already captured in earlier releases.

Does blocking CCBot stop all AI training on my site?

No. It stops one dataset. Other crawlers such as GPTBot, ClaudeBot, and Google-Extended gather training data under their own separate user-agent tokens, so each must be blocked individually. Blocking CCBot only addresses Common Crawl’s specific, openly redistributed corpus.

Is CCBot the same as GPTBot?

No. CCBot belongs to the nonprofit Common Crawl and feeds a public, shared archive; GPTBot belongs to OpenAI and feeds its private model training. They are different operators, different tokens, and different datasets, even though both are frequently blocked by publishers who want out of AI training.

The Bottom Line

CCBot is the crawler behind Common Crawl’s free web archive — one of the most widely reused training corpora in AI, having supplied the majority of GPT-3’s training tokens. Controlling it is a single-directive decision about a uniquely public dataset, not a way to opt out of every AI crawler at once; each of those keeps its own token.

Sources

CCBot — Common Crawl crawler and robots.txt policy — Common Crawl
Language Models are Few-Shot Learners (GPT-3, Brown et al., 2020) — Table 2.2, training dataset mix — arXiv

Rank & Cash — the weekly SEO breakdown

One practical teardown a week on ranking in search and getting cited by AI. No fluff.