What Is GPTBot?

How GPTBot Works

GPTBot is the automated crawler OpenAI uses to gather publicly available text from the open web so that content can be considered for training future GPT models. OpenAI introduced it in August 2023, alongside documentation telling site owners exactly how to control it. Mechanically it behaves like any well-behaved search crawler: it requests pages over HTTP, identifies itself with a distinct user-agent, and reads your robots.txt before fetching.

The user-agent is the key to recognizing it. GPTBot’s requests carry a token containing the string GPTBot and a pointer back to https://openai.com/gptbot — for instance, Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot. Because user-agents can be forged, OpenAI also publishes the crawler’s IP ranges at openai.com/gptbot.json, so a server can confirm that a request claiming to be GPTBot actually originates from OpenAI’s infrastructure.

Two things GPTBot does not do are worth stating plainly, because most of the confusion around it lives here. GPTBot does not fetch pages to answer a live user question, and it does not build the index that ChatGPT searches. Its only job is collecting content that may be used to train models. That single-purpose scope is why a robots.txt rule aimed at GPTBot changes what future models learn from you, but changes nothing about how the current ChatGPT retrieves and cites you. If you want to shape what AI systems can retrieve rather than what they train on, you are looking at a different layer — the same layer addressed by an llms.txt file and by keeping the retrieval-facing agents unblocked.

GPTBot vs OAI-SearchBot vs ChatGPT-User

OpenAI operates three separate agents, each with its own user-agent, its own purpose, and its own robots.txt controls. Blocking one is not blocking all of them, and this is the single most important thing to understand about GPTBot.

The practical consequence: if your goal is to stop your content from training models while remaining visible in ChatGPT’s answers, you block GPTBot and allow OAI-SearchBot. If you block everything with the word “bot” in it, you also remove yourself from the search index that feeds live answers — the opposite of what most publishers actually want. And ChatGPT-User is different again: it fires only when a human pastes your URL or explicitly asks ChatGPT to open a page, so OpenAI treats it as a user action rather than crawling that determines search eligibility.

How to Allow or Block GPTBot

GPTBot obeys the Robots Exclusion Protocol, so control lives in the robots.txt file at your site root. To block GPTBot entirely, add:

User-agent: GPTBot
Disallow: /

To keep GPTBot out of just part of the site — say a members area and a checkout flow — disallow only those paths:

User-agent: GPTBot
Disallow: /members/
Disallow: /checkout/

To explicitly allow it, either omit any GPTBot rule (the default is allow) or state it directly:

User-agent: GPTBot
Allow: /

The critical detail is that each agent must be named separately. A block on GPTBot says nothing about OAI-SearchBot or ChatGPT-User. If you want to opt out of training but stay in ChatGPT’s search results, your file needs two distinct stanzas — one disallowing GPTBot, one allowing OAI-SearchBot. Give changes time to register: OpenAI notes it can take roughly a day for a robots.txt edit to be reflected on its side.

Example of GPTBot

Picture a mid-size travel publisher that reads a headline about AI scraping and decides to “block the AI bots.” An engineer adds a broad rule that disallows every user-agent containing “GPT,” reasoning it will catch OpenAI’s crawlers in one stroke. Training traffic from GPTBot does drop to zero, exactly as intended.

Three months later, the marketing team notices the site has disappeared from ChatGPT’s cited sources for the destination guides it used to dominate. A competitor is now the source ChatGPT quotes. The cause is not a ranking change or a content problem — it is the robots.txt rule. The overly broad block also matched OAI-SearchBot, the agent responsible for the ChatGPT search index, so the publisher had quietly removed itself from the very system that surfaces live citations.

The fix is to separate the two intentions. The publisher rewrites robots.txt with an explicit GPTBot block (opting out of training) and an explicit OAI-SearchBot allow (opting back into search visibility), then verifies with server logs — cross-checking incoming requests against OpenAI’s published IP ranges — that OAI-SearchBot is once again crawling the guides. Within weeks the destination pages return to ChatGPT’s cited sources, while the training opt-out stays firmly in place.

The takeaway is that GPTBot is a precise instrument, not a blanket AI switch. It answers exactly one question — may OpenAI train on this content? — and every other question about how AI systems find, read, and cite your pages is governed by a different agent with a different name. Manage them individually, verify with IP ranges, and decide the training question on its own merits rather than bundling it with visibility you probably want to keep.

Frequently Asked Questions

The Bottom Line

GPTBot is the training crawler in OpenAI’s three-agent lineup, and treating it as a single on/off switch for "AI" is how sites accidentally erase themselves from ChatGPT. Decide the training question and the visibility question separately: block GPTBot if you do not want your content in future models, but keep OAI-SearchBot open if you want to be cited in AI answers. Manage each agent by its own name in robots.txt, and verify traffic against OpenAI’s published IP ranges.