What Is OAI-SearchBot?

How OAI-SearchBot Works

When someone uses the search feature inside ChatGPT, the answer is not written from the model’s memory alone. ChatGPT retrieves live web pages, reads them, and cites a handful as sources. OAI-SearchBot is the AI crawler that builds and refreshes the index those retrievals draw from. If OAI-SearchBot has never fetched your page, that page cannot appear as a cited source in ChatGPT search, no matter how well it ranks in Google.

OpenAI runs three separate crawlers, and the distinction is the whole point. OAI-SearchBot handles search indexing. GPTBot collects content to train foundation models. ChatGPT-User makes one-off fetches when a user or a Custom GPT triggers an action, and OpenAI states it is “not used for crawling the web in an automatic fashion.” Each announces itself with a different user-agent token, so a site can permit or refuse each job independently in its robots.txt.

OAI-SearchBot identifies itself with a user-agent string containing the token OAI-SearchBot/1.4 and the reference URL https://openai.com/searchbot. To let servers confirm a request is genuine rather than a spoofed impostor, OpenAI publishes the crawler’s IP ranges as a machine-readable file at openai.com/searchbot.json — a list of CIDR prefixes a server can match the connecting address against. This is the same verification pattern Google uses for Googlebot.

The three OpenAI crawlers

Example of OAI-SearchBot

OpenAI’s own bot documentation is the cleanest worked example of why OAI-SearchBot deserves its own line in robots.txt. The docs assign each crawler a distinct token, a distinct reference URL, and a distinct IP-range file: OAI-SearchBot points to openai.com/searchbot.json, GPTBot to openai.com/gptbot.json, and ChatGPT-User to openai.com/chatgpt-user.json. Fetching the searchbot file returns a small JSON object with a creationTime field and a prefixes array of ipv4Prefix CIDR blocks — a concrete, verifiable list a server can check requests against.

That separation has a direct consequence. A publisher who added User-agent: GPTBot / Disallow: / to keep their archive out of model training in 2023 did nothing to their ChatGPT search visibility, because OAI-SearchBot obeys a different token and was never mentioned. The reverse trap is more common today: a blanket User-agent: * disallow, written to stop training, silently also strips the site out of ChatGPT search results, since the wildcard matches OAI-SearchBot as well.

The behavior generalizes into a simple rule. Because OpenAI documents these as three independent tokens, your robots.txt is a three-way switch, not an on/off toggle. You can be invisible to training and fully visible to search at the same time — but only if you write the tokens out by name instead of reaching for the wildcard.

OAI-SearchBot vs GPTBot

The two are frequently confused because both are “OpenAI bots,” but they govern opposite outcomes. Blocking GPTBot is a stance on model training; blocking OAI-SearchBot is a stance on being found in ChatGPT. Decide each on its own terms — see AI bot blocking for how the directives interact.

Frequently Asked Questions

The Bottom Line

OAI-SearchBot is the ChatGPT search crawler: the one that determines whether your pages can be pulled up and cited when someone searches inside ChatGPT. It is deliberately distinct from the GPTBot training crawler, which means your robots.txt can welcome the crawler that earns you AI- search visibility while still turning away the one that feeds model training.