What Is Google-Extended?

How Google-Extended Works

Google-Extended is the odd one out among the tokens people group with AI crawlers. GPTBot, ClaudeBot, and PerplexityBot are actual bots — programs that request your pages over HTTP and identify themselves in the User-Agent header. Google-Extended is not. Google’s documentation is explicit: it “doesn’t have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings.” You will never see “Google-Extended” in your server logs, because nothing by that name ever knocks on your door.

Instead, Google-Extended is a control token — a name you can address in robots.txt to express a permission. When Google reads User-agent: Google-Extended followed by Disallow: /, it does not stop crawling your site; Googlebot keeps indexing you for Search. What changes is how the content Google already has may be used: a disallow rule tells Google not to feed your pages into training and grounding for its Gemini models. The crawling and the usage are separated, which is the whole point of the design.

That separation is why blocking Google-Extended carries no SEO penalty, and why it is best understood as a licensing decision rather than a technical one.

What Google-Extended Governs

Google-Extended has a deliberately narrow scope. Per Google’s documentation, it controls whether content may be used to train and ground the generative models behind two products:

• Gemini Apps — the consumer-facing Gemini assistant and its future model generations.
• Vertex AI API for Gemini — Google’s enterprise generative-AI platform.

And it has an equally explicit exclusion. Google states that Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.” Notably, it is a separate lever from the grounding that powers AI Overviews inside Search itself — Google-Extended targets Gemini training and grounding, not your appearance in Search results. Because it is set the same way as every other token, it fits naturally into a broader AI bot blocking policy or an llms.txt declaration, sitting alongside per-bot rules for GPTBot, ClaudeBot, and PerplexityBot.

Example of Google-Extended

Google-Extended’s origin is itself the clearest worked example of what it is for. On September 28, 2023, Google announced it in a post titled “An update on web publisher controls.” The framing in that announcement is the definitive statement of intent: Google-Extended was introduced as “a new control” letting web publishers “manage whether their sites help improve” Google’s generative APIs, “including future generations of models that power” those products — separate from and without affecting their presence in Google Search.

That launch matters because it set the template every other provider followed. Before Google-Extended, a publisher’s only blunt instrument was blocking Googlebot entirely — which would have destroyed their Search visibility to avoid AI training. Google-Extended decoupled the two: keep the Search crawl, opt out of the AI use. Google explicitly called making “simple and scalable controls” available through robots.txt “an important step” it believed “all providers of AI models should make available.” The design has since become the norm, but the September 2023 announcement is the primary record of the trade-off it was built to eliminate — visibility without conscription into model training.

Frequently Asked Questions

The Bottom Line

Google-Extended is a usage permission expressed in robots.txt, not a bot you can catch in your logs. It answers one question — may Google use your content to train and ground Gemini? — and deliberately leaves Search untouched. Disallowing it is a content-licensing choice with zero ranking cost, which is what makes it the least risky and most misunderstood AI control a publisher can set.