Google-Extended
Google-Extended is Google's robots.txt control token for AI training: publishers use it to manage whether content Google crawls from their sites may be used to train future Gemini models and for grounding — supplying content to the model at prompt time — in Gemini apps and Vertex AI. Per Google's crawler documentation, it does not affect a site's inclusion or ranking in Google Search.
Mechanically it differs from GPTBot and ClaudeBot: Google-Extended has no crawler user-agent string of its own. Crawling happens under Google's existing user agents, and the token works purely in a control capacity inside robots.txt — a directive addressed to "User-agent: Google-Extended" governs how already-crawled content may be used, rather than which requests arrive.
For API producers the isolation of concerns is the useful property. Disallowing Google-Extended keeps documentation out of Gemini training and grounding without touching Search visibility, and allowing it does the reverse of nothing for rankings — Google states it is used neither for inclusion nor as a ranking signal. The decision is therefore purely about the AI answer layer: grounded answers about an API can only draw on docs the token permits.
Google-Extended is the third AI crawler token Discry's robots.txt discovery check parses, together with GPTBot and ClaudeBot. The check reads the docs domain's robots.txt groups, treats a full Disallow with no Allow as a block, and grades pass, partial, or fail by how many of the three tokens are blocked.
Checks: robots.txt AI directives · how each is scored
What does an AI agent make of your API?
Find out in about a minute — no signup.
Discry your API — free