robots.txt for AI Crawlers: GPTBot, ClaudeBot, Google-Extended

The robots.txt file on your docs domain is where AI crawlers learn whether they are welcome, and for an API it quietly decides something bigger: whether the models behind coding agents ever learn your API exists. Across the Discry Index corpus, 80% of scanned APIs serve an AI-crawler-friendly robots.txt on their documentation domain. The rest fall into two observable patterns in our scan findings: teams that blocked AI crawlers deliberately, usually with a policy copied from the marketing site, and teams whose docs subdomain serves no robots.txt at all.

Both patterns cost the same thing. Blocking AI crawlers made sense as a content-protection default for publishers, where the crawled text is the product. API documentation works differently: the docs are how agents — and the models underneath them — learn to integrate your product. A developer asking a coding agent "which payments API should I use, and how do I call it" gets an answer assembled from what the model already learned and what the agent can fetch right now. A robots.txt that blocks AI crawlers on your docs domain removes you from the first source and, depending on which user agents you block, from the second too.

This article covers what each crawler token actually controls (verified against each vendor's own documentation), three copy-pasteable recipes, and exactly what the robots.txt AI directives check in the Discry methodology credits.

The AI crawler tokens, verified

Three vendors account for the tokens that matter most, and each splits its traffic into distinct user agents with different jobs. Blocking one is a different decision than blocking another.

OpenAI documents its crawlers at platform.openai.com/docs/bots:

• GPTBot crawls content that may be used to train OpenAI's generative AI foundation models. Disallowing GPTBot signals that your content should not be used for training.
• OAI-SearchBot surfaces websites in ChatGPT's search features. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers.
• ChatGPT-User fetches pages when a ChatGPT user asks about them. OpenAI notes that because these fetches are user-initiated, robots.txt rules may not apply to them.

Anthropic documents its crawlers in a support article:

• ClaudeBot collects web content that could contribute to training Anthropic's models. Anthropic states its bots honor robots.txt directives.
• Claude-User retrieves pages when a Claude user directs it to; disabling it prevents retrieval in response to user queries.
• Claude-SearchBot indexes content to improve search result quality.

Google documents Google-Extended on its crawler reference page. It is the odd one out: Google-Extended has no user-agent string of its own. Crawling happens under existing Google user agents, and Google-Extended works purely as a robots.txt control token governing whether crawled content may be used for training Gemini models and for grounding. Google states it does not affect a site's inclusion or ranking in Google Search — which also means blocking it costs you nothing in classic SEO and gains you nothing in agent visibility; it is strictly an AI-use control.

The pattern across all three vendors: training crawlers, search crawlers, and user-triggered fetchers are separate levers. A robots.txt that treats "AI" as one thing is making three decisions at once, usually without meaning to.

Protecting content and being discoverable are different decisions

The corpus makes the tension concrete. SubDownload's robots.txt disallows ClaudeBot, GPTBot, and Google-Extended outright — on a product built for AI agents. Pexels blocks the training crawlers with a blanket disallow but explicitly permits the user-triggered fetchers like Claude-User, a deliberate split: no training on our content, but an agent acting for a live user may read it. And Adyen's docs subdomain returns no robots.txt at all, so a crawler arriving there finds no guidance either way.

Each of these is a check-level observation, and each represents a different posture. The Pexels split is coherent for a stock-media business whose images are the product. The same split on an API's documentation domain is self-defeating, because the thing being "protected" — reference pages, auth guides, error tables — is material you publish precisely so that others build on it. Every model-training crawl you block is a future model that knows your competitor's endpoints and hallucinates yours.

The practical resolution is scoping. Set restrictive rules on the domains where content is the product. Set open rules on the domain where the docs live. If both live on one domain, use path-scoped rules — that is what the second recipe below does.

GPTBot robots.txt recipes you can paste

All three recipes are valid per RFC 9309, the Robots Exclusion Protocol standard. Rules are grouped by User-agent; a crawler follows the group that names it specifically, falling back to the * group.

Recipe 1 — open docs domain (the default we recommend for documentation). Explicitly naming the AI tokens is optional when nothing blocks them, but it documents intent and survives a later engineer adding a restrictive wildcard group:

# docs.example.com/robots.txt — documentation is public; agents welcome
User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://docs.example.com/sitemap.xml

Recipe 2 — docs-only carve-out. For a single domain hosting both product content you want protected and documentation you want discovered. The AI training crawlers are blocked everywhere except the docs paths and the agent-discovery files:

# example.com/robots.txt — protect app content, keep docs discoverable
User-agent: GPTBot
Disallow: /
Allow: /docs/
Allow: /api-reference/
Allow: /llms.txt
Allow: /llms-full.txt

User-agent: ClaudeBot
Disallow: /
Allow: /docs/
Allow: /api-reference/
Allow: /llms.txt
Allow: /llms-full.txt

User-agent: Google-Extended
Disallow: /
Allow: /docs/
Allow: /api-reference/
Allow: /llms.txt
Allow: /llms-full.txt

User-agent: *
Allow: /

Recipe 3 — per-bot split: no training, keep the live-answer surfaces. This blocks the training crawlers while leaving search and user-triggered fetchers open, the Pexels posture. It is a defensible choice for a marketing or content domain. Understand what it costs on a docs domain: the models powering tomorrow's agents will not have crawled your documentation, and the Discry robots.txt check will record all three AI tokens as blocked:

# robots.txt — opt out of model training only
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: *
Allow: /

One durability note: user-agent tokens change as vendors add crawlers. Verify names against the vendor pages linked above rather than copying lists from year-old blog posts — the corpus contains robots.txt files carefully blocking crawlers that no longer exist while missing ones that do.

What the Discry robots.txt check credits

Discry's discovery scan measures whether agents can find and understand your API from its public documentation surface — the concept is discoverability; the measurement is a set of live fetches against the domain your docs live on. The robots.txt AI directives signal works like this:

• The scanner fetches robots.txt from the scanned docs domain. No robots.txt means the check fails; the scanner treats absence as a missing signal rather than a neutral default.
• The file is parsed per RFC 9309, comments stripped, groups assembled by User-agent.
• For each of three tokens — GPTBot, ClaudeBot, and Google-Extended — the scanner finds the group that names the token specifically, falling back to the wildcard group. The token counts as blocked when that group contains Disallow: / with no Allow rule.
• None blocked: the check passes. Some blocked: partial credit. All three blocked: the check fails.

Two consequences worth knowing. First, a minimal robots.txt with just a wildcard allow passes — the check credits "agents are not turned away," and explicit welcome is optional. Second, Allow lines rescue a disallow group, which is why the docs-only carve-out in Recipe 2 retains credit: the group blocks broadly but affirmatively opens a path. Full semantics are on the methodology page.

The check also catches subtler self-sabotage than a blanket block. Ahrefs passes the AI-directives check — no AI token is blocked — but its robots.txt disallows /*.txt$, a pattern that would stop compliant crawlers from reading an llms.txt even if one were added. A rule written years ago for tidiness can silently fence off the files agents look for first.

The gate, then the map

robots.txt is the gate: it determines whether crawlers may read your docs at all. It says nothing about whether an agent that gets through can find the pages that matter — that is the job of the rest of the discovery layer, starting with an llms.txt index and the other signals covered in the agent-readiness pillar. The corpus shows the two travel together at the top: Storybook, for instance, pairs a robots.txt that explicitly allow-lists the major AI crawlers with a comprehensive llms.txt.

If you have just fixed your gate, the next steps are the map: what llms.txt is and why your API needs one, and the prioritized agent-ready checklist for the signals after that. An open robots.txt is the cheapest discovery fix in the entire stack — one static file, no build step, verifiable in a single fetch.

What grade does an AI agent give your API? Discry your API — free — 60 seconds, no signup.