AI crawler
An AI crawler is an automated user agent operated by an AI company to fetch web content for model training, for search grounding, or on demand when a user or agent requests a page. The prominent published controls are OpenAI's GPTBot, Anthropic's ClaudeBot, and Google's Google-Extended robots.txt token, each governed through standard robots.txt directives.
The label covers three distinct jobs that are easy to conflate. Training crawlers (GPTBot, ClaudeBot) collect content that may enter future models. Search and grounding agents (OAI-SearchBot, Claude-SearchBot) index content so AI answers can cite it. User-triggered fetchers (Claude-User) retrieve a page because someone asked about it right now. Vendors publish separate user agents for these purposes precisely so sites can allow one and refuse another.
API documentation sits awkwardly in the robots.txt conventions that grew out of content-scraping disputes. A news site blocking training crawlers is protecting its product; an API docs site copying the same template is cutting its documentation off from the models and answer engines developers actually consult. Treatment of AI crawlers in robots.txt has become part of an API's discovery posture — a mechanical, checkable signal of whether the docs are open to the agent channel.
Discry's robots.txt discovery check reads the docs domain's robots.txt and parses its rules for GPTBot, ClaudeBot, and Google-Extended, following standard group semantics including wildcard groups. No blocked crawlers passes; some blocked earns partial credit; all blocked, or a missing robots.txt, fails.
Checks: robots.txt AI directives · how each is scored
What does an AI agent make of your API?
Find out in about a minute — no signup.
Discry your API — free