ClaudeBot
ClaudeBot is Anthropic's web-crawler user agent, used to collect public web content that may contribute to training its Claude models. Site owners control it through robots.txt; restricting ClaudeBot signals that a site's future material should be excluded from Anthropic's training datasets. Anthropic operates separate agents for other jobs — Claude-User for user-initiated fetches and Claude-SearchBot for search — each controllable independently.
Anthropic publishes the principles its crawling follows: honoring industry-standard robots.txt directives, supporting the Crawl-delay extension to limit request rates, respecting anti-circumvention measures like CAPTCHAs, and disclosing the IP ranges its bots operate from so operators can verify traffic. Blocking by robots.txt is the supported opt-out; IP blocking alone can interfere with the crawler's ability to read the robots.txt it is meant to obey.
The three-agent split matters for API producers. Blocking ClaudeBot is a training-data decision. Blocking Claude-User is an availability decision: per Anthropic's documentation, it prevents Claude from retrieving the site's content when a user asks for it — which, for API documentation, means a developer asking Claude about the API gets no access to the docs. Templates that disallow every Anthropic agent make both decisions at once, usually without meaning to.
ClaudeBot is one of the three AI crawler tokens Discry's robots.txt discovery check parses for, alongside GPTBot and Google-Extended. A robots.txt that fully blocks all three fails the check; blocking some earns partial credit; blocking none passes. The parsed result publishes on the API's profile.
Checks: robots.txt AI directives · how each is scored
What does an AI agent make of your API?
Find out in about a minute — no signup.
Discry your API — free