ClaudeBot
Anthropic's training crawler.
- Vendor
- Anthropic
- robots.txt token
ClaudeBot- Purpose
- Training
- Respects robots.txt
- Yes, Anthropic says its bots honour robots.txt
- Source
- Vendor documentation
What it does
Anthropic describes ClaudeBot as "collecting web content that could potentially contribute to their training".
Anthropic says blocking it excludes your future site materials from AI model training datasets, and that its bots honour robots.txt.
Anthropic says ClaudeBot also supports the non-standard Crawl-delay directive. Anthropic also says blocking its IP addresses is not a reliable opt-out, because it can stop the bots reading robots.txt.
What blocking it changes
ClaudeBot collects data for model training. It is a different user agent from any AI search crawler, so a robots.txt rule for ClaudeBot does not apply to them. Whether your page can appear in Anthropic's search answers is controlled by Claude-SearchBot.
robots.txt
Block it
User-agent: ClaudeBot
Disallow: /Allow it
User-agent: ClaudeBot
Allow: /Block training but stay in Anthropic search
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /Other crawlers from Anthropic
- Claude-SearchBot (search)
- Claude-User (user-requested fetch)
Check your own site
The free Chompr.io extension shows each AI search crawler's status for the page you are on.