AI crawler
Claude-User
Anthropic's user-requested fetch crawler.
- Vendor
- Anthropic
- robots.txt token
Claude-User- Purpose
- User-requested fetch
- Respects robots.txt
- See vendor documentation
- Source
- Vendor documentation
What it does
Anthropic says that "when individuals ask questions to Claude, it may access websites using a Claude-User agent".
What blocking it changes
A robots.txt block for Claude-User is a statement of intent, not an enforced outcome. The vendor documentation below says these fetches may not follow robots.txt, because a user asked for the page.
Anthropic says blocking it stops your content being retrieved for user queries, which may reduce visibility in user-directed web search.
robots.txt
Block it
User-agent: Claude-User
Disallow: /Allow it
User-agent: Claude-User
Allow: /Other crawlers from Anthropic
- Claude-SearchBot (search)
- ClaudeBot (training)
Check your own site
The free Chompr.io extension shows each AI search crawler's status for the page you are on.