AI crawler
Claude-SearchBot
Anthropic's search crawler.
- Vendor
- Anthropic
- robots.txt token
Claude-SearchBot- Purpose
- Search
- Respects robots.txt
- Yes, Anthropic says its bots honour robots.txt
- Source
- Vendor documentation
What it does
Anthropic says Claude-SearchBot "navigates the web to improve search result quality for users".
It is separate from ClaudeBot, which collects content for training, and from Claude-User, which acts on a user's request.
What blocking it changes
Claude-SearchBot is how Anthropic's answer engine indexes pages, so blocking it removes your pages from that engine's answers.
Anthropic says that blocking it means your content is not indexed for search, which may reduce visibility and accuracy in user search results.
The effect of blocking Claude-SearchBot on citations has not been measured. The Tow Center study covered OpenAI and Perplexity only.
robots.txt
Block it
User-agent: Claude-SearchBot
Disallow: /Allow it
User-agent: Claude-SearchBot
Allow: /Block training but stay in Anthropic search
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /Other crawlers from Anthropic
- Claude-User (user-requested fetch)
- ClaudeBot (training)
Check your own site
The free Chompr.io extension shows each AI search crawler's status for the page you are on.