AI crawler
CCBot
Common Crawl's training crawler.
- Vendor
- Common Crawl
- robots.txt token
CCBot- Purpose
- Training
- Respects robots.txt
- See vendor documentation
- Source
- Vendor documentation
What it does
CCBot is the crawler for Common Crawl, a nonprofit that produces and maintains a free, open repository of web crawl data meant to be accessible and analysable by anyone.
Common Crawl's page shows robots.txt as the way to stop CCBot crawling, and links to an opt-out registry for site owners.
What blocking it changes
CCBot collects data for model training. It is a different user agent from any AI search crawler, so a robots.txt rule for CCBot does not apply to them.
Common Crawl data is an open dataset rather than the index of one AI product, so blocking CCBot is a statement about that dataset.
robots.txt
Block it
User-agent: CCBot
Disallow: /Allow it
User-agent: CCBot
Allow: /Check your own site
The free Chompr.io extension shows each AI search crawler's status for the page you are on.