AI crawlers
AI crawlers visit your site for three different reasons. Search crawlers index pages for an AI answer engine. User-requested fetchers load a page because a person asked an assistant to. Training crawlers collect data for model training.
Search and training crawlers are different user agents, even when they come from the same company. Blocking GPTBot does not block OAI-SearchBot, and blocking ClaudeBot does not block Claude-SearchBot. Your robots.txt needs a separate rule for each.
In the Tow Center (Columbia) generative-search study, which made 1,600 observations, pages from publishers that blocked an engine's crawler were cited about half as often (35.5% against 17.7%). OpenAI honoured blocks far more than Perplexity did. The data covered OpenAI and Perplexity only. See the crawler evidence.
Search
An AI answer engine indexes pages with these crawlers. Blocking one removes your pages from that engine's answers.
User-requested fetch
These fetch a page because a user asked the assistant to. Vendors say robots.txt may not apply, so a block is a statement of intent, not an enforced outcome.
Training
These collect data for model training. They are different user agents from the search crawlers, so a robots.txt rule for one does not apply to the other.
Check your own site
The free Chompr.io extension shows each AI search crawler's status for the page you are on.