OAI-SearchBot
OpenAI's search crawler.
- Vendor
- OpenAI
- robots.txt token
OAI-SearchBot- Purpose
- Search
- Respects robots.txt
- Yes, according to OpenAI's documentation
- Source
- Vendor documentation
What it does
OpenAI says OAI-SearchBot "is used to surface websites in search results in ChatGPT's search features".
It is separate from GPTBot, which OpenAI uses for training, and from ChatGPT-User, which handles user-initiated visits. OpenAI says each setting is independent.
OpenAI says that if a site allows both OAI-SearchBot and GPTBot, it may use the results of a single crawl for both purposes.
What blocking it changes
OAI-SearchBot is how OpenAI's answer engine indexes pages, so blocking it removes your pages from that engine's answers.
OpenAI says sites that opt out "will not be shown in ChatGPT search answers, though can still appear as navigational links". It adds that a robots.txt update can take about 24 hours to take effect.
In the Tow Center (Columbia) generative-search study, which made 1,600 observations, pages from publishers that blocked an engine's crawler were cited about half as often (35.5% against 17.7%). OpenAI honoured blocks far more than Perplexity did. The data covered OpenAI and Perplexity only. The effect is not guaranteed for your own site. See the crawler evidence for more.
robots.txt
Block it
User-agent: OAI-SearchBot
Disallow: /Allow it
User-agent: OAI-SearchBot
Allow: /Block training but stay in OpenAI search
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /Other crawlers from OpenAI
- ChatGPT-User (user-requested fetch)
- GPTBot (training)
Check your own site
The free Chompr.io extension shows each AI search crawler's status for the page you are on.