GPTBot
OpenAI's training crawler.
- Vendor
- OpenAI
- robots.txt token
GPTBot- Purpose
- Training
- Respects robots.txt
- Yes, according to OpenAI's documentation
- Source
- Vendor documentation
What it does
OpenAI says GPTBot crawls content that may be used for generative AI foundation model training.
It says "disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models".
OpenAI says you can allow OAI-SearchBot so you appear in search while disallowing GPTBot to keep content out of training. If a site allows both, OpenAI may use a single crawl for both purposes.
What blocking it changes
GPTBot collects data for model training. It is a different user agent from any AI search crawler, so a robots.txt rule for GPTBot does not apply to them. Whether your page can appear in OpenAI's search answers is controlled by OAI-SearchBot.
robots.txt
Block it
User-agent: GPTBot
Disallow: /Allow it
User-agent: GPTBot
Allow: /Block training but stay in OpenAI search
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /Other crawlers from OpenAI
- OAI-SearchBot (search)
- ChatGPT-User (user-requested fetch)
Check your own site
The free Chompr.io extension shows each AI search crawler's status for the page you are on.