meta-externalagent
Meta's training crawler.
- Vendor
- Meta
- robots.txt token
meta-externalagent- Purpose
- Training
- Respects robots.txt
- See vendor documentation
- Source
- Vendor documentation
What it does
Meta says meta-externalagent "crawls the web for use cases such as training foundation AI models or improving products by indexing content directly".
Meta's documentation tells site owners to add a disallow for it in robots.txt to block it. It says robots.txt changes can take up to 24 hours to take effect.
What blocking it changes
meta-externalagent collects data for model training. It is a different user agent from any AI search crawler, so a robots.txt rule for meta-externalagent does not apply to them.
Meta says the crawler is also used to improve products by indexing content directly, so blocking it is a choice about those uses as well as training. Chompr does not score Meta as an AI search engine.
robots.txt
Block it
User-agent: meta-externalagent
Disallow: /Allow it
User-agent: meta-externalagent
Allow: /Other crawlers from Meta
- meta-externalfetcher (user-requested fetch)
Check your own site
The free Chompr.io extension shows each AI search crawler's status for the page you are on.