AI crawler
meta-externalfetcher
Meta's user-requested fetch crawler.
- Vendor
- Meta
- robots.txt token
meta-externalfetcher- Purpose
- User-requested fetch
- Respects robots.txt
- Not necessarily. Meta says it may bypass robots.txt rules
- Source
- Vendor documentation
What it does
Meta says meta-externalfetcher fetches individual links when a user requests them, and supports product functions such as testing and improving agentic AI.
Meta says it "may bypass robots.txt rules" because the fetches are requested by the user.
What blocking it changes
A robots.txt block for meta-externalfetcher is a statement of intent, not an enforced outcome. The vendor documentation below says these fetches may not follow robots.txt, because a user asked for the page.
robots.txt
Block it
User-agent: meta-externalfetcher
Disallow: /Allow it
User-agent: meta-externalfetcher
Allow: /Other crawlers from Meta
- meta-externalagent (training)
Check your own site
The free Chompr.io extension shows each AI search crawler's status for the page you are on.