Google-Extended

Google's training crawler.

What it does

Google describes Google-Extended as "a standalone product token that web publishers can use to manage" whether content Google crawls may be used for training Gemini models and for grounding in Gemini Apps and Vertex AI.

Google says it "doesn't have a separate HTTP request user agent string", and that crawling is done with existing Google user agent strings. It is a control token read from robots.txt, not a separate crawler.

What blocking it changes

Google-Extended collects data for model training. It is a different user agent from any AI search crawler, so a robots.txt rule for Google-Extended does not apply to them.

Google says Google-Extended "does not impact a site's inclusion in Google Search" and is not used as a ranking signal in Google Search. Blocking it is a choice about training and grounding use only.

robots.txt

Block it

User-agent: Google-Extended
Disallow: /

Allow it

User-agent: Google-Extended
Allow: /