Google-Extended & robots.txt
Control Gemini AI Training
Block or allow Google's AI training crawler — without affecting your Google Search ranking.
What is Google-Extended?
Google-Extended is a robots.txt token (not a crawler) introduced by Google in September 2023. It controls whether your content is used for Gemini training and grounding — the Gemini app and Vertex AI. It is separate from Googlebot — blocking Google-Extended does not affect your search ranking, and it does not affect Google's AI Overviews or AI Mode either: those are built from the normal Search index that Googlebot crawls.
This separation is deliberate. Google created Google-Extended after publishers demanded a way to opt out of AI training without losing search visibility. You reference it in robots.txt as Google-Extended.
Google-Extended vs Googlebot
Googlebot
Search indexing. Affects your ranking. Never block this unless you want to disappear from Google.
Google-Extended
Gemini training/grounding only. No ranking impact, no AI Overviews impact. Safe to block if you want to opt out of Gemini training.
Want to control AI Overviews specifically? That's not Google-Extended's job — use nosnippet, max-snippet, or data-nosnippet instead.
robots.txt Syntax for Google-Extended
Copy-paste these examples into your robots.txt file.
Allow Google-Extended (AI training opt-in)
# Allow Google AI training
User-agent: Google-Extended
Allow: /
Allows Google to use your content for Gemini training and grounding. Your content may inform Gemini's answers.
Block Google-Extended (AI training opt-out)
# Block Google AI training
User-agent: Google-Extended
Disallow: /
# Keep Googlebot allowed for search
User-agent: Googlebot
Allow: /
Blocks AI training while keeping your site in Google Search. This is what many major publishers use.
Block All AI Crawlers (Google + Others)
# Block AI training from all major providers
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
# Keep search engines allowed
User-agent: Googlebot
Allow: /
See our full list of 17 AI crawlers to block them all.
When to Allow vs Block Google-Extended
Allow Google-Extended When...
- ✓You want Gemini to know about your brand/product
- ✓You want your content grounding Gemini app and Vertex AI answers
- ✓You publish public knowledge or documentation
- ✓You want maximum AI visibility across platforms
Block Google-Extended When...
- ✗You publish premium or paywalled content
- ✗You don't want Google training AI on your work
- ✗You have licensing or copyright concerns
- ✗You want search traffic but not your content in Gemini training (for AI Overviews control, use snippet directives)
Frequently Asked Questions
Does blocking Google-Extended affect my Google Search ranking?
No. Google-Extended only controls whether your content is used for training Gemini and other Google AI products. Your Google Search ranking is determined by Googlebot, which is a separate crawler. Blocking Google-Extended has zero impact on your position in Google Search results.
What's the difference between Googlebot and Google-Extended?
Googlebot crawls your site for Google Search indexing — it determines whether your pages appear in search results. Google-Extended crawls content specifically for training Google's AI models like Gemini. They are independent: you can block Google-Extended while keeping Googlebot fully allowed.
What does Google-Extended actually do with my content?
Google-Extended is a robots.txt token (not a separate crawler) that controls whether your content is used to train and ground Google's Gemini models (the Gemini app and Vertex AI). Allow it and your content can inform Gemini's answers; block it and your content is excluded from Gemini training and grounding. It does not control AI Overviews.
Does blocking Google-Extended remove me from Google AI Overviews?
No. AI Overviews and AI Mode are built on Google's normal Search index, crawled by Googlebot — Google-Extended has no effect on them. If you want granular control over how AI Overviews use your pages, use the nosnippet, max-snippet, or data-nosnippet directives instead. Blocking Google-Extended only opts you out of Gemini app/Vertex training and grounding.
Should I block or allow Google-Extended?
It's purely a training/grounding policy choice. Allow it if you want your content to inform Gemini's answers; block it to keep your content out of Gemini training and grounding. Either way, your Google Search ranking and your AI Overviews eligibility are unaffected. Many publishers block Google-Extended while keeping Googlebot allowed — it's a common and supported configuration.
When did Google introduce Google-Extended?
Google launched Google-Extended in September 2023 specifically to give publishers control over AI training data. Before this, there was no way to opt out of Google's AI training while keeping your site in search results. It was a direct response to publisher concerns about AI using their content.
Related robots.txt Guides
- GPTBot & robots.txt — OpenAI / ChatGPT
- ClaudeBot & robots.txt — Anthropic / Claude
- PerplexityBot & robots.txt — Perplexity AI
- CCBot & robots.txt — Common Crawl
- Check all 17 AI crawlers at once
Check Your Google-Extended Configuration
See if Google-Extended and other AI crawlers can access your site. Full report on all 17 AI bots.
No credit card required