AI robots.txt Generator
Check which AI crawlers your live robots.txt allows today, then generate corrected rules for all 18 of them — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and the rest.
Step 1: Check the robots.txt you have now
Enter your domain and we'll read your live /robots.txt and show which of the 18 AI crawlers it currently allows or blocks. Skip this if you're starting from scratch.
Step 2: Choose who gets access
Toggle any crawler between Allow and Block. The file updates as you go.
Search-Index Crawlers
The visibility gates — blocking one removes you from that engine's AI search answers
Training Crawlers
Collect data for AI model training — blocking is a policy choice and doesn't affect AI search visibility
User-Fetch Agents
Fetch a page only when a user asks the assistant to open it — several ignore robots.txt, so rules here are advisory
Generated robots.txt Rules
# AI Crawler Rules # Generated by BotView (https://botview.app/ai-robots-txt-generator) # 2026-09-01 # Allowed AI crawlers User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Google-Extended Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: Amazonbot Allow: / User-agent: CCBot Allow: / User-agent: Diffbot Allow: / User-agent: Applebot-Extended Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-AdsBot Allow: / User-agent: Claude-User Allow: / User-agent: Perplexity-User Allow: / User-agent: Meta-ExternalFetcher Allow: / User-agent: MistralAI-User Allow: / # Blocked AI crawlers User-agent: Bytespider Disallow: /
Add these rules to your existing robots.txt file. They won't affect Googlebot, Bingbot, or any other search engine crawlers.
Cleaning up an old robots.txt? Claude-Web, anthropic-ai, FacebookBot, and cohere-ai are dead tokens no current crawler uses — rules for them do nothing and can be removed.
How robots.txt works for AI crawlers
The robots.txt file sits at the root of your website (e.g. yoursite.com/robots.txt). Well-behaved crawlers fetch it before requesting any page, and obey the group of rules that matches their user-agent string. Each AI company has registered its own tokens, so a robots.txt written in 2022 for Googlebot and Bingbot says nothing at all about ChatGPT or Claude.
Matching is by longest prefix, and only one group wins. If a crawler finds a group naming it specifically, that group is the only one that applies — the User-agent: * group is ignored for that crawler entirely, including any Allow lines it contains. This is the single most misunderstood part of the format, and it is why adding one narrow block can silently change behaviour you thought was covered by your wildcard rules.
The corollary is the failure we see most often: a wildcard User-agent: * paired with Disallow: / blocks every AI crawler that has no group of its own. Sites carry this from a staging config into production and lose AI search visibility without ever making a decision about it.
Finally, robots.txt is a request rather than an access control. Allowing a crawler in robots.txt does not guarantee it can reach your pages — a CDN or WAF bot rule can still return 403 to a bot you explicitly welcomed.
Not every AI crawler costs you visibility
Most guides present AI crawlers as one undifferentiated list to allow or block. They do three different jobs, and only one of them decides whether you can appear in an AI answer.
Search-index crawlers
These feed live AI answer engines. Block one and your pages stop being eligible for citation in that engine — this is the only group where a Disallow costs you visibility.
OAI-SearchBot, Claude-SearchBot, PerplexityBot
Training crawlers
These collect text for model training. Blocking them is a content-policy decision. It does not remove you from ChatGPT, Claude or Perplexity answers.
GPTBot, ClaudeBot, Google-Extended, Meta-ExternalAgent, Amazonbot, CCBot, Bytespider, Diffbot, Applebot-Extended
User-fetch agents
These fire only when a person asks an assistant to open your URL. Several vendors say robots.txt may not apply to them, so rules here are advisory.
ChatGPT-User, OAI-AdsBot, Claude-User, Perplexity-User, Meta-ExternalFetcher, MistralAI-User
Copy-paste robots.txt rules for AI crawlers
Four configurations that cover most sites. Use the generator above if you want something in between.
Maximum AI visibility
You want to be cited in ChatGPT, Claude and Perplexity answers, and you don't mind training use.
User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: /
Cited, but not trained on
The most common configuration for publishers: stay visible in AI search, opt out of model training.
# Keep the search gates open User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / # Opt out of training User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: CCBot Disallow: /
Block AI crawlers entirely
You want no AI crawler access at all. Accept that you will not appear in AI search answers.
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-SearchBot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: /
Block one crawler only
Everything stays open except a single bot — usually Bytespider, which crawls aggressively.
User-agent: Bytespider Disallow: /
Every AI crawler user-agent token
The 18 tokens BotView tracks, with what each one actually controls. These are the exact strings to use after User-agent: — they are case-insensitive but must otherwise match.
| User-agent token | Vendor | What blocking it does |
|---|---|---|
| OAI-SearchBot | OpenAI | ChatGPT search index — the ChatGPT visibility gate |
| Claude-SearchBot | Anthropic | Claude search index — the Claude visibility gate |
| PerplexityBot | Perplexity | Perplexity search index — the Perplexity visibility gate |
| GPTBot | OpenAI | OpenAI model training (does not affect ChatGPT search) |
| ClaudeBot | Anthropic | Anthropic model training (does not affect Claude search) |
| Google-Extended | Gemini/Vertex training + grounding opt-out token (not a crawler; does NOT affect AI Overviews or Search) | |
| Meta-ExternalAgent | Meta | Meta AI training (replaced FacebookBot; top-4 crawler by volume) |
| Amazonbot | Amazon | Alexa answers + Amazon AI training |
| CCBot | Common Crawl | Common Crawl open corpus (used by many AI models) |
| Bytespider | ByteDance | TikTok/ByteDance AI training (undocumented, volatile) |
| Diffbot | Diffbot | Diffbot structured extraction / AI datasets |
| Applebot-Extended | Apple | Apple AI training opt-out token (Applebot does the crawling) |
| ChatGPT-User | OpenAI | User-triggered live fetch from ChatGPT (robots.txt may not apply)May not honour robots.txt |
| OAI-AdsBot | OpenAI | Validates landing pages submitted as ChatGPT ads (not used for training)May not honour robots.txt |
| Claude-User | Anthropic | User-triggered live fetch from ClaudeMay not honour robots.txt |
| Perplexity-User | Perplexity | User-triggered fetch from Perplexity (their docs say it generally ignores robots.txt)May not honour robots.txt |
| Meta-ExternalFetcher | Meta | User-triggered fetch for Meta AI assistants (may ignore robots.txt)May not honour robots.txt |
| MistralAI-User | Mistral | User-triggered fetch for Le Chat citations |
Dead tokens you can safely delete: Claude-Web, anthropic-ai, FacebookBot, cohere-ai. No current crawler sends them.
Five mistakes that make AI robots.txt rules backfire
Assuming a wildcard Allow covers AI crawlers when a specific block exists.
Only the most specific matching User-agent group applies. If you have a GPTBot block anywhere in the file, your `User-agent: *` rules are ignored for GPTBot entirely — including any Allow lines.
Blocking GPTBot to stay out of ChatGPT.
GPTBot is the training crawler. ChatGPT's search index is fed by OAI-SearchBot. Blocking GPTBot opts you out of training but leaves you citable; blocking OAI-SearchBot is what removes you from ChatGPT answers.
Blocking Google-Extended and expecting AI Overviews to stop.
Google-Extended is a Gemini training and grounding opt-out token. It is not a crawler and it does not control AI Overviews, which are served from the regular Google index via Googlebot.
Keeping rules for Claude-Web, anthropic-ai, FacebookBot or cohere-ai.
All four are dead tokens that no current crawler sends. The rules do nothing. Anthropic now uses ClaudeBot, Claude-SearchBot and Claude-User; Meta uses Meta-ExternalAgent and Meta-ExternalFetcher.
Writing robots.txt rules and assuming the bots then get through.
robots.txt is a request, not an access control. A WAF or bot-management rule at Cloudflare can still return 403 to a crawler you explicitly allowed. Allowing in robots.txt and blocking at the edge is the most common invisible failure.
Frequently Asked Questions
How do I block AI crawlers in robots.txt?
Add a separate User-agent block for each crawler followed by Disallow: /. For example, 'User-agent: GPTBot' then 'Disallow: /' blocks OpenAI's training crawler. There is no single directive that blocks all AI crawlers at once — each vendor uses its own user-agent token, so you need one block per bot. The generator on this page writes all 18 blocks for you.
What should my robots.txt say for ChatGPT?
It depends which behaviour you want. OAI-SearchBot controls whether your pages can be cited in ChatGPT search answers — allow it to stay visible. GPTBot controls whether your content is used for model training — block it if you want to opt out. ChatGPT-User fires when a user pastes your URL into ChatGPT. The three are independent, so allowing OAI-SearchBot while blocking GPTBot is a valid and common setup.
Do I need separate robots.txt rules for each AI crawler?
Yes. Each AI crawler has its own user-agent string. A rule for GPTBot doesn't affect ClaudeBot or PerplexityBot. You need a separate User-agent block for each crawler you want to control. This generator creates all the rules you need.
Where do I put these rules in my robots.txt?
Add the generated rules to your existing robots.txt file, which lives at your domain root (e.g., example.com/robots.txt). You can add AI crawler rules anywhere in the file. They won't affect your existing Googlebot or Bingbot rules.
Will blocking AI crawlers affect my Google rankings?
No. Blocking AI crawlers like GPTBot, ClaudeBot, or PerplexityBot has zero effect on your Google Search rankings. Even blocking Google-Extended only affects Gemini training and grounding, not your search position or your eligibility for AI Overviews.
Can I allow ChatGPT Search but block ChatGPT training?
Yes. Allow OAI-SearchBot (search citations) while blocking GPTBot (training data). This is a popular configuration. Your site gets cited in ChatGPT answers without contributing to model training.
Does 'User-agent: * / Disallow: /' block AI crawlers?
Yes, for every crawler that has no more specific group of its own. That is how most sites accidentally become invisible to AI search. But the reverse also matters: if you add a specific block for one bot, the wildcard group stops applying to that bot completely, including any Allow lines it contains.
How long until crawlers notice my robots.txt changes?
Most AI crawlers re-check robots.txt every few days to a few weeks. There's no instant effect, and no equivalent of a 'submit' button. After updating your robots.txt, re-run the check at the top of this page to confirm the file parses the way you intended.
Is robots.txt enough to keep AI crawlers out?
No. robots.txt is a voluntary standard. Well-behaved crawlers from OpenAI, Anthropic, Google and Perplexity honour it. User-triggered fetchers may not, and undocumented crawlers may ignore it entirely. If you need enforcement rather than a request, block at your firewall or CDN instead.
Individual Crawler Guides
Check a Specific Engine
Verify Your robots.txt After Updating
robots.txt states your policy. A scan proves what the crawlers actually get — including firewall blocks that robots.txt can't see. Check all 18 AI crawlers against your live site.
Scan Your Site FreeNo signup needed