GPTBot & robots.txt
Allow or Block OpenAI's Crawler
Check if GPTBot can access your site, and learn exactly how to configure your robots.txt for OpenAI's crawlers.
What is GPTBot?
GPTBot is OpenAI's official training crawler. It visits websites to collect content used for training OpenAI's models — and that is all it does. Blocking GPTBot does not remove your site from ChatGPT search; that is gated by OAI-SearchBot. GPTBot respects robots.txt rules and identifies itself with the user-agent string GPTBot.
OpenAI also operates ChatGPT-User (for real-time browsing) and OAI-SearchBot (for ChatGPT's search feature). Each can be controlled independently in robots.txt.
robots.txt Syntax for GPTBot
Copy-paste these examples into your robots.txt file.
Allow GPTBot (Recommended for most sites)
# Allow OpenAI crawlers
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: OAI-SearchBot
Allow: /
This allows all OpenAI crawlers full access to your site.
Block GPTBot
# Block OpenAI crawlers
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
This blocks all OpenAI crawlers from your entire site.
Partial Access (Allow some paths)
# Allow GPTBot but block private areas
User-agent: GPTBot
Allow: /blog/
Allow: /docs/
Disallow: /admin/
Disallow: /private/
Disallow: /api/
When to Allow vs Block GPTBot
Allow GPTBot When...
- ✓You want future OpenAI models to know your content
- ✓You want baked-in brand knowledge in AI answers
- ✓You publish public information or guides
- ✓(For citations and referral traffic, the one that matters is OAI-SearchBot)
Block GPTBot When...
- ✗Your content is behind a paywall
- ✗You don't want AI training on your content
- ✗You have licensing concerns
- ✗Your content is private or sensitive
Frequently Asked Questions
What is the GPTBot user agent string?
The full user agent string is: 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)'. In robots.txt, you reference it simply as 'GPTBot'.
What's the difference between GPTBot, OAI-SearchBot, and ChatGPT-User?
GPTBot crawls websites to collect training data for OpenAI's models. OAI-SearchBot builds ChatGPT's search index — it's the crawler that decides whether you appear in ChatGPT search results. ChatGPT-User is used when a ChatGPT user asks the AI to open a specific URL in real-time (OpenAI notes robots.txt may not apply to it). You can allow or block each independently in robots.txt.
Does blocking GPTBot remove my site from ChatGPT search?
No. This is the most common misconception about GPTBot. Blocking it only keeps your content out of future model training. ChatGPT search visibility is controlled by OAI-SearchBot — as long as OAI-SearchBot is allowed, ChatGPT can still find, show, and cite your site in search answers. Many sites deliberately block GPTBot while allowing OAI-SearchBot.
Can I allow GPTBot for some pages but not others?
Yes. Use specific path rules in robots.txt. For example, 'Disallow: /private/' blocks GPTBot from your /private/ directory while allowing access to everything else. You can mix Allow and Disallow rules for fine-grained control.
How long until GPTBot notices my robots.txt changes?
OpenAI states that GPTBot checks robots.txt regularly, but there's no guaranteed timeframe. Changes typically take effect within days to weeks. Use BotView to verify your current robots.txt rules are correctly configured.
Related robots.txt Guides
- ClaudeBot & robots.txt — Anthropic / Claude
- PerplexityBot & robots.txt — Perplexity AI
- Google-Extended & robots.txt — Google / Gemini
- CCBot & robots.txt — Common Crawl
- OAI-SearchBot & robots.txt — ChatGPT Search
- AI robots.txt Generator — Generate rules for all bots
- Check your ChatGPT visibility
Check Your GPTBot Configuration
See if GPTBot can access your site. Get a full report on all OpenAI crawler access.
No credit card required