GPTBot & robots.txt
Allow or Block OpenAI's Crawler
Check if GPTBot can access your site, and learn exactly how to configure your robots.txt for OpenAI's crawlers.
What is GPTBot?
GPTBot is OpenAI's official web crawler. It visits websites to collect content used for training ChatGPT and other OpenAI models. GPTBot respects robots.txt rules and identifies itself with the user-agent string GPTBot.
OpenAI also operates ChatGPT-User (for real-time browsing) and OAI-SearchBot (for ChatGPT's search feature). Each can be controlled independently in robots.txt.
robots.txt Syntax for GPTBot
Copy-paste these examples into your robots.txt file.
Allow GPTBot (Recommended for most sites)
# Allow OpenAI crawlers
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: OAI-SearchBot
Allow: /
This allows all OpenAI crawlers full access to your site.
Block GPTBot
# Block OpenAI crawlers
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
This blocks all OpenAI crawlers from your entire site.
Partial Access (Allow some paths)
# Allow GPTBot but block private areas
User-agent: GPTBot
Allow: /blog/
Allow: /docs/
Disallow: /admin/
Disallow: /private/
Disallow: /api/
When to Allow vs Block GPTBot
Allow GPTBot When...
- ✓You want ChatGPT to cite your content
- ✓You want brand visibility in AI answers
- ✓You publish public information or guides
- ✓You want AI-driven referral traffic
Block GPTBot When...
- ✗Your content is behind a paywall
- ✗You don't want AI training on your content
- ✗You have licensing concerns
- ✗Your content is private or sensitive
Frequently Asked Questions
What is the GPTBot user agent string?
The full user agent string is: 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)'. In robots.txt, you reference it simply as 'GPTBot'.
What's the difference between GPTBot and ChatGPT-User?
GPTBot crawls websites to collect training data for OpenAI's models. ChatGPT-User is used when a ChatGPT user asks the AI to browse a specific URL in real-time. You can allow one and block the other independently in robots.txt.
Does blocking GPTBot affect ChatGPT's knowledge of my site?
Yes. If GPTBot is blocked, OpenAI cannot crawl your content for model training. ChatGPT may still have older information from before the block, but it won't have current data. Blocking ChatGPT-User also prevents real-time browsing of your pages.
Can I allow GPTBot for some pages but not others?
Yes. Use specific path rules in robots.txt. For example, 'Disallow: /private/' blocks GPTBot from your /private/ directory while allowing access to everything else. You can mix Allow and Disallow rules for fine-grained control.
How long until GPTBot notices my robots.txt changes?
OpenAI states that GPTBot checks robots.txt regularly, but there's no guaranteed timeframe. Changes typically take effect within days to weeks. Use BotView to verify your current robots.txt rules are correctly configured.
Related robots.txt Guides
- ClaudeBot & robots.txt — Anthropic / Claude
- PerplexityBot & robots.txt — Perplexity AI
- Google-Extended & robots.txt — Google / Gemini
- CCBot & robots.txt — Common Crawl
- Check your ChatGPT visibility
Check Your GPTBot Configuration
See if GPTBot can access your site. Get a full report on all OpenAI crawler access.
No credit card required