GPTBot User Agent & robots.txt
Block or Allow OpenAI's Crawler
The exact user-agent string, OpenAI's published IP ranges for verifying it, and the robots.txt rules that block GPTBot without costing you ChatGPT search citations.
Last verified against OpenAI's crawler documentation: 21 August 2026
GPTBot in short
GPTBot is OpenAI's training crawler. In robots.txt you reference it by the token GPTBot. The full user-agent string it sends is:
To block it, add these two lines to the robots.txt at the root of your domain:
User-agent: GPTBot
Disallow: /
Blocking GPTBot opts you out of model training only. It does not remove you from ChatGPT search — those citations are gated by a different crawler, OAI-SearchBot. Blocking GPTBot is a licensing and policy decision, not an AI visibility one.
GPTBot: the facts
Taken from OpenAI's bot documentation and its published IP list, last verified 21 August 2026.
GPTBotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbothttps://openai.com/gptbot.jsonWhat is GPTBot?
GPTBot is OpenAI's official training crawler. It visits websites to collect content used for training OpenAI's models — and that is all it does. Blocking GPTBot does not remove your site from ChatGPT search; that is gated by OAI-SearchBot. GPTBot respects robots.txt rules and identifies itself with the user-agent string GPTBot.
OpenAI also operates ChatGPT-User (for real-time browsing) and OAI-SearchBot (for ChatGPT's search feature). Each can be controlled independently in robots.txt.
GPTBot user agent: identifying and verifying it
OpenAI runs three separate crawlers, each with its own user-agent string, its own robots.txt token, and its own published IP list. A rule written for one does not apply to the others.
| robots.txt token | Role | Full user-agent string / published IPs |
|---|---|---|
GPTBot | Training | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbothttps://openai.com/gptbot.json |
OAI-SearchBot | ChatGPT search index | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbothttps://openai.com/searchbot.json |
ChatGPT-User | Live user-triggered fetch | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bothttps://openai.com/chatgpt-user.json |
Match on the token — e.g. the substring GPTBot — never the whole string. OpenAI increments the version (GPTBot/1.0, /1.1, /1.4) without notice, so an exact match silently stops working.
The user-agent header proves nothing on its own
Any client can send GPTBot in its user-agent header, and scrapers routinely do it to inherit the access sites grant OpenAI. The only reliable check is the source IP: OpenAI publishes the ranges each crawler uses as JSON, and a request claiming to be GPTBot from an address outside gptbot.json is forged. The lists are per-bot and not interchangeable — an IP on the OAI-SearchBot list is not a valid GPTBot address.
Match GPTBot in nginx
if ($http_user_agent ~ GPTBot) {
return 403;
}
Match GPTBot in Apache
BrowserMatch GPTBot bad_bot
Deny from env=bad_bot
Blocking at the server or WAF layer is a hard block, unlike robots.txt which is a request. Use it to stop spoofers — but if you only want to opt out of training, robots.txt is the right tool, and a blanket server rule is easy to over-apply to OAI-SearchBot and cost you ChatGPT citations.
robots.txt Syntax for GPTBot
Copy-paste these examples into your robots.txt file.
Allow GPTBot (Recommended for most sites)
# Allow OpenAI crawlers
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: OAI-SearchBot
Allow: /
This allows all OpenAI crawlers full access to your site.
Block GPTBot (opt out of training, keep citations)
# No training, still citable in ChatGPT
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
This is what most sites mean by “block GPTBot”: your content stays out of OpenAI's training runs, and ChatGPT search can still find and cite you.
Partial Access (Allow some paths)
# Allow GPTBot but block private areas
User-agent: GPTBot
Allow: /blog/
Allow: /docs/
Disallow: /admin/
Disallow: /private/
Disallow: /api/
Block CCBot but allow GPTBot
# Out of Common Crawl, open to OpenAI
User-agent: CCBot
Disallow: /
User-agent: GPTBot
Allow: /
Rules are per-token, so each crawler needs its own group. See the CCBot guide for what the Common Crawl archive feeds.
Block every OpenAI crawler (full opt-out)
# Block all OpenAI crawlers
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
Only use this if you want out of ChatGPT entirely. Disallowing OAI-SearchBot is the line that removes you from ChatGPT search answers — the GPTBot line alone does not. OpenAI also notes that robots.txt may not apply to ChatGPT-User, since those fetches are triggered by a person asking for your page.
When to Allow vs Block GPTBot
Allow GPTBot When...
- ✓You want future OpenAI models to know your content
- ✓You want baked-in brand knowledge in AI answers
- ✓You publish public information or guides
- ✓(For citations and referral traffic, the one that matters is OAI-SearchBot)
Block GPTBot When...
- ✗Your content is behind a paywall
- ✗You don't want AI training on your content
- ✗You have licensing concerns
- ✗Your content is private or sensitive
Your robots.txt Allows GPTBot — Is It Actually Getting In?
A robots.txt rule is a request, not a guarantee. Two things trip up sites that think they're wide open to OpenAI.
Clean robots.txt, blocked by your WAF
Your robots.txt can say Allow: / while Cloudflare (or DataDome, Akamai, Imperva) quietly serves GPTBot a "Just a moment…" challenge or an Access Denied page. GPTBot can't solve a JavaScript challenge, so it leaves with nothing — and no robots.txt line explains why.
BotView fetches your pages as GPTBot itself and shows the real response — full page, redirect, challenge, or block. A robots.txt parser will tell you you're fine when you're not.
GPTBot is training — the citation gate is OAI-SearchBot
BotView scores your visibility per engine, not as one blended number. It separates GPTBot (training) from OAI-SearchBot (the crawler that decides whether ChatGPT can cite you), and checks whether a WAF is silently blocking either one.
If a fetch-as-bot probe times out or is reset, that engine is capped at "limited" with a could-not-verify flag — we never fabricate a block or claim visibility we couldn't confirm.
GPTBot vs OAI-SearchBot vs ChatGPT-User: what's the difference?
GPTBot collects training data for OpenAI's foundation models. OAI-SearchBot builds ChatGPT's search index — it is the only one of the three that decides whether ChatGPT can cite your pages. ChatGPT-User fetches a page when a user asks for it, and OpenAI says robots.txt rules may not apply to those requests.
Full details are in the OAI-SearchBot & robots.txt guide. Or check which OpenAI bots can reach your site right now.
GPTBot | OAI-SearchBot | ChatGPT-User | |
|---|---|---|---|
| Purpose | Crawls content that may be used to train OpenAI's generative AI foundation models. | Crawls and indexes pages so they can surface in ChatGPT's search features. | Fetches one page when a ChatGPT user or a Custom GPT asks for it. Not an automatic crawler. |
| Role in our taxonomy | Training. Blocking is a policy choice, not a visibility failure. | Search-index. This is the visibility gate for ChatGPT. | User-fetch. Informational — it does not gate anything. |
| What blocking it does | Signals that your content should not be used to train future models. Nothing else changes. | Opts you out of ChatGPT search. OpenAI: opted-out sites “will not be shown in ChatGPT search answers, though can still appear as navigational links.” | Little practical effect — the fetch is user-initiated, so robots.txt may be ignored. A WAF rule is the only hard block. |
| Effect on appearing in ChatGPT answers | None directly. Blocked sites can still be found and cited through ChatGPT search. | Direct. Block it and you drop out of cited ChatGPT search results. | None. OpenAI states ChatGPT-User is not used to determine whether content may appear in Search. |
| Respects robots.txt? | Yes. | Yes. OpenAI notes it can take about 24 hours after a robots.txt change for search to adjust. | Not necessarily. OpenAI: “Because these actions are initiated by a user, robots.txt rules may not apply.” |
| Correct robots.txt linesallow search, block training | User-agent: GPTBot
Disallow: / | User-agent: OAI-SearchBot
Allow: / | User-agent: ChatGPT-User
Allow: /Optional — advisory only, since user-initiated fetches may ignore robots.txt. |
The common mistake that actually removes you from ChatGPT
Most sites that vanish from ChatGPT search never meant to block it. A blanket User-agent: * Disallow: / rule, a copy-pasted “block all AI crawlers” list, or a WAF bot-fight mode takes out OAI-SearchBot along with GPTBot. GPTBot was the one you wanted to stop; OAI-SearchBot is the one whose loss removes you from cited answers. Name the bots separately instead:
# Allow ChatGPT search, opt out of training
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
Then confirm it works in practice — robots.txt can say Allow: / while a CDN challenge page still turns OAI-SearchBot away. Check which OpenAI bots can reach your site right now.
Frequently Asked Questions
What is the GPTBot user agent string?
The full user agent string is: 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot'. In robots.txt, you reference it simply as 'GPTBot'.
What's the difference between GPTBot, OAI-SearchBot, and ChatGPT-User?
GPTBot crawls websites to collect training data for OpenAI's models. OAI-SearchBot builds ChatGPT's search index — it's the crawler that decides whether you appear in ChatGPT search results. ChatGPT-User is used when a ChatGPT user asks the AI to open a specific URL in real-time (OpenAI notes robots.txt may not apply to it). You can allow or block each independently in robots.txt.
What's the difference between GPTBot and OAI-SearchBot?
GPTBot is OpenAI's training crawler: blocking it only keeps your content out of future model training. OAI-SearchBot is the crawler behind ChatGPT search — it decides whether ChatGPT can find and cite your pages. Block OAI-SearchBot and you drop out of ChatGPT search answers; block GPTBot and you don't.
Does blocking GPTBot remove my site from ChatGPT search?
No. This is the most common misconception about GPTBot. Blocking it only keeps your content out of future model training. ChatGPT search visibility is controlled by OAI-SearchBot — as long as OAI-SearchBot is allowed, ChatGPT can still find, show, and cite your site in search answers. Many sites deliberately block GPTBot while allowing OAI-SearchBot.
Can I allow GPTBot for some pages but not others?
Yes. Use specific path rules in robots.txt. For example, 'Disallow: /private/' blocks GPTBot from your /private/ directory while allowing access to everything else. You can mix Allow and Disallow rules for fine-grained control.
How long until GPTBot notices my robots.txt changes?
OpenAI states that GPTBot checks robots.txt regularly, but there's no guaranteed timeframe. Changes typically take effect within days to weeks. Use BotView to verify your current robots.txt rules are correctly configured.
My robots.txt allows GPTBot but it's still being blocked — why?
This is almost always a WAF or CDN, not robots.txt. Services like Cloudflare, DataDome, Akamai, and Imperva can serve GPTBot a 'Just a moment…' JavaScript challenge or an Access Denied page regardless of what robots.txt says. GPTBot can't solve a browser challenge, so it receives nothing and simply moves on. BotView catches this by fetching your pages as GPTBot and reporting the actual response — a robots.txt checker alone will report you as open when you're really walled off. Fix it by allowlisting OpenAI's published GPTBot IP ranges (or setting a bot-management exception for the GPTBot user agent) in your WAF.
How do I verify a request is really from GPTBot?
Check the requesting IP against OpenAI's published GPTBot ranges at https://openai.com/gptbot.json. The user-agent header is trivially forged, so the string alone proves nothing — anything claiming to be GPTBot from an IP outside those ranges is a spoofer, and you can block it without affecting OpenAI. OpenAI publishes separate lists for OAI-SearchBot (searchbot.json) and ChatGPT-User (chatgpt-user.json); an IP is only valid for the bot whose list it appears on.
How do I block GPTBot but still get cited in ChatGPT search?
Disallow GPTBot and allow OAI-SearchBot. GPTBot only feeds model training, while OAI-SearchBot is the crawler that builds ChatGPT's search index and decides whether ChatGPT can find and cite your pages. Blocking GPTBot alone keeps your content out of training runs and leaves ChatGPT search citations intact. Blocking both is what actually removes you from ChatGPT search answers.
How do I block CCBot but allow GPTBot?
robots.txt rules are per-token, so give each crawler its own group: 'User-agent: CCBot' with 'Disallow: /', then 'User-agent: GPTBot' with 'Allow: /'. CCBot is Common Crawl's crawler and its archive is used by many AI labs, so blocking CCBot while allowing GPTBot means OpenAI can still crawl you directly but you drop out of the shared Common Crawl dataset.
How do I match GPTBot in nginx, Apache, or Cloudflare rules?
Match on the substring 'GPTBot' rather than the full user-agent string, because OpenAI increments the version number (GPTBot/1.0, /1.1, /1.4) without notice and an exact match will silently stop working. In nginx use a case-sensitive regex such as 'if ($http_user_agent ~ GPTBot)'; in Apache use 'BrowserMatch GPTBot'; in Cloudflare use a firewall expression on http.user_agent contains "GPTBot". Note that 'GPTBot' is not a substring of ChatGPT-User or OAI-SearchBot, so this will not catch OpenAI's other crawlers.
Related robots.txt Guides
- ClaudeBot & robots.txt — Anthropic / Claude
- PerplexityBot & robots.txt — Perplexity AI
- Google-Extended & robots.txt — Google / Gemini
- CCBot & robots.txt — Common Crawl
- OAI-SearchBot & robots.txt — ChatGPT Search
- AI robots.txt Generator — Generate rules for all bots
- Check your ChatGPT visibility
Check Your GPTBot Configuration
See if GPTBot can access your site. Get a full report on all OpenAI crawler access.
No credit card required