OAI-SearchBot & robots.txt
Control ChatGPT Search Access to Your Site
OAI-SearchBot is the crawler behind ChatGPT Search. When ChatGPT cites a source in its answers, this is the bot that fetched it. Here's how to control it.
What is OAI-SearchBot?
OAI-SearchBot is OpenAI's dedicated crawler for ChatGPT Search (the feature formerly known as SearchGPT). It indexes web pages so ChatGPT can cite them as sources when answering questions that need current information.
This is different from GPTBot, which collects training data. OAI-SearchBot is closer to how Googlebot works: it crawls your pages so they can appear in search results and get cited with a link back to your site.
OpenAI's Three Crawlers
| Bot | Purpose | Impact |
|---|---|---|
GPTBot | Model training data | Your content trains future models |
ChatGPT-User | Live page browsing | User asks ChatGPT to visit your URL |
OAI-SearchBot | Search index | ChatGPT cites your page in answers |
robots.txt Syntax for OAI-SearchBot
Copy-paste these examples into your robots.txt file.
Allow OAI-SearchBot (Recommended)
# Allow ChatGPT Search to cite your pages
User-agent: OAI-SearchBot
Allow: /
ChatGPT Search can index and cite your content with source links.
Block OAI-SearchBot
# Block ChatGPT Search
User-agent: OAI-SearchBot
Disallow: /
Your pages won't appear as cited sources in ChatGPT Search.
Common Setup: Allow Search, Block Training
# Allow ChatGPT Search citations but block training
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
# Block training data collection
User-agent: GPTBot
Disallow: /
This is a popular configuration. Your site gets cited in ChatGPT answers and users can browse your pages, but your content isn't used for training future models.
Why OAI-SearchBot Matters
Referral Traffic
When ChatGPT cites your page, it includes a clickable source link. Users click through to read more. This is a new traffic channel.
Brand Visibility
Being cited by ChatGPT builds authority. Your brand appears alongside the answer, visible to millions of ChatGPT users searching daily.
No Training Cost
Unlike GPTBot, OAI-SearchBot doesn't collect training data. You get the citation benefits without contributing to model training.
GPTBot vs OAI-SearchBot vs ChatGPT-User: what's the difference?
GPTBot collects training data for OpenAI's foundation models. OAI-SearchBot builds ChatGPT's search index — it is the only one of the three that decides whether ChatGPT can cite your pages. ChatGPT-User fetches a page when a user asks for it, and OpenAI says robots.txt rules may not apply to those requests.
Full details are in the GPTBot & robots.txt guide. Or check which OpenAI bots can reach your site right now.
GPTBot | OAI-SearchBot | ChatGPT-User | |
|---|---|---|---|
| Purpose | Crawls content that may be used to train OpenAI's generative AI foundation models. | Crawls and indexes pages so they can surface in ChatGPT's search features. | Fetches one page when a ChatGPT user or a Custom GPT asks for it. Not an automatic crawler. |
| Role in our taxonomy | Training. Blocking is a policy choice, not a visibility failure. | Search-index. This is the visibility gate for ChatGPT. | User-fetch. Informational — it does not gate anything. |
| What blocking it does | Signals that your content should not be used to train future models. Nothing else changes. | Opts you out of ChatGPT search. OpenAI: opted-out sites “will not be shown in ChatGPT search answers, though can still appear as navigational links.” | Little practical effect — the fetch is user-initiated, so robots.txt may be ignored. A WAF rule is the only hard block. |
| Effect on appearing in ChatGPT answers | None directly. Blocked sites can still be found and cited through ChatGPT search. | Direct. Block it and you drop out of cited ChatGPT search results. | None. OpenAI states ChatGPT-User is not used to determine whether content may appear in Search. |
| Respects robots.txt? | Yes. | Yes. OpenAI notes it can take about 24 hours after a robots.txt change for search to adjust. | Not necessarily. OpenAI: “Because these actions are initiated by a user, robots.txt rules may not apply.” |
| Correct robots.txt linesallow search, block training | User-agent: GPTBot
Disallow: / | User-agent: OAI-SearchBot
Allow: / | User-agent: ChatGPT-User
Allow: /Optional — advisory only, since user-initiated fetches may ignore robots.txt. |
The common mistake that actually removes you from ChatGPT
Most sites that vanish from ChatGPT search never meant to block it. A blanket User-agent: * Disallow: / rule, a copy-pasted “block all AI crawlers” list, or a WAF bot-fight mode takes out OAI-SearchBot along with GPTBot. GPTBot was the one you wanted to stop; OAI-SearchBot is the one whose loss removes you from cited answers. Name the bots separately instead:
# Allow ChatGPT search, opt out of training
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
Then confirm it works in practice — robots.txt can say Allow: / while a CDN challenge page still turns OAI-SearchBot away. Check which OpenAI bots can reach your site right now.
Frequently Asked Questions
What is OAI-SearchBot?
OAI-SearchBot is OpenAI's web crawler for ChatGPT Search (formerly SearchGPT). When a ChatGPT user asks a question that needs current information, OAI-SearchBot fetches and indexes web pages so ChatGPT can cite them in its answer with a source link.
What's the difference between OAI-SearchBot, GPTBot, and ChatGPT-User?
GPTBot crawls pages for AI model training data. ChatGPT-User browses specific URLs when a user asks ChatGPT to visit a page. OAI-SearchBot indexes pages for ChatGPT's search feature so it can cite sources in real-time answers. Each has a separate user-agent and can be controlled independently in robots.txt.
What's the difference between GPTBot and OAI-SearchBot?
GPTBot is OpenAI's training crawler: blocking it only keeps your content out of future model training. OAI-SearchBot is the crawler behind ChatGPT search — it decides whether ChatGPT can find and cite your pages. Block OAI-SearchBot and you drop out of ChatGPT search answers; block GPTBot and you don't.
If I block OAI-SearchBot, will my site still appear in ChatGPT answers?
ChatGPT may still mention your site from its training data (via GPTBot), but it won't be able to cite your pages as a live source in search results. Blocking OAI-SearchBot specifically removes you from ChatGPT Search citations.
Does OAI-SearchBot respect robots.txt?
Yes. OpenAI has confirmed that OAI-SearchBot respects robots.txt directives. If you add a Disallow rule for OAI-SearchBot, it will stop crawling those pages for ChatGPT Search.
Should I allow OAI-SearchBot?
For most websites, yes. Allowing OAI-SearchBot means ChatGPT Search can cite your content and link back to your site, driving referral traffic. Unlike GPTBot (training), OAI-SearchBot is about real-time search citations, similar to how Googlebot indexes pages for Google Search.
What is the OAI-SearchBot user agent string?
OpenAI's published example is: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot'. The version number changes, so match on the token 'OAI-SearchBot'. In robots.txt, reference it as 'OAI-SearchBot'.
Related robots.txt Guides
- GPTBot & robots.txt — OpenAI training crawler
- ClaudeBot & robots.txt — Anthropic / Claude
- PerplexityBot & robots.txt — Perplexity AI
- Applebot-Extended & robots.txt — Apple Intelligence
- Complete AI Crawlers Guide — All 18 bots explained
Check Your OAI-SearchBot Configuration
See if ChatGPT Search can access your site. Instant report on all 18 AI crawler rules.
Free scan, no signup needed