OAI-SearchBot & robots.txt
Control ChatGPT Search Access to Your Site
OAI-SearchBot is the crawler behind ChatGPT Search. When ChatGPT cites a source in its answers, this is the bot that fetched it. Here's how to control it.
What is OAI-SearchBot?
OAI-SearchBot is OpenAI's dedicated crawler for ChatGPT Search (the feature formerly known as SearchGPT). It indexes web pages so ChatGPT can cite them as sources when answering questions that need current information.
This is different from GPTBot, which collects training data. OAI-SearchBot is closer to how Googlebot works: it crawls your pages so they can appear in search results and get cited with a link back to your site.
OpenAI's Three Crawlers
| Bot | Purpose | Impact |
|---|---|---|
GPTBot | Model training data | Your content trains future models |
ChatGPT-User | Live page browsing | User asks ChatGPT to visit your URL |
OAI-SearchBot | Search index | ChatGPT cites your page in answers |
robots.txt Syntax for OAI-SearchBot
Copy-paste these examples into your robots.txt file.
Allow OAI-SearchBot (Recommended)
# Allow ChatGPT Search to cite your pages
User-agent: OAI-SearchBot
Allow: /
ChatGPT Search can index and cite your content with source links.
Block OAI-SearchBot
# Block ChatGPT Search
User-agent: OAI-SearchBot
Disallow: /
Your pages won't appear as cited sources in ChatGPT Search.
Common Setup: Allow Search, Block Training
# Allow ChatGPT Search citations but block training
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
# Block training data collection
User-agent: GPTBot
Disallow: /
This is a popular configuration. Your site gets cited in ChatGPT answers and users can browse your pages, but your content isn't used for training future models.
Why OAI-SearchBot Matters
Referral Traffic
When ChatGPT cites your page, it includes a clickable source link. Users click through to read more. This is a new traffic channel.
Brand Visibility
Being cited by ChatGPT builds authority. Your brand appears alongside the answer, visible to millions of ChatGPT users searching daily.
No Training Cost
Unlike GPTBot, OAI-SearchBot doesn't collect training data. You get the citation benefits without contributing to model training.
Frequently Asked Questions
What is OAI-SearchBot?
OAI-SearchBot is OpenAI's web crawler for ChatGPT Search (formerly SearchGPT). When a ChatGPT user asks a question that needs current information, OAI-SearchBot fetches and indexes web pages so ChatGPT can cite them in its answer with a source link.
What's the difference between OAI-SearchBot, GPTBot, and ChatGPT-User?
GPTBot crawls pages for AI model training data. ChatGPT-User browses specific URLs when a user asks ChatGPT to visit a page. OAI-SearchBot indexes pages for ChatGPT's search feature so it can cite sources in real-time answers. Each has a separate user-agent and can be controlled independently in robots.txt.
If I block OAI-SearchBot, will my site still appear in ChatGPT answers?
ChatGPT may still mention your site from its training data (via GPTBot), but it won't be able to cite your pages as a live source in search results. Blocking OAI-SearchBot specifically removes you from ChatGPT Search citations.
Does OAI-SearchBot respect robots.txt?
Yes. OpenAI has confirmed that OAI-SearchBot respects robots.txt directives. If you add a Disallow rule for OAI-SearchBot, it will stop crawling those pages for ChatGPT Search.
Should I allow OAI-SearchBot?
For most websites, yes. Allowing OAI-SearchBot means ChatGPT Search can cite your content and link back to your site, driving referral traffic. Unlike GPTBot (training), OAI-SearchBot is about real-time search citations, similar to how Googlebot indexes pages for Google Search.
What is the OAI-SearchBot user agent string?
The user agent string is: 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)'. In robots.txt, reference it as 'OAI-SearchBot'.
Related robots.txt Guides
- GPTBot & robots.txt — OpenAI training crawler
- ClaudeBot & robots.txt — Anthropic / Claude
- PerplexityBot & robots.txt — Perplexity AI
- Applebot-Extended & robots.txt — Apple Intelligence
- Complete AI Crawlers Guide — All 14 bots explained
Check Your OAI-SearchBot Configuration
See if ChatGPT Search can access your site. Instant report on all 14 AI crawler rules.
Free scan, no signup needed