Your robots.txt Says Allow. Your Firewall Says 403.
Cloudflare's bot settings silently block GPTBot, ClaudeBot, and PerplexityBot at the edge — before robots.txt is even read. Most robots.txt checkers can't see it. A live fetch can.
Test It Live: What Does Each Bot Actually Get?
We request your page as 8 AI crawler user agents (the 3 search gates plus GPTBot, ClaudeBot, Google-Extended, CCBot and ChatGPT-User) and a normal-browser baseline, then show the real HTTP response per bot — including WAF 403s and challenge pages, with the provider identified.
Live HTTP check — we request your page with each bot's user agent and show you exactly what comes back. No signup needed.
Why robots.txt Isn't the Whole Story
Bot access is decided at two independent layers, and most tooling only looks at one of them.
Layer 1: robots.txt (advisory). A text file the crawler downloads and chooses to respect. It works because well-behaved crawlers — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot — honor it. It blocks nothing by itself.
Layer 2: the firewall/CDN (enforced). Cloudflare sits in front of your origin and decides which requests get through at all. If its bot rules match an AI crawler's user agent, the request is rejected with a 403 or a JavaScript challenge the bot can't solve. The crawler never reaches your content — and in many configurations it can't even fetch your robots.txt to see the Allow you wrote.
The failure mode is invisible from the inside: your robots.txt audit passes, your pages load fine in a browser, and meanwhile every AI engine's crawler has been bouncing off a 403 for months. In mid-2024 Cloudflare shipped a one-click "block AI bots" setting and later enabled AI-crawler blocking by default for new zones — a lot of sites are blocking AI search without ever having decided to.
The Cloudflare Settings That Block AI Crawlers
Four separate features can 403 an AI bot. Any one of them is enough.
"Block AI Bots" toggle
Security → Bots. One switch that blocks known AI crawler user agents (GPTBot, ClaudeBot, PerplexityBot, CCBot, and more) across your whole zone. Enabled by default on many newer zones — the most common cause of a surprise GPTBot 403.
Bot Fight Mode / Super Bot Fight Mode
Scores every request. Non-browser traffic lands in "definitely automated" and gets challenged or blocked depending on your action settings. The "Verified bots" category has its own action — set it to Block and even Cloudflare-verified AI search crawlers bounce.
WAF custom rules
Rules matching user-agent substrings like "bot" or "GPT", often added during a scraping incident and forgotten. They fire before your origin sees anything, and they don't distinguish a search-index gate bot from a scraper.
Under Attack mode & managed challenges
Serves a JavaScript challenge ("Just a moment...") that humans pass invisibly and crawlers fail. If it's left on site-wide, every AI crawler sees the challenge page instead of your content.
How to Fix It
Confirm it's the firewall, not robots.txt
Run the live check at the top of this page. A green baseline with amber "WAF 403" rows means requests die at the edge. If the pills say Cloudflare, you know which dashboard to open.
Turn off the blanket AI-bot block
Cloudflare dashboard → Security → Bots → disable "Block AI bots" (or configure it to allow the crawlers you want). If you use Cloudflare's robots.txt management, make sure it isn't injecting Disallow rules you didn't write.
Allow verified bots in Super Bot Fight Mode
Set the "Verified bots" action to Allow. Major AI search crawlers are on Cloudflare's verified list; this lets them through while automated junk still gets challenged. For bots not on the list, add a WAF custom rule with action "Skip" matching their exact user agents, ordered above any block rules.
Keep your policy in robots.txt, then re-test
The firewall decides who gets in; robots.txt states your policy for well-behaved crawlers. Allow the search gates, block training bots if that's your choice — our robots.txt generator builds the rules. Then run the live check again and confirm every gate bot gets a 200.
Frequently Asked Questions
Why is GPTBot getting a 403 from my site?
A 403 to GPTBot almost always comes from a firewall or CDN layer, not your application. On Cloudflare, the usual causes are the 'Block AI bots' setting (on by default for many new zones since mid-2024), Super Bot Fight Mode challenging non-browser traffic, or a WAF custom rule matching bot user agents. Your robots.txt is irrelevant here — the request is rejected before your server or robots.txt policy ever applies.
Does robots.txt override Cloudflare's bot blocking?
No — they operate at different layers. robots.txt is a polite convention the crawler reads and chooses to obey. Cloudflare's firewall enforces at the network edge: if it returns a 403 or a JavaScript challenge, the crawler never gets your content no matter what robots.txt says. An 'Allow' in robots.txt with a WAF block still means invisible.
Which Cloudflare setting blocks AI crawlers?
Check three places: (1) Security → Bots → 'Block AI bots' (also surfaced as 'AI Scrapers and Crawlers') — a one-click block of known AI crawler user agents; (2) Super Bot Fight Mode — its 'Definitely automated' and 'Verified bots' actions can challenge or block crawlers; (3) WAF custom rules — many sites have a rule matching user agents like 'GPTBot' or 'bot' added during a scraping scare.
Should I let AI crawlers through Cloudflare?
Separate the roles. Search-index bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are visibility gates — blocking them removes you from ChatGPT search, Claude search, and Perplexity answers. Training bots (GPTBot, ClaudeBot, CCBot) are a policy choice; blocking them costs no AI-search visibility. A sensible setup allows the gates at the firewall and expresses your training policy in robots.txt.
How do I allow only some AI bots through Cloudflare?
Turn off the blanket 'Block AI bots' toggle, then use a WAF custom rule with the action 'Skip' matching the user agents you want to admit (e.g., OAI-SearchBot, Claude-SearchBot, PerplexityBot), ordered before any blocking rules. In Super Bot Fight Mode, set 'Verified bots' to Allow — most major AI search crawlers are on Cloudflare's verified bots list.
How can I tell if a bot block is Cloudflare and not my server?
Cloudflare fingerprints: the 403 response carries a cf-ray header, the body is often Cloudflare's 'Sorry, you have been blocked' or 'Just a moment...' challenge page, and challenge responses include a cf-mitigated header. If a normal browser gets a 200 while a bot user agent gets a challenge page from the same URL, it's edge-level blocking. Our live checker above shows exactly this comparison, including the challenge provider.
Related
- Check all 18 AI crawlers at once — Full robots.txt + access report
- AI robots.txt generator — Build your allow/block rules
- GPTBot & robots.txt guide — OpenAI's training crawler
- AI crawler monitoring — Get alerted if a firewall change blocks you again
- Why isn't my site in ChatGPT?
Find Out What Your Firewall Is Really Doing
Run a full scan: robots.txt rules, live fetch-as-bot probes for all 18 crawlers, WAF detection, and a per-LLM visibility score.
Scan Your Site FreeNo signup needed