PerplexityBot & robots.txt
Allow or Block Perplexity's Crawler
Configure your robots.txt for Perplexity AI's crawler. Check if PerplexityBot can actually access your site.
Last verified against Perplexity's bot documentation: 2026-10-18
PerplexityBot: the facts
Every fact below is taken from Perplexity's bot documentationand the published IP lists, last verified 2026-10-18.
PerplexityBotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)https://www.perplexity.com/perplexitybot.jsonMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)https://www.perplexity.com/perplexity-user.jsonWhat is PerplexityBot?
PerplexityBot is Perplexity AI's web crawler. It indexes content for Perplexity's AI-powered search engine, which generates answers with source citations and links back to the original pages. Unlike some AI crawlers, Perplexity's model always attributes and links to its sources.
The crawler identifies itself with the user-agent token PerplexityBot. Perplexity operates two agents: PerplexityBot builds the search index (and respects robots.txt), while Perplexity-User fetches pages on a direct user request — Perplexity's own docs say it generally ignores robots.txt. Compare OpenAI (which has GPTBot, ChatGPT-User, and OAI-SearchBot) or Anthropic (which has ClaudeBot, Claude-SearchBot, and Claude-User).
The Cloudflare Crawling Report
On August 4, 2025, Cloudflare reported that Perplexity used an undeclared crawler after its declared crawlers were blocked, and removed Perplexity from its verified bot list. BotView has not independently tested this. This is why checking actual crawler access — not just robots.txt rules — matters. Check your Perplexity visibility.
robots.txt Syntax for PerplexityBot
Copy-paste these examples into your robots.txt file.
Allow PerplexityBot (Recommended)
# Allow Perplexity crawler
User-agent: PerplexityBot
Allow: /
Allows Perplexity to index your content for its AI search engine. Your pages can appear as cited sources in Perplexity answers.
Block PerplexityBot
# Block Perplexity crawler
User-agent: PerplexityBot
Disallow: /
Blocks PerplexityBot from your entire site. Note: due to past compliance issues, consider verifying with an actual crawler check.
Partial Access (Allow some paths)
# Allow PerplexityBot for public content only
User-agent: PerplexityBot
Allow: /blog/
Allow: /docs/
Allow: /guides/
Disallow: /admin/
Disallow: /private/
Disallow: /api/
When to Allow vs Block PerplexityBot
Allow PerplexityBot When...
- ✓You want cited referral traffic from Perplexity
- ✓You publish guides, docs, or reference content
- ✓You want brand visibility in AI-powered search
- ✓You value attributed citations (Perplexity always links)
Block PerplexityBot When...
- ✗Your content is behind a paywall
- ✗You're concerned about AI summarizing without full context
- ✗You have licensing restrictions on content
- ✗You don't trust the crawling practices (see controversy above)
Frequently Asked Questions
What is the PerplexityBot user agent string?
Perplexity's documentation gives the full user-agent string as: 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)'. Match on the token 'PerplexityBot'; in robots.txt, reference it as 'User-agent: PerplexityBot'. Source: docs.perplexity.ai/guides/bots, verified 2026-10-18.
How do I verify a request is really from PerplexityBot or Perplexity-User?
Perplexity publishes IP lists at https://www.perplexity.com/perplexitybot.json and https://www.perplexity.com/perplexity-user.json. Check the requesting IP against the matching list; a request that claims the user agent from outside the published ranges is not from Perplexity.
Did Perplexity really ignore robots.txt?
On August 4, 2025, Cloudflare reported that Perplexity switched to an undeclared crawler (a generic browser user agent, IPs outside Perplexity's published ranges) after its declared crawlers were blocked, and removed Perplexity from its verified bot list (https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/, checked 2026-10-18). BotView has not independently tested this. BotView checks your actual page accessibility, not just your robots.txt rules.
Does Perplexity send referral traffic back to my site?
Yes. Unlike some AI tools that summarize content without linking, Perplexity includes source citations with clickable links in every answer. Sites that allow PerplexityBot can receive meaningful referral traffic when their content is cited.
What's the difference between blocking PerplexityBot and blocking GPTBot?
PerplexityBot powers Perplexity's AI search engine, which always cites and links to sources. GPTBot powers ChatGPT's training data. Blocking PerplexityBot stops your content from appearing in Perplexity's cited answers. Blocking GPTBot stops OpenAI from training on your content. They are independent — blocking one doesn't affect the other.
How quickly does Perplexity respect robots.txt changes?
Perplexity re-checks robots.txt on subsequent crawl attempts. Changes typically take effect within days. Use BotView to verify your robots.txt is correctly configured for PerplexityBot and to check whether your content is actually accessible.
Related robots.txt Guides
- AI robots.txt checker — See which of 18 AI crawlers your live robots.txt blocks
- GPTBot & robots.txt — OpenAI / ChatGPT
- ClaudeBot & robots.txt — Anthropic / Claude
- Google-Extended & robots.txt — Google / Gemini
- CCBot & robots.txt — Common Crawl
- Check all 18 AI crawlers at once
Check Your PerplexityBot Configuration
See if PerplexityBot can actually access your site — not just what your robots.txt says.
No credit card required