PerplexityBot & robots.txt
Allow or Block Perplexity's Crawler
Configure your robots.txt for Perplexity AI's crawler. Check if PerplexityBot can actually access your site.
What is PerplexityBot?
PerplexityBot is Perplexity AI's web crawler. It indexes content for Perplexity's AI-powered search engine, which generates answers with source citations and links back to the original pages. Unlike some AI crawlers, Perplexity's model always attributes and links to its sources.
The crawler identifies itself with the user-agent token PerplexityBot. Perplexity operates a single official crawler, unlike OpenAI (which has GPTBot, ChatGPT-User, and OAI-SearchBot) or Anthropic (which has ClaudeBot, Claude-Web, and anthropic-ai).
The 2024 Crawling Controversy
In 2024, Wired and Cloudflare reported that Perplexity was accessing sites that had blocked PerplexityBot, potentially using unmarked IP addresses. Perplexity acknowledged the issue and committed to stricter compliance. This is why checking actual crawler access — not just robots.txt rules — matters. Check your Perplexity visibility.
robots.txt Syntax for PerplexityBot
Copy-paste these examples into your robots.txt file.
Allow PerplexityBot (Recommended)
# Allow Perplexity crawler
User-agent: PerplexityBot
Allow: /
Allows Perplexity to index your content for its AI search engine. Your pages can appear as cited sources in Perplexity answers.
Block PerplexityBot
# Block Perplexity crawler
User-agent: PerplexityBot
Disallow: /
Blocks PerplexityBot from your entire site. Note: due to past compliance issues, consider verifying with an actual crawler check.
Partial Access (Allow some paths)
# Allow PerplexityBot for public content only
User-agent: PerplexityBot
Allow: /blog/
Allow: /docs/
Allow: /guides/
Disallow: /admin/
Disallow: /private/
Disallow: /api/
When to Allow vs Block PerplexityBot
Allow PerplexityBot When...
- ✓You want cited referral traffic from Perplexity
- ✓You publish guides, docs, or reference content
- ✓You want brand visibility in AI-powered search
- ✓You value attributed citations (Perplexity always links)
Block PerplexityBot When...
- ✗Your content is behind a paywall
- ✗You're concerned about AI summarizing without full context
- ✗You have licensing restrictions on content
- ✗You don't trust the crawling practices (see controversy above)
Frequently Asked Questions
What is the PerplexityBot user agent string?
PerplexityBot identifies itself with the user-agent token 'PerplexityBot'. In robots.txt, reference it as 'User-agent: PerplexityBot'. Perplexity has published its crawler details at perplexity.ai/bot.
Did Perplexity really ignore robots.txt?
In mid-2024, investigations by Wired and analysis from Cloudflare revealed that Perplexity was accessing websites that had blocked PerplexityBot, likely through unmarked IP addresses or third-party services. Perplexity acknowledged the issue and committed to stricter robots.txt compliance. BotView checks your actual page accessibility, not just your robots.txt rules.
Does Perplexity send referral traffic back to my site?
Yes. Unlike some AI tools that summarize content without linking, Perplexity includes source citations with clickable links in every answer. Sites that allow PerplexityBot can receive meaningful referral traffic when their content is cited.
What's the difference between blocking PerplexityBot and blocking GPTBot?
PerplexityBot powers Perplexity's AI search engine, which always cites and links to sources. GPTBot powers ChatGPT's training data. Blocking PerplexityBot stops your content from appearing in Perplexity's cited answers. Blocking GPTBot stops OpenAI from training on your content. They are independent — blocking one doesn't affect the other.
How quickly does Perplexity respect robots.txt changes?
Perplexity re-checks robots.txt on subsequent crawl attempts. Changes typically take effect within days. Use BotView to verify your robots.txt is correctly configured for PerplexityBot and to check whether your content is actually accessible.
Related robots.txt Guides
- GPTBot & robots.txt — OpenAI / ChatGPT
- ClaudeBot & robots.txt — Anthropic / Claude
- Google-Extended & robots.txt — Google / Gemini
- CCBot & robots.txt — Common Crawl
- Check all 14 AI crawlers at once
Check Your PerplexityBot Configuration
See if PerplexityBot can actually access your site — not just what your robots.txt says.
No credit card required