Crawler policy
In a full scan, inspect which robots.txt rule applies to the crawler and URL. Read the bot’s role before changing a training or search policy.
Understand crawler roles →18 AI Crawlers Checked
AI Crawler Checker — Test HTTP Access
Probe your page using 18 AI crawler user agents and inspect the HTTP responses. Full scans add robots.txt and rendering analysis.
Live HTTP check — we request your page with each bot’s user agent and show what comes back. No signup needed.
Requests come from BotView’s network, not the real crawlers. A successful probe does not guarantee indexing or an AI citation.
Need screenshots and HTML analysis? Run a full scan →Three kinds of evidence, three different questions
HTTP response
What did BotView receive using a bot user agent? A 403 or challenge is a reason to inspect CDN logs, not proof of how a verified crawler was treated.
Investigate a blocked request →Rendered content
Does the main content depend on JavaScript? A full scan compares initial HTML and browser rendering, with screenshots and failed-resource analysis.
See a worked rendering example →Looking for business mentions and citations? AI Answer Check records selected OpenAI answers with web search on paid plans. An access check does not measure those answers.
All 18 AI Crawlers We Check
Crawler roles identify search, training and user-fetch requests. The quick check probes HTTP access, not answers from 18 engines.
OAI-SearchBot
OpenAI
ChatGPT search index
GPTBot
OpenAI
Model training
ChatGPT-User
OpenAI
User-triggered fetch
OAI-AdsBot
OpenAI
ChatGPT ad landing-page checks
Claude-SearchBot
Anthropic
Claude search index
ClaudeBot
Anthropic
Model training
Claude-User
Anthropic
User-triggered fetch
PerplexityBot
Perplexity
Search index
Perplexity-User
Perplexity
User-triggered fetch
Google-Extended
Gemini training/grounding opt-out
Meta-ExternalAgent
Meta
Meta AI training
Meta-ExternalFetcher
Meta
User-triggered fetch
Amazonbot
Amazon
Alexa answers + AI training
MistralAI-User
Mistral
Le Chat citations
CCBot
Common Crawl
Open dataset
Bytespider
ByteDance
TikTok AI
Diffbot
Diffbot
Knowledge graph
Applebot-Extended
Apple
Apple AI opt-out
Frequently Asked Questions
What makes BotView different from other AI crawler checkers?
The live checker tests HTTP responses using bot user-agent strings. A separate full scan adds robots.txt analysis, initial-versus-rendered content, failed resources and regular-browser/Googlebot-like screenshots. Requests come from BotView, not the providers’ crawler networks; each result describes that diagnostic request.
How many AI crawlers does BotView check?
BotView checks 18 AI crawlers across all major platforms: OpenAI (OAI-SearchBot, GPTBot, ChatGPT-User, OAI-AdsBot), Anthropic (Claude-SearchBot, ClaudeBot, Claude-User), Perplexity (PerplexityBot, Perplexity-User), Google (Google-Extended), Meta (Meta-ExternalAgent, Meta-ExternalFetcher), Amazon (Amazonbot), Mistral (MistralAI-User), Common Crawl (CCBot), ByteDance (Bytespider), Diffbot, and Apple (Applebot-Extended). We group them by role: search-index bots are the visibility gates, training bots are a policy choice, and user-fetch agents are informational.
Why would my site be invisible to AI crawlers even if robots.txt allows them?
robots.txt is just one factor. Your content might be invisible because it's rendered entirely by JavaScript (which many crawlers can't execute), behind authentication, loaded dynamically via API calls, or served as a Single Page Application shell with no server-side content. BotView checks all of these issues.
Should I allow all AI crawlers?
Separate the roles. For AI-search visibility, allow the search-index bots: OAI-SearchBot (ChatGPT search), Claude-SearchBot (Claude), and PerplexityBot (Perplexity) — restrictions can affect retrieval in those providers’ search services. Training bots (GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Bytespider) are a policy choice: their robots.txt policies are separate from search crawler access. An opt-out does not establish what is already in a model’s training data. BotView's report shows each crawler's role so you can decide per bot.
How often should I check my AI crawler access?
Check after any CMS update, plugin installation, CDN configuration change, or security audit. These commonly introduce AI crawler blocks. For ongoing monitoring, BotView Pro offers 50 scans per month to track changes over time.
Add Screenshots and Content Analysis
Compare initial HTML and browser-rendered content, then investigate missing text or failed resources. Free scans cover up to 3 pages.
No credit card required