Free crawler check
Can Claude’s crawlers access your website?
Check Claude-SearchBot, ClaudeBot and Claude-User separately. See the robots.txt policy and the response to a request using each crawler’s user agent.
Live HTTP check — we request your page with each bot’s user agent and show what comes back. No signup needed.
Free quick check, no signup. BotView sends the requests from its own network. A challenge here does not prove that Anthropic’s verified crawlers receive the same response.
Need screenshots and a GEO content score? Run a full scan.What to do with your result
Claude-SearchBot disallowed
The matching robots.txt policy restricts the bot Anthropic uses to improve search responses.
Next step: Review that rule if you want the page available for search. Do not change ClaudeBot’s training policy just to fix a search restriction.
ClaudeBot disallowed, search allowed
The site has different policies for training and search. That can be intentional.
Next step: Keep the training restriction if it matches your preference. Inspect Claude-SearchBot and Claude-User independently.
Could not verify access
A challenge, timeout or other failed request prevented a reliable check.
Next step: Look up the request in your CDN logs. Verify crawler identity using Anthropic’s current documentation before deciding whether to adjust a rule.
Which Claude crawler does what?
Anthropic documents three roles: ClaudeBot collects material that could contribute to training; Claude-SearchBot analyzes content to improve search results; Claude-User retrieves content in response to a user request.
Anthropic says restricting Claude-SearchBot may reduce visibility and accuracy in search results. It does not promise citations when the bot is allowed. These roles should be checked separately.
Find the matching rule before changing it
Open the robots.txt file on the exact hostname you are testing. A policy for example.com may differ from one for docs.example.com. Check the full page path too: an allowed homepage does not establish access to a restricted directory.
Keep existing private-path restrictions. If you edit a rule, rerun the check on the affected page rather than relying on a homepage result.
Check traffic from Claude in your logs
Search the user-agent field for Claude-SearchBot, ClaudeBot and Claude-User. Record the requested path, response status and any firewall action. Logs can include blocked requests, depending on where the block occurred and which logs you inspect.
Crawler visits are different from people arriving through Claude. Use referral analytics for visitors, and server or CDN logs for crawler requests. Neither a user-agent string nor a crawler visit proves that Claude cited your page.
Questions about this check
Does allowing ClaudeBot also allow Claude-SearchBot?
They are different user-agent tokens. Evaluate each against the matching robots.txt group and any firewall policy. A rule naming one does not automatically name the other.
My robots.txt allows Claude. Why does the check show a challenge?
Robots.txt describes a crawl policy; a CDN or firewall can still challenge a request. BotView cannot bypass that challenge. Check the security event and compare it with verified Anthropic requests.
Can this checker measure Claude referrals or citations?
No. It checks access signals. The full scan adds content analysis, screenshots and a GEO score. Use your analytics to measure referred visitors; this tool does not count Claude citations.
Sources and further checks
Crawler documentation checked September 6, 2026.
Check the content behind the response
The full scan compares screenshots, initial HTML and rendered content, and checks on-page GEO signals. Free scans cover up to three pages. Results describe this scan; they do not measure search rankings or AI citations.