How to Detect ClaudeBot Traffic on Your Site
Anthropic runs three different crawlers, and they mean three very different things when they show up in your logs. Here are the exact user-agent strings, the commands to find them, and how to read what you see.
The Three Anthropic User Agents
All three follow the same format: a token, a version, and a contact address. Match on the token.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected])
Anthropic collecting data to train Claude models. Often the highest-volume of the three. Respects robots.txt; blocking it is a policy choice that doesn't affect Claude search visibility.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; [email protected])
A real person asked Claude something and Claude fetched your page live to answer. This is the strongest signal in the group — actual humans are reaching your content through Claude.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; [email protected])
Builds the index behind Claude's web search. If this bot can't reach you, your site can't appear in Claude search results — treat 403s to it as a visibility incident.
Cleaning up old configs: Claude-Web and anthropic-ai are dead tokens Anthropic no longer uses — rules targeting them do nothing.
How to check traffic from Claude in your logs
All Claude visits in your nginx/Apache access log (any of the three agents):
grep -iE "claudebot|claude-user|claude-searchbot" /var/log/nginx/access.log
Count visits per agent to see the mix:
grep -ioE "claudebot|claude-user|claude-searchbot" /var/log/nginx/access.log \ | sort | uniq -c | sort -rn
Which pages ClaudeBot is crawling most:
grep -i "claudebot" /var/log/nginx/access.log \
| awk '{print $7}' | sort | uniq -c | sort -rn | head -20And crucially — whether you're accidentally rejecting Claude. Non-200 responses to any Claude agent:
grep -iE "claudebot|claude-user|claude-searchbot" /var/log/nginx/access.log \
| awk '$9 != 200 {print $9, $7}' | sort | uniq -c | sort -rnA wall of 403s to Claude-SearchBot means your firewall or WAF is blocking the Claude search index — usually a CDN bot rule rather than anything in your own config. See Cloudflare blocking AI crawlers for the fix. On Cloudflare, you can also see this traffic without server logs: Security → Bots lists AI crawler visits per bot.
Check the Other Direction: Can Claude Reach You?
Your logs show what Claude did in the past. This checks what happens right now — we fetch your page as ClaudeBot and Claude-SearchBot (plus 6 other major AI crawlers) and show the actual response each one gets.
Live HTTP check — we request your page with each bot's user agent and show you exactly what comes back. No signup needed.
Frequently Asked Questions
What user agent does ClaudeBot use?
ClaudeBot identifies itself with a user-agent string containing 'ClaudeBot/1.0' and a contact address: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected]). Match on the substring 'ClaudeBot' in your logs. Anthropic's other agents use 'Claude-User' and 'Claude-SearchBot' tokens in the same format.
How do I know the traffic is really from Anthropic?
User agents can be spoofed by anyone, and Anthropic doesn't offer a reverse-DNS verification mechanism the way Googlebot does. Anthropic publishes the IP ranges its crawlers use in its docs, so for high-stakes decisions (like firewall rules), match UA plus source IP against the published ranges rather than trusting the UA alone. For analytics purposes, UA matching is usually good enough.
Is ClaudeBot traffic good or bad for my site?
Depends on the agent. Claude-SearchBot visits are unambiguously good — they mean you're being indexed for Claude search. Claude-User visits are the best signal of all: real users reaching you through Claude. ClaudeBot (training) is a policy question — it consumes bandwidth and feeds model training; some sites allow it, some block it in robots.txt. None of them execute JavaScript or load ads, so they won't show up in your JS-based analytics.
Why doesn't Claude traffic appear in Google Analytics?
Crawlers don't run JavaScript analytics tags, so GA4 never sees them. Claude-referred human visits can show up if the user clicks a citation link (look for referrer claude.ai). For the bots themselves, server logs, CDN logs, or Cloudflare's bot analytics are the only places you'll see them.
Should I block ClaudeBot?
Block the training crawler only if you've decided you don't want your content in Claude's training data — add 'User-agent: ClaudeBot / Disallow: /' to robots.txt. Don't block Claude-SearchBot unless you want out of Claude search entirely, and be careful with blanket firewall rules: a rule matching 'Claude' blocks all three agents, including the ones sending you humans.
Related
- Check Claude visibility — Full Claude crawler access check
- ClaudeBot & robots.txt guide — Allow or block Anthropic's crawlers
- Cloudflare blocking AI crawlers — When 403s aren't your fault
- Complete AI crawlers guide — All 18 crawlers explained
- AI crawler monitoring
See Your Full Claude Visibility Picture
robots.txt rules, live fetch-as-bot probes, rendering analysis, and a Claude-specific visibility score — in about 30 seconds.
Scan Your Site FreeNo signup needed