ClaudeBot & robots.txt
Allow or Block Anthropic's Crawler
Check if ClaudeBot can access your site, and learn exactly how to configure your robots.txt for Anthropic's crawlers.
What is ClaudeBot?
ClaudeBot is Anthropic's official web crawler. It collects content used for training Claude AI models. ClaudeBot respects robots.txt rules and identifies itself with the user-agent string ClaudeBot.
Anthropic also operates Claude-SearchBot (builds Claude's search index — the crawler that decides whether Claude can cite you) and Claude-User (fetches a page when a user asks Claude to open it). Each can be controlled independently in robots.txt.
Note: the older Claude-Web and anthropic-ai tokens are deprecated — Anthropic removed them from its documentation and no current crawler matches them. Rules for them do nothing and can be deleted.
robots.txt Syntax for ClaudeBot
Copy-paste these examples into your robots.txt file.
Allow ClaudeBot (Recommended for most sites)
# Allow Anthropic crawlers
User-agent: Claude-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-User
Allow: /
This allows all Anthropic crawlers full access to your site.
Block ClaudeBot
# Block Anthropic crawlers
User-agent: Claude-SearchBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-User
Disallow: /
This blocks all Anthropic crawlers. Blocking Claude-SearchBot removes you from Claude search; blocking only ClaudeBot is a training opt-out that keeps you visible in Claude.
Partial Access (Allow some paths)
# Allow ClaudeBot but block private areas
User-agent: ClaudeBot
Allow: /blog/
Allow: /docs/
Disallow: /admin/
Disallow: /private/
Disallow: /api/
When to Allow vs Block ClaudeBot
Allow ClaudeBot When...
- ✓You want Claude to cite your content
- ✓You serve enterprise or B2B audiences
- ✓You publish technical docs or knowledge bases
- ✓You want visibility in Claude-powered tools
Block ClaudeBot When...
- ✗Your content is behind a paywall
- ✗You don't want AI training on your content
- ✗You have content licensing restrictions
- ✗Your content is proprietary or confidential
Frequently Asked Questions
What are Anthropic's crawler user agent strings?
Anthropic currently uses three identifiers: 'ClaudeBot' (training data crawler), 'Claude-SearchBot' (builds Claude's search index — the visibility gate), and 'Claude-User' (fetches a page when a user asks Claude to open it). You can control each independently in robots.txt. The old 'Claude-Web' and 'anthropic-ai' tokens are deprecated and no longer used.
What's the difference between ClaudeBot and Claude-SearchBot?
ClaudeBot collects training data for Claude models — blocking it is a policy choice that keeps your content out of training. Claude-SearchBot builds the index behind Claude's web search — blocking it removes your site from Claude's search answers and citations. If you care about being visible in Claude, Claude-SearchBot is the one that matters.
Does blocking ClaudeBot remove my site from Claude's answers?
No. Blocking ClaudeBot only keeps your content out of future model training. Claude's live search answers come from the index built by Claude-SearchBot — as long as that's allowed, Claude can still find and cite your site. Block Claude-SearchBot and you disappear from Claude search regardless of your ClaudeBot setting.
Should I still have rules for Claude-Web or anthropic-ai?
No — both are dead tokens. Anthropic has removed Claude-Web and anthropic-ai from its documentation, and no current Anthropic crawler matches them. Rules referencing them do nothing; you can safely delete them from your robots.txt. The current set is ClaudeBot, Claude-SearchBot, and Claude-User.
How do I verify my ClaudeBot robots.txt changes are working?
Use BotView's free scanner to check your site. It analyzes your robots.txt for all Anthropic crawler identifiers and shows you exactly what's allowed or blocked. It also checks whether your content is actually accessible beyond just the robots.txt rules.
Related robots.txt Guides
- GPTBot & robots.txt — OpenAI / ChatGPT
- PerplexityBot & robots.txt — Perplexity AI
- Google-Extended & robots.txt — Google / Gemini
- CCBot & robots.txt — Common Crawl
- Check your Claude visibility
Check Your ClaudeBot Configuration
See if ClaudeBot can access your site. Get a full report on all Anthropic crawler access.
No credit card required