ClaudeBot & robots.txt
Allow or Block Anthropic's Crawler
Check if ClaudeBot can access your site, and learn exactly how to configure your robots.txt for Anthropic's crawlers.
What is ClaudeBot?
ClaudeBot is Anthropic's official web crawler. It collects content used for training Claude AI models. ClaudeBot respects robots.txt rules and identifies itself with the user-agent string ClaudeBot.
Anthropic also operates Claude-Web (for real-time web browsing) and anthropic-ai (a general Anthropic crawler identifier). Each can be controlled independently in robots.txt.
robots.txt Syntax for ClaudeBot
Copy-paste these examples into your robots.txt file.
Allow ClaudeBot (Recommended for most sites)
# Allow Anthropic crawlers
User-agent: ClaudeBot
Allow: /
User-agent: Claude-Web
Allow: /
User-agent: anthropic-ai
Allow: /
This allows all Anthropic crawlers full access to your site.
Block ClaudeBot
# Block Anthropic crawlers
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: anthropic-ai
Disallow: /
This blocks all Anthropic crawlers from your entire site.
Partial Access (Allow some paths)
# Allow ClaudeBot but block private areas
User-agent: ClaudeBot
Allow: /blog/
Allow: /docs/
Disallow: /admin/
Disallow: /private/
Disallow: /api/
When to Allow vs Block ClaudeBot
Allow ClaudeBot When...
- ✓You want Claude to cite your content
- ✓You serve enterprise or B2B audiences
- ✓You publish technical docs or knowledge bases
- ✓You want visibility in Claude-powered tools
Block ClaudeBot When...
- ✗Your content is behind a paywall
- ✗You don't want AI training on your content
- ✗You have content licensing restrictions
- ✗Your content is proprietary or confidential
Frequently Asked Questions
What are Anthropic's crawler user agent strings?
Anthropic uses three identifiers: 'ClaudeBot' (primary training data crawler), 'Claude-Web' (real-time web browsing), and 'anthropic-ai' (general Anthropic crawler identifier). You can control each independently in robots.txt.
What's the difference between ClaudeBot and Claude-Web?
ClaudeBot crawls websites to collect training data for Claude AI models — this determines what Claude 'knows.' Claude-Web is used when a user asks Claude to browse a URL in real-time. Blocking ClaudeBot affects Claude's training data; blocking Claude-Web prevents live browsing.
Does blocking ClaudeBot remove my site from Claude's knowledge?
Blocking ClaudeBot prevents future crawling, but Claude may still have information from before the block was added. Over time, this information becomes less accurate. If you want Claude to have current information about your site, allow ClaudeBot.
Should I block anthropic-ai in addition to ClaudeBot?
If you want to block all Anthropic crawlers, yes — add rules for ClaudeBot, Claude-Web, and anthropic-ai. The anthropic-ai identifier is a catch-all. If you only block ClaudeBot, other Anthropic crawlers might still access your site.
How do I verify my ClaudeBot robots.txt changes are working?
Use BotView's free scanner to check your site. It analyzes your robots.txt for all Anthropic crawler identifiers and shows you exactly what's allowed or blocked. It also checks whether your content is actually accessible beyond just the robots.txt rules.
Related robots.txt Guides
- GPTBot & robots.txt — OpenAI / ChatGPT
- PerplexityBot & robots.txt — Perplexity AI
- Google-Extended & robots.txt — Google / Gemini
- CCBot & robots.txt — Common Crawl
- Check your Claude visibility
Check Your ClaudeBot Configuration
See if ClaudeBot can access your site. Get a full report on all Anthropic crawler access.
No credit card required