robots.txt Guide + Checker

ClaudeBot & robots.txt

Allow or Block Anthropic's Crawler

Check if ClaudeBot can access your site, and learn exactly how to configure your robots.txt for Anthropic's crawlers.

Last verified against Anthropic's crawler help article: 2026-10-18

ClaudeBot: the facts

Every fact below is taken from Anthropic's crawler help article(dated April 7, 2026) and the IP list it links, last verified 2026-10-18.

Operator

Anthropic

robots.txt tokens

ClaudeBot · Claude-SearchBot · Claude-User

ClaudeBot

Collects web content that could contribute to training Anthropic’s generative AI models. Restricting it signals that your future materials should be excluded from training datasets

Claude-SearchBot

Navigates the web to improve search result quality. Anthropic says disabling it prevents indexing your content for search, which may reduce your visibility and accuracy in Claude search results

Claude-User

Accesses websites when a person asks Claude a question. Anthropic says disabling it may reduce your visibility for user-directed web search

Full user-agent strings

Not published. Anthropic’s article names the three robot tokens but gives no full user-agent string

Published IP list

https://claude.com/crawling/bots.json

Honours robots.txt?

Anthropic states its bots honour industry-standard robots.txt directives and support the non-standard Crawl-delay extension

Block by IP?

Anthropic warns that blocking IPs may not work and impedes its ability to read your robots.txt; it asks for a robots.txt rule on every subdomain

What is ClaudeBot?

ClaudeBot is Anthropic's official web crawler. It collects content used for training Claude AI models. ClaudeBot respects robots.txt rules and identifies itself with the user-agent string ClaudeBot.

Anthropic also operates Claude-SearchBot (builds Claude's search index — the crawler that decides whether Claude can cite you) and Claude-User (fetches a page when a user asks Claude to open it). Each can be controlled independently in robots.txt.

Note: the older Claude-Web and anthropic-ai tokens are not in Anthropic's current help article, which names only ClaudeBot, Claude-SearchBot and Claude-User. BotView no longer tracks them, and rules for them are unlikely to match a current crawler.

robots.txt Syntax for ClaudeBot

Copy-paste these examples into your robots.txt file.

Allow ClaudeBot (Recommended for most sites)

# Allow Anthropic crawlers

User-agent: Claude-SearchBot

Allow: /

User-agent: ClaudeBot

Allow: /

User-agent: Claude-User

Allow: /

This allows all Anthropic crawlers full access to your site.

Block ClaudeBot

# Block Anthropic crawlers

User-agent: Claude-SearchBot

Disallow: /

User-agent: ClaudeBot

Disallow: /

User-agent: Claude-User

Disallow: /

This blocks all Anthropic crawlers. Blocking Claude-SearchBot removes you from Claude search; blocking only ClaudeBot is a training opt-out that keeps you visible in Claude.

Partial Access (Allow some paths)

# Allow ClaudeBot but block private areas

User-agent: ClaudeBot

Allow: /blog/

Allow: /docs/

Disallow: /admin/

Disallow: /private/

Disallow: /api/

When to Allow vs Block ClaudeBot

Allow ClaudeBot When...

✓You want Claude to cite your content
✓You serve enterprise or B2B audiences
✓You publish technical docs or knowledge bases
✓You want visibility in Claude-powered tools

Block ClaudeBot When...

✗Your content is behind a paywall
✗You don't want AI training on your content
✗You have content licensing restrictions
✗Your content is proprietary or confidential

Frequently Asked Questions

What are Anthropic's crawler user agent strings?

Anthropic's help article names three robots: 'ClaudeBot' (collects content that could contribute to model training), 'Claude-SearchBot' (improves search result quality) and 'Claude-User' (fetches a page when a person asks Claude a question). Each has its own robots.txt token. The article does not publish a full user-agent string for any of them, so match on the token. Source: Anthropic help article dated April 7, 2026, verified 2026-10-18. The old 'Claude-Web' and 'anthropic-ai' tokens are not in that article.

How do I verify a request is really from an Anthropic crawler?

Anthropic's article says a request from a source IP on its published list, https://claude.com/crawling/bots.json, comes from Anthropic. It also warns that blocking by IP may not reliably opt you out and that a robots.txt rule is the supported control.

What's the difference between ClaudeBot and Claude-SearchBot?

ClaudeBot collects training data for Claude models — blocking it is a policy choice that keeps your content out of training. Claude-SearchBot builds the index behind Claude's web search — blocking it removes your site from Claude's search answers and citations. If you care about being visible in Claude, Claude-SearchBot is the one that matters.

Does blocking ClaudeBot remove my site from Claude's answers?

No. Blocking ClaudeBot only keeps your content out of future model training. Claude's live search answers come from the index built by Claude-SearchBot — as long as that's allowed, Claude can still find and cite your site. Block Claude-SearchBot and you disappear from Claude search regardless of your ClaudeBot setting.

Should I still have rules for Claude-Web or anthropic-ai?

Probably not. Anthropic's current crawler help article (dated April 7, 2026; checked 2026-10-18) names only ClaudeBot, Claude-SearchBot and Claude-User and does not mention Claude-Web or anthropic-ai. BotView no longer tracks them. Rules for them are unlikely to match any current Anthropic crawler, so you can delete them unless you have a reason to keep them.

How do I verify my ClaudeBot robots.txt changes are working?

Use BotView's free scanner to check your site. It analyzes your robots.txt for all Anthropic crawler identifiers and shows you exactly what's allowed or blocked. It also checks whether your content is actually accessible beyond just the robots.txt rules.

Related robots.txt Guides

AI robots.txt checker — See which of 18 AI crawlers your live robots.txt blocks
GPTBot & robots.txt — OpenAI / ChatGPT
PerplexityBot & robots.txt — Perplexity AI
Google-Extended & robots.txt — Google / Gemini
CCBot & robots.txt — Common Crawl
Check your Claude visibility

Check Your ClaudeBot Configuration

See if ClaudeBot can access your site. Get a full report on all Anthropic crawler access.

No credit card required