What is ClaudeBot?
ClaudeBot is one of three AI crawlers Anthropic lists. Anthropic says ClaudeBot collects "web content that could potentially contribute to their training", meaning the training of its AI models (Anthropic: Does Anthropic crawl data from the web?, dated 7 April 2026, read 1 November 2026).
Not to be confused with Claude itself. Many people search "claude bot" for the chatbot. ClaudeBot is the crawler.
What are Anthropic's three bots?
| Bot | What it does | If you block it |
|---|---|---|
| ClaudeBot | Collects web content that could be used for model training | Your site's future content should be left out of training data |
| Claude-User | Fetches pages when a person asks Claude a question | Claude cannot fetch your page for a user, which may reduce visibility in user-directed web search |
| Claude-SearchBot | Crawls to improve the quality of search results | Your content is not indexed for search, which may reduce visibility in search results |
Source: Anthropic help center, dated 7 April 2026, read 1 November 2026. Wording paraphrased.
Anthropic writes that restricting ClaudeBot "signals that the site's future materials should be excluded" from its training datasets. Note the word future: the page does not say blocking removes content collected before.
Why does ClaudeBot matter for B2B SaaS?
It is easy to block ClaudeBot and think you have blocked Claude. You have not. Claude's search and its fetches for users run on the other two bots. If you want Claude to find and cite your pages, the setting that matters most is Claude-SearchBot and Claude-User. ClaudeBot is the training choice.
Whether you allow training is a business decision. No source we read shows how training data changes later brand mentions.
How do you block or limit ClaudeBot?
Anthropic says its bots honor "industry standard directives in robots.txt". Put the rules in the robots.txt file in your top-level directory, and do it for each subdomain you want to opt out.
- To block training: User-agent: ClaudeBot, then Disallow: /
- To slow it down instead: User-agent: ClaudeBot, then Crawl-delay: 1
Crawl-delay is not part of the robots.txt standard, and Anthropic calls it "non-standard". Anthropic supports it; other crawlers may not.
Anthropic advises against blocking by IP address instead of using robots.txt. It says IP blocking may not work reliably, because it can stop its bots from reading your robots.txt. It publishes its crawler IP addresses at claude.com/crawling/bots.json, so you can check that a visit is real.
A hypothetical example
A hypothetical security software company wants its docs kept out of model training but wants Claude to cite its public product pages. Its robots.txt blocks ClaudeBot for the whole site and allows Claude-SearchBot and Claude-User. Claude can still fetch and cite the product pages. Whether it does is up to Claude.
How does ClaudeBot relate to third-party lists and citations?
Your robots.txt covers only your site. Other sites that review or list your product set their own rules, and Claude can read them if they allow it. Getting named on those pages is the work the LLM SEO guide covers.