ClaudeBot: What It Is and What It Means for Your Site
- Harold Bell
- Aug 9
- 5 min read
Updated: Aug 16
Key Takeaways
|
ClaudeBot tends to arrive in a conversation two ways. Either someone spotted it in the logs and wants to know if it's a threat, or someone noticed Claude describing a competitor accurately and them not at all.
Both are reasonable starting points, and they need different answers. In more than 16 years of building content programs for enterprise technology brands, I've found the second question is almost always the more valuable one, and it's almost never the one people ask first.
What is ClaudeBot
ClaudeBot is Anthropic's training crawler. It collects web content for model training. Anthropic runs two other agents alongside it, Claude-SearchBot for search indexing and Claude-User for fetches triggered by a live person in a Claude conversation. |
The same three way split
If that structure sounds familiar, it should. It maps exactly onto OpenAI's split between GPTBot, OAI-SearchBot, and ChatGPT-User. The vendors picked different names for the same three jobs, which is most of why this territory confuses people.
Once you see the pattern, every new AI vendor becomes a lookup rather than a research project. Ask which of their bots trains, which indexes, and which fetches on demand. The answer determines everything you'd want to do about them.
The legacy tokens still in your robots.txt
There are two older tokens worth knowing because they're still sitting in a lot of files. anthropic-ai was used for bulk model training and claude-web for web focused crawling.
If your robots.txt references those and nothing else, it's likely been untouched for a couple of years and isn't governing the bots actually visiting you. That's a common state and it's worth ten minutes to fix, because a rule aimed at a retired user agent is a rule that does nothing.
Which Anthropic crawler affects whether Claude cites you
Claude-SearchBot and Claude-User. Those two decide whether your pages can be retrieved and quoted in a Claude answer. ClaudeBot only affects what future models absorb during training. |
The Brave dependency
There's an extra wrinkle on Anthropic's side that doesn't apply to OpenAI. Claude's live web answers also lean on the Brave search index, which means Brave visibility feeds Claude citations indirectly.
That's a second retrieval path most B2B teams have never audited, and it behaves like the Bing relationship that decides ChatGPT citations. The practical consequence is that a Claude specific absence routes to a different diagnosis than a ChatGPT absence does, and treating them as one problem wastes a cycle.
The three input citation stack
So Claude citations depend on three things stacking. Your pages being reachable by Claude-SearchBot. Your pages being present in Brave's index. And your content being structured so a retrieved chunk actually answers something on its own.
The third one is where most of the work is, and it's the only one that isn't a binary check. The first two you can confirm in an afternoon. The third is a content discipline you either have or don't.
Should you block ClaudeBot
Same calculus as any training collector, and the same two defensible answers. If your archive has independent commercial value and you'd rather license it than contribute it, blocking training while keeping retrieval open is coherent.
If you're a B2B technology company, the marginal training contribution costs you little and the absence from model memory costs you something you can't easily see. That's the memory half of large language model optimization, and it moves slowly enough that you won't notice the loss until it's years old.
The blanket block mistake
What I'd push back on is blocking Anthropic wholesale. A rule that disallows every user agent containing the word claude takes out Claude-SearchBot and Claude-User alongside ClaudeBot, and removes you from Claude answers entirely.
That's a much bigger decision than the one most teams think they're making, and nothing will tell you it happened. Write it per bot.
# Decline training, stay citable
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
Why is ClaudeBot hitting my site so often
High request volume from any AI crawler is a crawl efficiency signal, not an attack. It usually traces to a site structure that generates many low value URLs, such as faceted filters, calendar pages, or parameterised variants. |
Diagnosing the volume
Before you treat it as a problem, look at what's being requested rather than how much. Pull the top requested paths for that user agent from your server logs and see whether they're pages you'd want indexed.
If the answer is a few hundred filter permutations of the same listing page, the bot is behaving reasonably and your URL structure is generating the work. If the answer is your actual content, the volume is doing you a favour.
Fixing it properly
The fix is the same fix that helps every AI crawler and every search engine. Tighten your internal linking so the crawl path leads to pages that matter, disallow the parameter patterns that generate infinite variants, and make sure your sitemap reflects your canonical set rather than everything your CMS can render.
If volume is genuinely disruptive, crawl delay directives and rate limiting at the server level are the tools. Blocking the bot outright to solve a load problem is using a permanent policy decision to answer a temporary infrastructure question, and you'll forget you did it.
How do you verify ClaudeBot is genuine
Anthropic publishes IP ranges for its crawlers. Check the request IP against the published range rather than trusting the user agent string, since impersonating well known AI bots is common and getting more so as allowlisting spreads.
Your server logs are where this happens, and it's worth building the habit of checking user agents against ranges as routine rather than as incident response.
Knowing which bots reach you, at what frequency, and which pages they favour is a better AI visibility diagnostic than most of the tools sold for the purpose. It's also free, which is more than the tools are.
What's next
Most teams have never audited whether they're reachable by Anthropic's retrieval bots or present in Brave's index. Both are quick checks and both gate Claude citations.
Book a 30 minute working session and we'll look at your setup together.
Frequently asked questions
Is ClaudeBot safe to allow
Yes. ClaudeBot is a standard web crawler that respects robots.txt and collects content for model training. It doesn't execute anything on your site or access anything a normal visitor couldn't reach.
What's the difference between ClaudeBot and Claude-SearchBot
ClaudeBot collects training data that shapes future models. Claude-SearchBot builds the retrieval index Claude uses to answer questions with live web sources. Only the second one affects whether Claude can cite you today.
Does blocking ClaudeBot stop Claude from mentioning my brand
Not directly. Claude can still retrieve and cite your pages through Claude-SearchBot and Claude-User. Blocking ClaudeBot only affects what future models learn during training.
What are anthropic-ai and claude-web
Older Anthropic user agent tokens, anthropic-ai for bulk training and claude-web for web focused crawling. They still appear in older robots.txt files. If yours only references these, it's out of date.
Does Claude use Bing like ChatGPT does
No. Claude's live web answers lean on the Brave search index rather than Bing. That makes Brave visibility a distinct retrieval path worth auditing separately from your Bing work.
Comments