AI Crawlers: Understanding the Bots Visiting Your Site
- Harold Bell
- Aug 9
- 7 min read
Updated: Aug 16
Key Takeaways
|
Somebody on your team pulls the server logs, sees ClaudeBot, GPTBot, OAI-SearchBot and PerplexityBot in the same afternoon, and asks the obvious question. Should we be letting all of these in?
It's a good question with a bad default answer. In more than 16 years of building content programs for enterprise technology brands, I've watched teams block AI bots as a reflex and then spend the next two quarters wondering why they never show up in ChatGPT. The bots are not interchangeable, and the decision is not one decision.
There's one organizing idea that makes all of this manageable, and once you have it the rest is a lookup table.
What are AI crawlers
AI crawlers are automated agents that language model companies send across the web to collect training data, build retrieval indexes, and fetch pages in real time when a user asks a question. Unlike search crawlers, which have done one job for decades, AI crawlers split into three distinct roles and are multiplying fast. |
That three way split is the only piece of this you need to internalize. Everything else is a naming table you can look up, and the names change more often than the roles do. Major new AI bot user agents appear two to four times a year, usually when a large AI company launches a product or changes its crawling strategy.
How they differ from search crawlers
Googlebot and Bingbot have done one thing for twenty five years. They fetch pages, render them, and decide whether to store them in an index that serves ranked results. One bot, one purpose, one decision for you to make about it.
AI vendors split that work across multiple agents because the jobs genuinely diverge. Collecting a training corpus, maintaining a live retrieval index, and fetching one page because a person just asked about it are three different operations with three different schedules and three different implications for you. A single allow or deny rule per vendor cannot express a sensible position on all three.
What are the three jobs an AI crawler does
Training collection, retrieval indexing, and user triggered fetching. The job decides the stakes. Blocking a retrieval crawler makes you invisible to that engine's answers. Blocking a training collector is a policy choice with far less visibility cost. |
Training collectors
These gather content that shapes what a future model knows. Their visits produce no citations today and no measurable effect this quarter. They influence what next year's model recalls without looking anything up, which is the territory large language model optimization deals with.
There's no dashboard, no report, and no feedback loop. A page collected this quarter may influence what a model says about your category two model generations from now, or it may be diluted into nothing. That invisibility is exactly why the decision here is a policy question rather than a performance one.
Retrieval crawlers
These build the search indexes AI answers get assembled from, and they're the ones that matter most for citations. If a retrieval crawler can't reach your page, that page cannot be retrieved, which means it cannot be cited, no matter how well written it is.
This is a binary gate sitting in front of everything else you might do. I've audited programs where months of content restructuring produced nothing, and the cause was a retrieval bot that had been denied access in a robots.txt nobody had opened since 2023.
User triggered fetchers
These act on behalf of a live person. Someone in ChatGPT opens your page, or an answer needs your current content, and the fetcher goes and gets it. These are the closest thing to a human visit in the whole category.
Two things follow. Several providers note that user triggered fetchers may not follow robots.txt strictly, on the reasoning that a person initiated the request. And their presence in your logs is a demand signal worth tracking on its own, because it means real buyers are handing your pages to assistants right now.
Which AI crawlers should you know by name
OpenAI runs GPTBot for training, OAI-SearchBot for its search index, and ChatGPT-User for live fetches. Anthropic runs ClaudeBot, Claude-SearchBot, and Claude-User. Perplexity runs PerplexityBot and Perplexity-User. Google routes AI Overviews and AI Mode through Googlebot, with Google-Extended as a training opt out token. |
OpenAI. GPTBot collects training data. OAI-SearchBot populates the index behind ChatGPT search. ChatGPT-User fires when a person browses to your page from ChatGPT. The pair that controls citations is OAI-SearchBot plus ChatGPT-User. GPTBot only affects training.
Anthropic. ClaudeBot for training, Claude-SearchBot for search indexing, Claude-User for user triggered fetches. Legacy tokens anthropic-ai and claude-web still appear in older robots.txt files.
Perplexity. PerplexityBot builds the index, Perplexity-User handles live fetches for a specific answer.
Google. Googlebot does classic search and also feeds AI Overviews and AI Mode. Google-Extended is a robots.txt token controlling Gemini and Vertex AI training, not a separate crawler.
Apple. Applebot crawls, Applebot-Extended is the training control token.
Meta. Meta-ExternalAgent collects, Meta-ExternalFetcher handles user triggered requests.
Others worth recognizing. CCBot for Common Crawl, Amazonbot, Bytespider from ByteDance, cohere-ai, Diffbot, AI2Bot, DuckAssistBot.
Which of these actually affect whether you get cited
The retrieval and user triggered layers. Blocking OAI-SearchBot, Claude-SearchBot, PerplexityBot, or Bingbot removes you from those engines' answers. Blocking training collectors like GPTBot, ClaudeBot, or CCBot is a policy choice with far less visibility cost. |
The mistake that costs the most
This distinction is where most of the damage happens, and the mechanism is almost always the same. The training bots are the ones with recognisable names and the ones that generate publisher outrage, so they're the ones teams decide to block.
Then somebody writes a rule that catches every user agent containing openai, or claude, or the vendor name. The search bot goes down with the training bot. The team believes it declined to donate training data. What it actually did was remove itself from ChatGPT search, and nothing in any dashboard will tell it so.
The Googlebot trap
There's a trap on the Google side worth stating plainly, because it comes up in nearly every conversation about AI Overviews.
You can't opt out of AI Overviews without leaving Google Search. Both run through Googlebot. Google-Extended only controls training for Gemini and Vertex AI, and it is not a crawler you can watch in your logs. Anyone who tells you they blocked AI Overviews and kept their rankings has done one of those two things and not the other.
How do you tell a real AI crawler from a fake one
Verify by IP range, not by user agent string. User agent strings are trivially spoofed. OpenAI, Anthropic, and Perplexity all publish the IP ranges their crawlers operate from, so reverse lookups against those ranges are the reliable check. |
Why spoofing has become common
This matters more than it used to, and the reason is a straightforward incentive. Site owners started allowlisting recognisable AI bot names, which created a category of user agent string that gets waved through. Scrapers noticed.
If you're making access decisions based on user agent alone, you're making them on a field anyone can type. Pull the distinct IPs behind each user agent in your server logs and check them against the current published ranges rather than a copy you saved, because the ranges change.
Which bots actually honour robots.txt
Compliance varies and it's worth knowing where. The major operators, OpenAI, Anthropic, Google, Apple, and Perplexity, all state their crawlers respect robots.txt. CCBot, Bytespider, and Diffbot have mixed records.
Since robots.txt is a request rather than an enforcement mechanism, anything you genuinely need stopped needs server level middleware returning a 403 alongside the directive. Treat robots.txt as a statement of intent that cooperative bots honour, not as a wall.
What should most B2B sites actually do
For almost every B2B technology company I work with, the answer is to allow the retrieval layer without hesitation, allow the user triggered fetchers, and treat training collectors as a genuine policy question with two defensible answers.
The case on each side of training
Publishers with licensing leverage increasingly block training while keeping retrieval open, and that's coherent when your content is the product. If your archive has independent commercial value, you'd rather sell access than donate it, and the visibility cost of declining is genuinely small.
Most B2B companies conclude the opposite, and I think correctly. Your marginal contribution to a training corpus costs you approximately nothing. Being absent from what tomorrow's model recalls about your category costs you something real, and it's the kind of cost you can't detect until it's already years old.
Check what you already have first
The step people skip is auditing the current state before writing anything new. Plenty of hosting platforms and content management systems ship AI crawler blocks by default, which means teams are frequently blocking bots they never decided to block, and have been for as long as the site has existed.
Read your current robots.txt before you write a new one, and run the same review over your llms.txt if you maintain one. Then confirm the change landed by watching your logs, because a wildcard rule higher in the file can silently override everything you wrote below it.
What's next
If you're not certain what your robots.txt is currently doing to AI crawlers, that's worth twenty minutes. We audit this as a matter of routine for B2B technology clients, and it's the single most common silent visibility leak we find.
Book a 30 minute working session and we'll look at your setup together.
Frequently asked questions
Should I block AI crawlers
Not the retrieval ones. Blocking OAI-SearchBot, Claude-SearchBot, or PerplexityBot removes you from those engines' answers entirely. Blocking training collectors such as GPTBot or ClaudeBot is a separate policy decision with a much smaller visibility cost.
Does blocking GPTBot stop ChatGPT from citing me
No. GPTBot collects training data. ChatGPT search citations run through OAI-SearchBot and ChatGPT-User. Blocking GPTBot alone leaves your citation path intact.
Can I block AI Overviews without losing Google rankings
No. AI Overviews and AI Mode both run through Googlebot, the same crawler that powers Google Search. Google-Extended only opts you out of Gemini and Vertex AI training.
Do AI crawlers respect robots.txt
The major operators state that they do. Compliance is mixed for CCBot, Bytespider, and Diffbot. Since robots.txt is a request rather than an enforcement mechanism, server level blocking is the only reliable stop.
How do I see which AI crawlers are hitting my site
Server logs, filtered by user agent and then verified against published IP ranges. Bing Webmaster Tools also reports Bingbot crawl activity, which matters because Bing's index feeds ChatGPT search.
Comments