What is GPTBot? What Blocking it Actually Costs.
- Harold Bell
- Aug 9
- 5 min read
Updated: Aug 16
Key Takeaways
|
GPTBot is the AI crawler most people can name, which makes it the one most people block. That's an unfortunate pairing, because it's also the OpenAI bot with the least to do with whether ChatGPT recommends you.
I've spent more than 16 years building content programs for enterprise technology brands, and this specific confusion has cost more clients more visibility than any other single technical decision I audit. It's a naming problem that turned into a strategy problem.
What is GPTBot
GPTBot is OpenAI's training crawler. It collects web content that may be used to train future models. It does not build the index behind ChatGPT search, and it does not fetch pages when a user asks a question. |
The three OpenAI agents
OpenAI separated its crawlers in late 2024 into three distinct agents, and the separation is the important part. GPTBot handles training collection. OAI-SearchBot populates the search index that ChatGPT retrieves from. ChatGPT-User fires when a person in ChatGPT opens or browses your page.
Three bots, three jobs, one company. If you treat them as one thing, every decision you make about them is wrong in at least one direction, and the direction you'll be wrong in is the expensive one.
Why OpenAI split them
The split happened under pressure from publishers who wanted to decline training without disappearing from a product their readers were using. That history is worth knowing, because it tells you the separation is deliberate and is meant to be used.
OpenAI built you a control surface with three switches. Most teams flip all three at once and then wonder why the outcome doesn't match the intent.
What does GPTBot actually do with your content
It contributes to the corpus a future model learns from. That's a slow, diffuse, and entirely invisible process. There's no report, no dashboard, and no feedback loop.
The memory layer, not the retrieval layer
This is the memory layer, and it's the territory large language model optimization actually operates in. Memory responds across model generations, which means quarters or longer. Retrieval responds in weeks.
Most people who think they're optimising for GPTBot are actually trying to influence retrieval, which GPTBot has nothing to do with. The tell is the timeframe in the question. If someone asks how long until we see results, they mean retrieval, and they should be looking at a different bot entirely.
Why you can't measure it
A page GPTBot collected this quarter may influence what a model recalls about your category two model generations from now, or it may be diluted into nothing by a corpus measured in trillions of tokens.
There is no experiment you can run that isolates this. Nobody publishes a control group. That's uncomfortable, and it's why I'd push you to treat the training question as a policy decision with values behind it rather than a performance decision with numbers behind it. The numbers don't exist.
Should you block GPTBot
It's a genuine policy decision with two defensible answers, and a much smaller visibility cost than most people assume. Blocking GPTBot removes your content from future training corpora. It does not affect ChatGPT search citations, which run through OAI-SearchBot and ChatGPT-User. |
The case for blocking
Strongest when your content is your product. Publishers with licensing leverage increasingly block training collectors while keeping retrieval bots open, because they'd rather sell access than donate it.
If you're a research firm, a media company, or anyone whose archive has independent commercial value, that's a coherent position and the visibility cost of taking it is genuinely small. You stay citable. You just stop contributing for free.
The case for allowing
Strongest for B2B technology companies, which is most of who I work with. Your marginal contribution to a training corpus costs you approximately nothing, since you're one voice among millions and you weren't going to license it to anyone anyway.
Being absent from what tomorrow's model recalls about your category costs you something real. Anthropic's ClaudeBot presents the identical decision under a different name, so whatever you conclude here should apply there too. And it's the kind of cost you can't detect until it's already three years old, which is the worst property a cost can have.
What isn't defensible
Blocking GPTBot while believing you've protected yourself from something you haven't. Or blocking it as part of a blanket OpenAI block that takes the search bot with it, which is the version I find most often.
If ChatGPT isn't citing you, Bing indexing is a far more likely culprit than anything GPTBot did or didn't collect. Rule that out before you touch your training policy, because the two problems have nothing to do with each other and only one of them is fixable this quarter.
How do you configure GPTBot access correctly
Name the bots individually. Never block by vendor.
# Allow ChatGPT search citations, decline training
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: GPTBot
Disallow: /
# Or allow everything, which most B2B sites should
User-agent: GPTBot
Allow: /
Disallow: /admin/
Disallow: /api/
Two checks after you ship
Confirm you haven't inherited a block you never chose. Plenty of hosting platforms and content management systems ship AI crawler blocks by default, and a robots.txt you've never opened has been making this decision for you since launch.
Confirm a wildcard rule higher in the file isn't overriding what you just wrote. A Disallow under User-agent star can silently outrank the specific rules below it depending on how a given crawler resolves precedence. Fetch the served file and read it top to bottom rather than trusting the editor.
How do you verify GPTBot is real
Check the request IP against OpenAI's published crawler IP ranges. User agent strings are trivially spoofed, and impersonating recognisable AI bots has become common precisely because site owners started allowlisting them. |
Your server logs are the place this lives. Filter for the GPTBot user agent, then reverse lookup the IPs against the published ranges. Anything claiming to be GPTBot from outside those ranges is something else wearing the name.
That finding changes what you do about the volume. A real crawler hitting you hard is a crawl efficiency conversation. A scraper wearing a trusted name is a different conversation entirely, and it's one your infrastructure team will want to have.
What's next
The most common finding in our technical audits is a robots.txt blocking a bot nobody remembers deciding to block. If you want a read on what yours is currently doing, that's a short session.
Book a 30 minute working session at cal.com/mqlmagnet/30min and we'll look at your setup together.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT
No. GPTBot only collects training data. ChatGPT search retrieves through OAI-SearchBot and fetches live pages through ChatGPT-User. Blocking GPTBot leaves both of those paths intact.
Is GPTBot the same as ChatGPT-User
No. GPTBot crawls on OpenAI's schedule to collect training data. ChatGPT-User fires in response to a specific person browsing to your page from within ChatGPT, which makes it closer to a human visit than a crawl.
Does GPTBot respect robots.txt
OpenAI states that GPTBot honours robots.txt directives. Since robots.txt is a request rather than an enforcement mechanism, server level blocking is the only way to guarantee a stop.
What IP ranges does GPTBot use
OpenAI publishes its crawler IP ranges publicly and updates them. Verify against the current published list rather than a copy, since the ranges change.
Should a B2B SaaS company block GPTBot
Usually not. Your marginal contribution to a training corpus costs you little, and absence from future model memory costs you something you cannot easily measure or reverse. The calculation is different if your content is itself the product you sell.
Comments