How to Configure robots.txt for AI Crawlers
- Harold Bell
- Aug 9
- 6 min read
Updated: 4 days ago
Key Takeaways
|
Your robots.txt is doing something to AI crawlers right now, whether or not anyone decided what. In more than 16 years of building content programs for enterprise technology brands, the single most common silent visibility leak I find is a robots.txt that somebody inherited, nobody reviewed, and everybody assumed was fine.
With that said, read below to get the scoop on robots txt AI crawlers, the three configurations worth choosing between, and the mistakes that cost citations without ever generating an error.
How does robots.txt work for AI crawlers
Exactly the same way it works for search crawlers. You name a user agent and grant or deny paths. The difference is that AI vendors run multiple bots with different jobs, so a single rule per vendor is almost always the wrong shape. |
What compliance actually means
That's the first thing separating it from llms.txt, which nobody has committed to honoring. Every major AI operator states its crawlers honor robots.txt. Compliance is mixed for CCBot, Bytespider, and Diffbot.
And because robots.txt is a request rather than an enforcement mechanism, anything you genuinely need stopped needs server level middleware returning a 403 alongside the directive. Treat the file as a statement of intent that cooperative systems respect, not as a wall. If the distinction matters to your legal or security posture, build the wall too.
The user triggered exception
One mechanical note that trips people up. User triggered fetchers, the bots that fire when a live person browses to your page from inside an assistant, may not follow robots.txt strictly. The reasoning is that a human initiated the request, so it resembles a browser visit more than a crawl.
Design your rules knowing that. If a page must never be fetched by anything, robots.txt is not the mechanism, and authentication is.
What are the three configurations worth choosing between
Open, retrieval only, and locked. Almost every B2B technology company should be running the first, some publishers should run the second, and the third is rarer than the discourse suggests.
Open
Allow everything, restrict only genuinely private paths. This is what I recommend for most B2B technology companies. Your training contribution costs you little, retrieval access is what earns citations, and there's no upside to being selectively invisible.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: CCBot
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /account/
Retrieval only
Decline training, stay fully citable. It declines GPTBot, ClaudeBot, and the other collectors while leaving every search bot open. This is the publisher position, and it's coherent whenever your archive has independent commercial value you'd rather license than donate.
# Training collectors declined
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
Disallow: /
# Retrieval and user fetches allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
Worth noting what this configuration cannot do. Google-Extended blocks Gemini and Vertex AI training, but Googlebot still crawls you for search and still feeds AI Overviews. There is no retrieval only posture available on the Google side, and pretending otherwise is the most common error in this configuration.
Locked
Block everything AI related. Understand what you're buying. You are choosing to be absent from ChatGPT, Claude, and Perplexity answers, and you're accepting that a competitor will occupy the space where your brand would have appeared.
You'll still be indexed by Google and Bing, since those are separate crawlers governed by separate rules, but being indexed and being citable are different questions. There are businesses for which this is correct. Very few of them are B2B software companies trying to grow.
What are the mistakes that cost citations
Blocking by vendor rather than by bot. A rule catching every user agent containing openai or claude takes the search bots with it and removes you from those engines entirely.
Inheriting a block you never chose. Many hosting platforms and content management systems ship AI crawler blocks by default. Read your current file before you write a new one.
A wildcard rule higher in the file overriding your specific rules. Order and specificity both matter.
Believing Google-Extended controls AI Overviews. It controls Gemini and Vertex AI training. AI Overviews and AI Mode both run through Googlebot.
Blocking a bot to solve a server load problem. That's an infrastructure issue and crawl delay or rate limiting is the right tool.
Never checking whether it worked. Ship the file, then read your server logs and confirm the bots you allowed are arriving and the ones you declined stopped.
Why the vendor block is the one that keeps happening
It's worth understanding the mechanism, because knowing it is how you stop repeating it. Teams don't decide to leave ChatGPT search. They decide to decline AI training, which is a reasonable position, and then express that decision at the wrong level of granularity.
Somebody writes a rule aimed at a company rather than a job. The company runs three bots. Two of them were the ones earning citations. Nothing errors, nothing alerts, and the traffic that never arrives leaves no trace to investigate. Six months later the conversation is about content quality when it should be about a text file.
How do you verify your configuration is live
Fetch yourdomain.com/robots.txt directly and read it as served rather than as authored, then check your server logs over the following two weeks to confirm bot behaviour matches your rules. |
Reading the served file
As served is the operative phrase. Platform level defaults can be appended or prepended in ways that aren't visible from the editor, so what you typed and what a crawler receives are two different artifacts until you've confirmed otherwise.
On Wix specifically, robots.txt is editable through the SEO settings. Fetch the live URL afterward and read the whole thing top to bottom, paying attention to anything above your rules that might claim precedence.
Closing the loop on the outcome
Then measure the outcome rather than the file. If the point of the change was citation eligibility, your AI visibility tracking should show movement within weeks on the retrieval side.
If nothing moves, the constraint was never robots.txt, and you've cheaply ruled out a variable. That's a genuinely useful result. Most AI visibility work is elimination before it's optimisation, and access is the cheapest thing on the list to eliminate.
What's next
We audit robots.txt as the first step of every AI visibility engagement, because it's the cheapest place to find a problem. If you want a read on what yours is doing, that's twenty minutes.
Book a 30 minute working session and we'll look at your setup together.
Frequently asked questions
Should I allow or block AI crawlers in robots.txt
Allow the retrieval and user triggered bots without hesitation, since blocking them removes you from those engines' answers. Treat training collectors as a genuine policy question. Most B2B technology companies should allow everything.
Does robots.txt actually stop AI crawlers
For the major operators, yes, they state they honor it. It's a request rather than enforcement, so CCBot, Bytespider, and Diffbot have mixed records, and server level blocking is the only guaranteed stop.
Can I block AI Overviews in robots.txt
No. AI Overviews and AI Mode run through Googlebot, the same crawler that powers Google Search, so blocking one blocks the other. Google-Extended only controls Gemini and Vertex AI training.
What's the minimum robots.txt for AI visibility
Explicitly allow OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, and Perplexity-User, and confirm no wildcard rule above them is blocking those paths. That covers the retrieval layer that decides citations.
Do I need to list every AI bot
No, but naming them explicitly protects you from wildcard rules and makes the file a record of a decision rather than a default. New AI bots appear a few times a year, so review it quarterly.
Comments