What Is llms.txt and Does It Actually Work
- Harold Bell
- Aug 8
- 12 min read
Key Takeaways
|
Every few months a file appears that promises to fix AI visibility, and llms.txt is the current one. I've had four separate conversations this quarter that opened with a version of the same question. Should we add llms.txt, and will it get us cited by ChatGPT?
I've spent more than 16 years building content programs for enterprise technology brands, and I've watched this pattern enough times to recognise the shape of it. A technically reasonable proposal gets restated slightly wrong, the restatement spreads faster than the original, and eighteen months later half the market is doing something that never claimed to do what they think it does.
So here's the honest version, including the parts that argue against bothering. There's a real use case for this file. It just isn't the one you've been sold.
What is llms.txt
llms.txt is a plain text file at the root of your domain that lists your most important pages as Markdown links, so a language model can find your canonical content without parsing cluttered HTML. It was proposed by Jeremy Howard in 2024. It is a community convention, not a ratified standard, and it has no enforcement mechanism. |
The reasoning behind it is sound. Language models work inside finite context windows. A modern web page wraps its actual content in navigation, cookie banners, embedded widgets, and scripts, and a model that has to wade through all of that to reach three useful paragraphs wastes most of its budget on the wrapper. A curated index of clean Markdown skips the wrapper entirely.
Where the format came from
The proposal emerged from the developer tooling world rather than the marketing world, and that origin explains almost everything about how it behaves. It was designed for documentation sites being consumed by AI assistants during a coding session, where the model needs the right reference page immediately and every token spent on your header is a token not spent on the answer.
It has no backing from the W3C, the IETF, or any recognised standards body. There's no registry, no gatekeeper, and no validation step. You publish the file and hope the systems that matter choose to read it, which is the same voluntary posture robots.txt has, minus the twenty five years of universal adoption that makes robots.txt actually work.
What llms-full.txt does differently
There's a companion format worth knowing about. Where llms.txt is a table of contents, llms-full.txt holds your entire site content in a single Markdown document for deep ingestion in one fetch. It's the difference between handing someone an index and handing them the book.
Most sites only need the first one. Documentation heavy products sometimes maintain both, and that's defensible when your reference material is genuinely large and genuinely stable. What isn't defensible is shipping a second file you'll never update, because a stale full text dump is a confident, comprehensive, wrong description of your product.
The confusion between discoverability and visibility
Here's the category error that grew around the file, and it's worth stating precisely because it's the root of nearly every bad decision I see in this territory.
The original claim was that llms.txt helps AI systems locate important resources. The claim that spread was that llms.txt helps websites appear in AI results. Those two sentences look similar and they describe different things. One is about discoverability, meaning whether a system can find your content when it goes looking. The other is about visibility, meaning whether your brand shows up in an answer a buyer reads.
Discoverability is necessary for visibility and nowhere near sufficient for it. Every page in your sitemap is discoverable. Almost none of them get cited. The gap between those two states is filled by relevance, structure, authority, and corroboration, none of which a file at your domain root can supply.
Does llms.txt actually improve AI visibility
There is no evidence that it does. SE Ranking built a model to predict AI citation frequency and tested llms.txt presence as a variable. Removing the variable improved the model's accuracy, which means the file was contributing noise rather than signal. |
What the citation modeling actually found
That result deserves a moment, because it's stronger than a null finding and most coverage flattens the two together.
A variable that simply doesn't matter leaves a model's accuracy roughly unchanged when you remove it. You'd conclude the file is neutral and move on. A variable that actively degrades accuracy when included is telling you something different. Either the correlation runs the opposite direction to the one everybody assumes, or the file is confounded with some other property of the sites that adopt it.
My read is the second one. Early adopters of llms.txt skew toward documentation heavy technical companies, which is a population with unusual content structure, unusual link profiles, and unusual buyer behaviour. A model trying to learn citation patterns from that group gets a distorted signal, and the file is the visible marker of a much messier underlying difference.
Why Google's position matters here
John Mueller has been blunt about the file, comparing it to the keywords meta tag and noting that AI services haven't said they use it and don't appear to request it in server logs. The comparison is pointed. The keywords meta tag was a self declared summary of your own page that search engines eventually ignored entirely, because self declaration is worthless when everyone can self declare.
His deeper objection is the more interesting one and it's the argument I'd want anyone selling this file to answer. A model that has already downloaded your real content has no reason to trust a separate file describing that content. And anyone could list one set of pages in llms.txt while serving something different to users, which is cloaking with a new coat of paint. Any system that weighted the file heavily would immediately create an incentive to game it.
What the null result does and does not rule out
I want to be fair to the file, because the evidence has limits and the honest position acknowledges them.
What the data rules out is a direct, measurable, near term citation lift from adding llms.txt to a typical site. That claim is not supported and you should stop repeating it. What the data does not rule out is future adoption, value in agentic contexts that citation studies never measured, or benefit to a specific narrow cohort that gets averaged away in a broad panel.
As of the first quarter of 2026, no major AI company, OpenAI, Google, Anthropic, Meta, or Mistral, has publicly committed to reading or acting on llms.txt in production. GPTBot fetches the file occasionally. Fetching a file is not the same as weighting it, and the distinction gets collapsed constantly in coverage that wants the answer to be yes.
Who actually reads llms.txt today
Coding agents. Cursor, Claude Code, GitHub Copilot, Windsurf, Cline, and Aider all look for /llms.txt and /llms-full.txt when pointed at a documentation site. That's the one confirmed and repeatable consumer. |
The coding agent case
This is the part most marketing coverage buries, and it's the part that should change your decision. The file works, reliably, in the developer tooling layer, and it works for exactly the reason the original proposal described.
An engineer points their assistant at your documentation. The assistant fetches your llms.txt, reads a curated list of thirty pages with one line descriptions, and retrieves the two that matter. Without the file it crawls your docs site, burns context on navigation and sidebars, and guesses at your URL structure. The difference is measurable in tokens and in whether the engineer gets a correct answer about your product on the first try.
That's not a consolation prize. It's a real integration surface, it's live now, and nobody had to wait for a standards body to ratify anything.
What this means if you sell to developers
If your buyer evaluates software with an AI assistant open, and increasingly they do, then your documentation is being read by a machine before it's read by a person. The question stops being whether llms.txt helps you get cited and becomes whether your product is legible to the tool your buyer is using to evaluate it.
That reframing is worth more than the file itself. It's the same shift that makes large language model optimization worth taking seriously, and it applies whether or not you ever ship an llms.txt.
What does the adoption data actually say
Adoption is cohort shaped rather than a single number. The 2025 Web Almanac measured 2.13% of desktop sites, but 39.6% of those files came from a plugin default rather than a deliberate decision. Panels of AI and developer tool companies run far higher. |
The plugin stub problem
That 39.6% figure is the one to hold onto, because it quietly invalidates most adoption commentary. When roughly four in ten existing files are auto generated defaults nobody chose, headline adoption numbers are describing plugin install rates rather than strategic decisions. A stub that lists your homepage, an about page, and a contact form tells a model nothing it couldn't get from your sitemap in less time. It exists on disk and carries no information.
This matters for anyone running an experiment on the file. If you're comparing sites with llms.txt against sites without it, and four in ten of your treatment group are running stubs, you've contaminated the comparison before you started. It's another reason to treat the citation modelling result as directionally useful rather than definitive.
Why the adoption numbers contradict each other
You'll see wildly different figures quoted and they're mostly all correct, which is confusing until you notice they're measuring different populations.
A fixed panel of documentation heavy hosts measured this August put adoption above 50%. A tracker of the general top thousand domains put it under 16% of reachable roots. The Web Almanac's broad crawl put it near 2%. Same file, three populations, three answers.
The practical lesson is to ignore any adoption statistic that doesn't tell you the cohort. The only number relevant to your decision is adoption among companies that look like yours, and for most B2B technology firms that number sits somewhere between the general web and the developer tooling panel.
When is llms.txt worth shipping
My rule is simple. Ship it if the file would carry real information, and skip it if you'd be shipping a stub to check a box. With that said, let's take a look at some other examples:
Ship it if you maintain substantial technical documentation, API references, or a developer portal. Coding agents will use it today.
Ship it if your product is bought by engineers who evaluate through AI assistants. That's a live retrieval path with a live consumer.
Ship it if you already have clean canonical pages and the file is thirty minutes of work. The downside is close to zero.
Skip it if it would list four pages you already have in your sitemap. That's a stub, and stubs are noise.
Skip it if it's replacing indexing work. That trade is straightforwardly bad and I've watched teams make it.
There's one failure mode worth naming explicitly, because it's common and it's genuinely harmful. A popular implementation approach generates an individual Markdown copy of every page on the site. If those Markdown files are indexable, you've just created duplicate content at scale, which dilutes crawl budget and can suppress the original pages. Adding a file meant to help AI find you should not make search engines trust you less.
If you've decided it's worth shipping, my build guide walks through how to create an llms.txt file that carries real signal rather than acting as a stub.
The duplicate content failure mode
There's one failure mode worth naming explicitly, because it's common and it's genuinely harmful rather than merely useless.
A popular implementation approach generates an individual Markdown copy of every page on the site. If those Markdown files are indexable, you've just created duplicate content at scale. Every page on your site now has a twin, crawl budget gets split between them, and search engines have to decide which version is canonical. Sometimes they choose wrong.
Adding a file meant to help AI find you should not make search engines trust you less. If you generate mirrors, block them in robots.txt and apply noindex, then verify in Search Console that the originals are still the indexed version. I've seen this cost a site more traffic in a quarter than llms.txt could plausibly return in a decade.
If you've decided it's worth shipping, my build guide walks through how to create an llms.txt file that carries real signal rather than acting as a stub.
What should you do instead
Everything that actually moves citation rate sits upstream of this file, and none of it is glamorous. The work belongs to large language model optimization and to the answer engine side of the matrix, not to a text file sitting at your domain root.
Confirm index presence first. It's binary and it gates everything else. ChatGPT search retrieves through Bing's index, so a page Bing hasn't indexed scores zero no matter how good it is. Bing Webmaster Tools is where you check that, and IndexNow is how you push new URLs there the day you publish.
Make claims extractable. Short declarative sentences that stand alone when pulled out of the page and dropped into an answer.
Structure headings so each section is a self contained span. Retrieval works on chunks, not whole documents.
Surface published and updated dates in HTML, use visible bylines, and deploy Article schema.
Build consistent entity descriptions across your site, directories, and profiles, so corroboration is available when a model checks.
Earn third party coverage. Corroboration from sources the engines already trust does more than any file you control.
Check what your robots.txt does to AI crawlers, since a vendor wide block is the most common way teams remove themselves from AI answers without noticing.
Where this sits in the Engine Optimization Matrix
The Engine Optimization Matrix maps four engines against five levers, and llms.txt occupies exactly one cell of it. It's a Distribution play for the LLMO engine, and it's a minor one within that cell, sitting well below crawler access and indexing.
That's the right amount of weight to give it. The reason I'm this insistent about the framing is that attention is the scarce resource in most marketing teams, and a file with a memorable name and a simple implementation will always attract more of it than the unglamorous indexing work that actually decides outcomes.
What should you do instead
Everything that actually moves citation rate sits upstream of this file, and none of it is glamorous. The work belongs to large language model optimization and to the answer engine side of the matrix, not to a text file sitting at your domain root.
Confirm index presence first. It's binary and it gates everything else. ChatGPT search retrieves through Bing's index, so a page Bing hasn't indexed scores zero no matter how good it is. Bing Webmaster Tools is where you check that, and IndexNow is how you push new URLs there the day you publish.
Make claims extractable. Short declarative sentences that stand alone when pulled out of the page and dropped into an answer.
Structure headings so each section is a self contained span. Retrieval works on chunks, not whole documents.
Surface published and updated dates in HTML, use visible bylines, and deploy Article schema.
Build consistent entity descriptions across your site, directories, and profiles, so corroboration is available when a model checks.
Earn third party coverage. Corroboration from sources the engines already trust does more than any file you control.
The honest summary
llms.txt is technically coherent and practically unproven for the use case most marketers care about. It's proven for a use case most marketers haven't noticed. Ship it if you're documentation heavy, keep it accurate, and don't let it substitute for the indexing and extractability work that actually decides whether an engine can cite you.
The risk of skipping it today is low. The cost of ignoring it as agent traffic scales might not stay low. Both of those things are true, and anyone telling you only one of them is selling something.
What's next
If you're weighing llms.txt against the rest of your AI visibility work, the ordering matters more than the file does. We help B2B technology teams sequence this properly, indexing first, extractability second, distribution files last.
Book a 30 minute working session and we'll look at your setup together.
Frequently asked questions
Does llms.txt help with Google rankings
No. There's no evidence llms.txt influences Google's organic ranking systems, and Google has not indicated it uses the file. Content quality, relevance, authority, and links remain the ranking signals that matter.
Where does llms.txt go on my site
At the root of your domain, so it resolves at yourdomain.com/llms.txt. Same placement convention as robots.txt. There's no registration step and no gatekeeper.
What's the difference between llms.txt and robots.txt
They do opposite jobs. robots.txt controls crawler access, telling bots what they may and may not fetch, and it's honoured by every major AI crawler. llms.txt is a content index that suggests which pages matter, and no major AI provider has committed to reading it. If you only do one, do robots.txt.
Does GPTBot read llms.txt
GPTBot has been observed fetching the file occasionally. OpenAI has not stated that the file influences how ChatGPT sources, ranks, or cites content, so a fetch should not be read as an endorsement.
Should a B2B SaaS company add llms.txt
If you maintain real documentation, yes, because coding agents fetch it today and it takes under an hour. If your site is marketing pages only, the file would be a stub and your effort is better spent on Bing indexing and extractable page structure.
Comments