How to Create and Add an llms.txt File for Your B2B Site
- Harold Bell
- Aug 9
- 8 min read
Updated: Aug 16
Key Takeaways
|
I've written elsewhere about whether llms.txt is worth your time, and the short version is that it depends entirely on whether you have documentation worth pointing at. If you've decided you do, this is the build.
In more than 16 years of running content programs for enterprise technology brands, I've learned that the difference between a file that does something and a file that sits there is almost always curation. The format takes 10 minutes. The judgement about what belongs in it is the actual work.
With that said, let's dive into what an llms.txt file is, and how to add llms.txt to your site.
What does an llms.txt file look like
It's a markdown document with one H1 carrying your site or company name, an optional blockquote summarising what you do, then H2 sections grouping markdown links, each with a short description after a colon. Plain text, served at yourdomain.com/llms.txt. |
The required structure
Here's the shape, using a B2B software company as the example.
# Acme Data Platform
> Acme is a data observability platform for enterprise
> engineering teams. This file indexes our documentation,
> product references, and technical guides.
## Documentation
- [Quickstart](https://acme.com/docs/quickstart): Install and
send your first events in ten minutes.
- [API Reference](https://acme.com/docs/api): Full endpoint
reference with request and response schemas.
- [Pipeline Configuration](https://acme.com/docs/pipelines):
How to define, monitor, and alert on data pipelines.
## Product
- [Platform Overview](https://acme.com/platform): What the
platform does and which teams use it.
- [Integrations](https://acme.com/integrations): Supported
warehouses, orchestrators, and BI tools.
## Optional
- [Changelog](https://acme.com/changelog): Release history.
- [Security](https://acme.com/security): Compliance posture
and certifications.
Nothing about that is technically demanding. Every element is plain Markdown and any text editor produces it. Which is precisely why the format gets shipped badly so often, since there's no validator to fail and no error message to read.
What the Optional section signals
The Optional heading has a defined meaning in the proposal rather than being a casual label. Items under it can be skipped when an agent is working with a tight context budget. Everything above it is treated as core.
That's the one piece of structural signalling the format gives you, and most files ignore it entirely. Use it. Put your reference material and canonical explanations above the line, and put changelogs, legal pages, and anything periodic below it. You're telling a system with finite attention what to drop first, which is a small amount of control worth exercising.
How do you add an llms.txt file
Six steps, in order:
Pull your page inventory. Export your sitemap and sort by whatever traffic or engagement data you have. You're looking for pages that answer questions, not pages that sell.
Cut to the canonical set. For most B2B sites this lands between fifteen and forty URLs. If you're over eighty, you're building a sitemap, not an index.
Group into H2 sections that match how someone would ask. Documentation, Product, Guides, Company. Match your buyer's mental model rather than your navigation.
Write one line per link. Describe what the page answers, not what it's called. 'How to configure SSO for Okta and Entra ID' beats 'SSO Setup'.
Add the blockquote summary. Two or three sentences on what your company does and who it serves. This is the entity description a model may lean on, so keep it identical to the one you use everywhere else.
Serve it at the root as text/plain or text/markdown, then fetch it in a browser to confirm it renders as text rather than downloading as a file.
Curating down to the canonical set
This is where the hour actually goes, and it's worth spending. The instinct is to include everything on the theory that more coverage is better. That instinct is wrong here in a way it isn't wrong for sitemaps.
A sitemap exists to be comprehensive, because a search engine crawls broadly and decides for itself what matters. This file exists to be selective, because an agent with a finite context window is asking you which pages matter and taking your word for it. A list of two hundred URLs answers that question with a shrug. A list of thirty answers it with a position.
The practical test I use is whether I'd hand the page to a prospect who asked a technical question. If yes, it belongs. If I'd hedge, it doesn't.
Writing descriptions that earn their tokens
Every description costs context budget, so each one has to buy something. The common failure is copying the title tag, which produces a line that restates the link text and adds nothing.
Write the description as the answer to what will I learn here. So not SSO Setup, but how to configure SSO for Okta and Entra ID including group mapping. The second version lets an agent decide whether to fetch the page without fetching it, which is the entire economic argument for the file existing.
Serving it correctly, including on Wix
The file has to resolve at the domain root and return plain text. On most stacks that's a static file drop. On Wix specifically you'll need to serve it through a Velo route or the site's file handling rather than a page, because a normal page won't produce a clean plain text response at a root path.
Verify the response headers after you ship. A file served as text/html defeats the purpose, and so does one that triggers a download prompt. Fetch it in a browser and in a terminal, and confirm both behave.
What should go in the file and what should stay out
Include documentation, technical guides, reference pages, and canonical product explanations. Exclude gated assets, thin marketing pages, pricing tables you change often, and anything requiring authentication. |
The quotability test
The test I use is whether the page would embarrass you if a model quoted it verbatim. A reference page that states things plainly passes. A landing page built around a form and three benefit statements does not.
That's not an aesthetic judgement. There's nothing on that landing page to quote, so the model will either skip it or paraphrase something vague back at your prospect. Either outcome is worse than the page not being listed, because you've spent a fetch to produce a weaker impression than silence would have.
The same logic governs your blockquote summary. Keep it identical to the entity description you use everywhere else, because inconsistent self descriptions across your properties give any system checking you a reason to trust none of them.
Why gated assets don't belong
Listing a URL that returns a form wall wastes a fetch and teaches the agent that your file is unreliable. It's the closest thing to a broken promise the format allows, since the link resolves and delivers nothing.
If the content behind the gate genuinely matters, publish an ungated summary and list that instead. You keep the conversion path and you stop burning trust on a fetch that returns a login screen.
What are the failure modes that make a file worse than none
Five, and I've seen all of them in the wild:
Markdown mirrors of every page, left indexable. This is the serious one and it gets its own section below.
The plugin stub. Roughly four in ten existing files are auto generated defaults listing a homepage and a contact page. That's noise with a filename.
Links that 404. A stale file is a broken promise, and agents that hit dead links stop treating the file as authoritative.
Descriptions copied from title tags. If the description adds nothing beyond the link text, delete it.
Treating it as a replacement for indexing work. Every hour spent perfecting this file is an hour not spent confirming Bing has your pages, and only one of those two things gates citation eligibility.
The mirror problem in detail
Several popular implementations generate an individual Markdown copy of every page on the site, so /about also exists as /about.md. If those files are indexable, every page on your site now has a twin.
Crawl budget splits between the pair. Search engines have to pick a canonical version and sometimes pick wrong. You've taken a file intended to help AI find you and used it to make search engines trust you less, which is a genuinely bad trade at any scale.
If you generate mirrors, block them in robots.txt, apply noindex, and then verify in Search Console that the originals are still the indexed version. Don't assume, check.
How do you verify it's working
Check your server logs for requests to /llms.txt and note the user agents making them. Expect coding agents and occasional GPTBot fetches. Do not expect a citation lift, and do not measure the file against one. |
Setting the success criterion honestly
Your server logs are where this gets confirmed. Set the criterion before you ship, because otherwise you'll end up attributing an unrelated citation improvement to a file that had nothing to do with it, and then repeating that claim to someone who believes you.
Reasonable criteria are that the file resolves, returns plain text, contains no dead links, and gets fetched, including the occasional GPTBot request. That's it. That's the full extent of what this file can prove about itself, and a build that clears all four has done its job.
Putting it on a review cadence
Then treat it like any other structural asset. Quarterly review alongside your sitemap, plus an immediate check any time you rename a slug, retire a page, or ship new documentation.
This is the same discipline your internal links need and it fails the same way when nobody owns it. A file that was accurate at launch and hasn't been touched in a year is a list of confident pointers to pages that moved.
What's next
Most teams get the file right and the sequencing wrong. If you want a second pair of eyes on where llms.txt sits against your indexing and extractability work, that's a short conversation.
Book a 30 minute working session and we'll look at your setup together.
Frequently asked questions
Do I need llms-full.txt as well
Only if you're documentation heavy. llms-full.txt holds your full content in a single Markdown document for deep ingestion rather than acting as an index. Most sites only need llms.txt, and maintaining a second file you don't update is worse than not having it.
How many URLs should llms.txt contain
For most B2B sites, fifteen to forty. The point is curation. If you're listing everything, an agent gains nothing over reading your sitemap, and you've spent the effort for no signal.
Does llms.txt need to be in robots.txt
No. There's no reference or registration mechanism. Agents that look for it request the root path directly. What does belong in robots.txt is a block on any generated Markdown mirrors, so they never get indexed.
Will adding llms.txt get my site cited by ChatGPT
There's no evidence it will. Citation eligibility runs through index presence and content extractability. Build the file if you have documentation worth indexing for coding agents, and treat any citation change as coincidental unless you can trace it.
How often should I update llms.txt
Quarterly at minimum, and immediately after any slug rename or page retirement. Dead links in the file are the fastest way to make it useless.
Comments