llms.txt is a proposed web standard: a plain-text Markdown file placed at the root of your domain that hands large language models a curated map of your most important content. It is not an access-control file like robots.txt, and it does not change how you rank in Google. Its single job is discoverability — giving AI systems a clean, summarised index of the pages you most want them to read and cite.
That is the honest one-line answer. The longer answer is more interesting, because llms.txt sits at the intersection of genuine utility and a lot of hype. In this guide I will walk through what the file actually is, how it differs from robots.txt and sitemaps, whether any AI system currently uses it, and a decision framework for whether it deserves a slot on your roadmap in 2026.
What llms.txt actually is
The format was proposed by Jeremy Howard, co-founder of Answer.AI, in September 2024 and documented at llmstxt.org. The problem it targets is real. Raw HTML is noisy — navigation, cookie banners, ads, JavaScript, and boilerplate all compete with the content a language model is trying to read, and model context windows are finite. A dedicated Markdown file strips that away and points straight at what matters.
The structure is deliberately simple. You start with an H1 carrying the site or project name, add an optional blockquote with a one-line summary, then group links under H2 sections. Each link is a standard Markdown link, optionally followed by a short note explaining what the page covers.
# Acme Docs
> API reference and guides for the Acme platform.
## Docs
- [Quickstart](https://acme.com/docs/quickstart): Get running in five minutes.
- [Authentication](https://acme.com/docs/auth): API keys and OAuth.
## Optional
- [Changelog](https://acme.com/changelog): Release history.There is also a companion convention, llms-full.txt, which inlines the full Markdown content of those pages into a single file so a model can ingest everything in one request. The core file is an index; the full file is the whole library flattened. Both live at your root, served as plain text.
How llms.txt differs from robots.txt and sitemaps
This is where most confusion starts, so it is worth being precise. These three files look similar — small text files at your domain root — but they do completely different jobs, and llms.txt does not replace either of the others.
robots.txt is an access-control file. It tells crawlers, including AI crawlers like GPTBot and ClaudeBot, what they are allowed to fetch. It is a permission layer, standardised as IETF RFC 9309, and it is the file that actually governs whether AI bots can touch your content at all.
An XML sitemap is a machine-readable list of every URL you want discovered, built for search engine crawlers to find and queue pages efficiently. It is exhaustive and unopinionated — it lists everything indexable, not just your best work.
llms.txt is neither of those. It grants no permissions and blocks nothing, and it is not meant to be exhaustive. It is a curated, human-and-machine-readable summary of the content you most want an LLM to prioritise. Think of robots.txt as the door policy, the sitemap as the full inventory, and llms.txt as the concierge pointing at the ten things worth seeing. If your robots.txt blocks AI crawlers, no llms.txt file will bring them back — the permission layer wins. Sorting out that permission layer is a core part of any technical SEO audit, and it matters far more than the curation file on top of it.
Does llms.txt actually work? The honest state in 2026
Here is the uncomfortable part that most tutorials skip. As of mid-2026, no major AI vendor has publicly confirmed that its search or answer products consume llms.txt. Not OpenAI, not Anthropic, not Perplexity, and pointedly not Google.
Google’s John Mueller has been blunt about it, noting that no AI service he is aware of actually uses llms.txt and likening it to the old keywords meta tag — a signal you can fill in that nobody on the receiving end reads (Google, 2025). That comparison stings precisely because it is apt. The keywords meta tag also seemed logical, saw wide adoption, and turned out to be ignored.
Ahrefs reached a similar conclusion after checking real server logs, reporting that it saw no meaningful requests for the file from AI crawlers and advising teams not to prioritise it (Ahrefs Blog, 2025). If the bots are not fetching the file, the file cannot influence anything, no matter how clean it is.
So why does adoption keep climbing? Developer-facing companies — the crowd that read Howard’s proposal first — have published llms.txt widely, and the directory at llmstxt.site now catalogues thousands of domains that host one. But adoption is not consumption. Thousands of sites publishing a file that no crawler requests is a community betting on a future standard, not evidence of a working one.
In my experience auditing sites over the past year, I have yet to see a single case where adding llms.txt produced a measurable change in AI citations or referral traffic. That does not make it worthless — it makes it speculative. Treat it as a cheap option on a possible future, not as a lever you can pull today.
Should you implement llms.txt? A decision framework
Because the upside is unproven but the cost is low, the right question is not “does it work” but “is it cheap enough to be worth the bet for my site.” Run it through this quick framework.
-
Is your site documentation-heavy or developer-facing? This was the original use case, and it is where llms.txt makes the most sense. Structured docs map cleanly onto the format, and your audience already uses AI tools to query technical content.
-
Can you generate and maintain it automatically? If your build can produce llms.txt from existing content with zero ongoing hand-editing, the maintenance cost is effectively nil and the bet is easy. If keeping it accurate means manual work every week, the maths gets worse fast.
-
Are your real SEO fundamentals already solid? Crawlability, clean semantic HTML, structured data, internal linking, and genuinely useful content are what actually get you surfaced in AI answers. If any of those are shaky, they beat llms.txt every time for attention and budget.
-
Would this displace higher-value work? A 30-minute automated task is fine. A multi-day manual curation project competing with a content SEO programme or a technical fix is a misallocation.
The practitioner’s verdict: publish it if you can automate it, ignore it if you cannot, and never let it jump the queue ahead of the fundamentals that demonstrably move AI visibility.
How to implement llms.txt step by step
If you have decided it clears the bar, here is a clean process that keeps the effort proportional to the payoff.
-
Select your priority pages. Pick the twenty to fifty URLs you most want cited — core docs, cornerstone guides, key service or product pages, your highest-value posts. Curation is the entire point; a file that lists everything is just a worse sitemap.
-
Write the file. Create llms.txt at your root. Lead with an H1 name, a blockquote summary, and H2 sections. Add a short, honest description after each link so a model understands what the page contains without fetching it.
-
Generate llms-full.txt if it fits. For docs and evergreen guides, inline the full Markdown into llms-full.txt so a model can ingest the content directly. Skip it for large or fast-changing sites where the file would balloon.
-
Automate it in your build. Plugins and integrations exist for Astro, Next.js, and most documentation frameworks. Wire generation into your pipeline so the file rebuilds whenever content changes and never drifts out of sync.
-
Serve and validate it. Return it as
text/plain, confirm it resolves athttps://yourdomain.com/llms.txt, and check the Markdown parses cleanly. Keep it consistent with the canonical URLs in your sitemap. -
Leave your permission layer alone. llms.txt is additive. Make sure robots.txt still allows the search and retrieval crawlers you want citing you — that is the file doing the real work. If your technical SEO setup blocks those bots, fix that first.
The whole exercise should take an afternoon at most if you automate it. If it is taking longer, you have over-engineered a speculative file.
llms.txt and the bigger AI search picture
Step back and the priorities are clear. What gets you cited in ChatGPT, Perplexity, and Google’s AI answers today is being crawlable, publishing clean semantic HTML, marking up entities with structured data, and writing content genuinely worth quoting. Those are the same generative engine optimization fundamentals that already decide who wins AI visibility, and they are where your effort compounds.
llms.txt is a low-cost hedge layered on top of that foundation, not a substitute for it. Publish it if you can automate it, keep it honest and curated, and hold realistic expectations about a standard that no major model has yet committed to reading. If you want an outside read on where llms.txt fits against the technical and content work that actually moves the needle, that is exactly the kind of prioritisation call worth taking to an experienced SEO consultant before you spend real time on it. It is also the first thing worth testing when you are deciding whether to hire an AI search consultant at all.
Frequently Asked Questions
What is llms.txt?
llms.txt is a proposed standard file, written in Markdown and placed at your domain root, that gives large language models a curated index of your most important pages. It exists to make your best content easy for AI systems to find and understand, stripping away the HTML noise that clutters a normal page. It is a discoverability aid, not an access-control or ranking mechanism.
Does Google use llms.txt?
No. Google has publicly stated it does not use llms.txt, with John Mueller comparing it to the long-ignored keywords meta tag. As of mid-2026, no major AI provider — Google, OpenAI, Anthropic, or Perplexity — has confirmed that its systems consume the file, so it should be treated as speculative rather than a confirmed ranking or citation factor.
Is llms.txt the same as robots.txt?
No. robots.txt controls access — it tells crawlers, including AI bots, what they may and may not fetch. llms.txt grants no permissions and blocks nothing; it only curates and summarises content for models that choose to read it. If robots.txt blocks a crawler, llms.txt cannot override that, because the permission layer always wins.
Where do I put the llms.txt file?
At the root of your domain, so it resolves at https://yourdomain.com/llms.txt, served as plain text. The optional companion file, llms-full.txt, lives in the same location and inlines the full Markdown content of your priority pages for models that want to ingest everything in one request.
Does llms.txt improve SEO?
Not in any measurable way today. It does not affect traditional Google rankings, and there is no confirmed evidence that AI crawlers currently read it. It may become useful if adoption by AI vendors materialises, so publishing an automated version is a reasonable low-cost hedge — but it should never displace crawlability, structured data, and strong content, which are what actually drive AI visibility.
