llms.txt: what the logs actually show

Somewhere in the past year, a new item landed on your marketing checklist: add an llms.txt file to your website. An agency recommended it. A LinkedIn post insisted on it. Your website platform may have quietly generated one for you already.

The pitch sounds reasonable. It is a small text file at the root of your domain that tells AI systems what your website is about and which pages matter. Cheap to make, supposedly essential for getting recommended by AI assistants.

If you run a local expertise business, a real estate team, a hotel, a short-term rental operation, you have limited hours for this work. Your website, your blog, your branded content hub: every one of them competes for the same afternoon. So before llms.txt takes one of those afternoons, it is worth asking a plain question. Does anything actually read the file?

We now have an answer, because someone finally checked the logs at scale. The short version: almost nothing reads it, no major engine has committed to it, and the effort is better spent on how your content is structured. The longer version is more interesting, because the file is not entirely useless. It is just useless for the thing most people publish it for.

What is llms.txt supposed to do?

The idea comes from Jeremy Howard, co-founder of Answer.AI, who proposed llms.txt in 2024 as a markdown index file at a website's root. It summarizes what the website is and links its most important pages, so language models and AI agents can orient themselves without crawling everything.

Two clarifications, because the name misleads people:

It is not a robots.txt-style directive. It controls nothing and blocks nothing.

It is also not the same thing as publishing markdown copies of your pages. That is a separate tactic with separate problems.

The "publish this and AI will recommend you" framing was attached later, as the idea spread through the SEO industry on speculation that AI platforms would reward the file. That speculation is the part the evidence now speaks to.

Who actually reads llms.txt?

In mid-2026, Ahrefs analyzed server logs and live traffic across 137,000 domains to measure who requests these files. Three findings carry most of the weight:

First, adoption is real: 28% of the domains in the study publish an llms.txt file.

Second, readership is not: 97% of those files received zero requests in May 2026. Not fewer than expected. Zero. Nothing fetched them at all.

Third, of the requests that did arrive at the remaining files, the bots that actually answer live AI-search queries (the retrieval bots behind products like ChatGPT search and Perplexity) accounted for 1.1%. In Ahrefs' words: "Slackbot alone fetched llms.txt files more often than PerplexityBot did." A chat app's link-preview bot out-fetched an AI search engine the file was supposedly built for.

There is a fourth finding that closes the loop. Ahrefs also examined requests for llms.txt files that do not exist, and found zero AI bots probing for missing files. AI systems do not go looking for llms.txt. Publishing one does not put you on any radar; a bot fetches it only when something explicitly points there.

The Reader Ledger

To decide whether any file like this deserves your time, sort its actual audience into a ledger: who shows up, and whether that reader can affect your visibility. Here is the ledger for llms.txt, using the request categories from the Ahrefs study:

One reading of that table matters more than the rest: the audience the file was marketed for is near the bottom of its own ledger. The measurable readership is coding agents and the tooling industry auditing itself.

Keep in mind these percentages describe only the 3% of files that got any traffic. And a fetch is not proof of use; Ahrefs notes every figure is a ceiling on actual consumption.

What do the search engines say about it?

No major AI platform has committed to reading llms.txt. Google has gone further and said the opposite, in writing.

Google's guide to AI features in Search lists llms.txt under a section titled "Mythbusting generative AI search: what you don't need to do." You do not need machine-readable files, AI text files, or special markup to appear in Google Search, including its generative AI features, "as Google Search itself doesn't use them."

Google's John Mueller was blunter. Asked about llms.txt, he compared it to the keywords meta tag: "AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it). To me, it's comparable to the keywords meta tag." The keywords meta tag is the canonical example of a self-declared signal engines learned to ignore, because a website claiming what it is about is worthless when you can just read the website.

That is the pattern to internalize. Engines trust what they can verify by fetching your actual pages. Self-description shortcuts get discounted.

So is llms.txt ever worth publishing?

Yes, in one narrow case, and it is honest to name it rather than round down to "ignore the file entirely."

The measurable readership of llms.txt is AI agents and coding tools. In the Ahrefs data, agents and agentic infrastructure were the largest AI category of readers, and Anthropic's Claude Code fetched llms.txt more than any AI retrieval bot or assistant. Training crawlers like GPTBot fetch it too. "Ignored by AI search" is accurate. "Ignored by everything" is not.

Peec, whose prompt-tracking platform we use for AI visibility monitoring, lands in the same place: for SEOs chasing traffic from ChatGPT and Perplexity, llms.txt and .md files are "a distraction without any upside," but the file is "a helpful tool to give AI agents and coding tools a pointer" on how to interact with your website.

So the decision rule is short. If your customers are developers, if agents complete tasks on your website, or if you publish API documentation, a minimal llms.txt is a cheap hedge. If you are a local expertise business trying to get recommended by AI assistants, it is decoration.

One caution before you publish even the hedge version: Ahrefs found a research crawler identifying itself as prompt-injection-survey systematically studying llms.txt files. Agents are built to trust this file, which makes it an attack surface. A stale or tampered file misleads every agent that reads it, so if you keep one, treat it like code: version-control it, restrict who edits it, and keep the contents to plain links and descriptions.

What should you do instead?

Redirect the afternoon. The evidence says AI systems learn about you the unglamorous way: by fetching your real pages, over and over, and reading what renders there.

That traffic is not hypothetical: the same logs that show llms.txt sitting untouched show AI crawlers returning to content pages day after day. The crawling budget goes to content pages, not index files.

So spend your effort where the crawlers already are:

Make every important page self-explanatory. An AI system landing on one page gets no surrounding context. The page itself has to say what it is, where it applies, and who it is for.

Prefer structured, specific resources over undifferentiated prose. A page that cleanly answers "which neighborhoods near [town] are walkable to schools and coffee" gives an engine something to cite. A 3,000-word brand essay does not.

Keep it fresh and keep it flat. Content that updates gets re-fetched. Content buried behind logins, scripts, or walled platforms does not get fetched at all.

This is the reasoning behind how Stacklist structures content. A real estate agent's hub is a set of stacks, each stack a curated collection of cards on one specific topic, each rendering as a clean, crawlable page that explains itself. Nothing about that requires Stacklist; it is a structure you can build anywhere. The hub just makes it the default instead of a discipline.

And whichever way you build, watch your own logs rather than arguing from theory. Filter your server logs for requests to /llms.txt and see who shows up (for most domains, the honest answer will be nobody). If you monitor AI visibility in Peec, compare its crawl insights against your prompt-tracking data: which bots fetch your real pages, and which pages they fetch, tells you where your actual AI surface area is.

What this won't tell you

Honest limits of everything above:

The Ahrefs study measures fetches, not reads. It cannot see what a bot did with a file after fetching it, and its population skews technical, so the 28% adoption figure is an upper bound.

Log evidence describes today. If agents end up mediating more of AI search, a file agents read could start to matter more. That is a reason to keep the hedge cheap, not a reason to bet on it.

None of this measures whether you appear in AI answers. Logs tell you what got fetched. Whether you get cited is a separate question with separate tooling, and a separate guide.

And skipping llms.txt will not fix weak content. The file was never the bottleneck; it just was not the fix either.

Here's what to check, in order

1. Your own logs for /llms.txt requests. Before publishing or maintaining the file, get the base rate. Ahrefs' data says 97% odds of zero readers.
2. Whether you have an agent or developer audience. API docs, developer customers, or agents transacting on your website justify a minimal llms.txt. Otherwise skip it.
3. If you keep the file: lock it down. Version control, restricted edits, plain links only, nothing instruction-shaped. It is a trusted input for agents and already being probed for prompt injection.
4. What AI bots fetch on your real pages. Server logs or Peec crawl insights. This is your actual AI surface area; grow it.
5. Whether your important pages stand alone. Each one should state its own context: what, where, for whom.
6. Where your best answers live. If they are trapped in feeds or walled platforms, restructure them into specific, crawlable resources: a hub of stacks, or the equivalent you build yourself.

The file was a shortcut that nothing took. The pages were the point all along.

We collected every study and primary source cited in this guide into a companion stack you can browse and check against our reading: