Clients keep asking me whether they need an llms.txt file before AI search leaves them behind. It is a fair question, because plenty of tools now nag you to add one. So I checked the original proposal, the provider docs and the published server logs, and here is what I found.
Is llms.txt standardized yet?
No. Jeremy Howard, co-founder of Answer.AI, published the proposal on llmstxt.org on September 3, 2024. The page still calls itself a proposal, and it received a v2 update in August 2026. It is a community convention, not an IETF or W3C standard.
The format is plain markdown at /llms.txt. The only required part is an H1 with the site name. After that come an optional blockquote summary, some notes, and H2 sections that list links with short descriptions.
It does not replace robots.txt or your sitemap. Robots.txt tells crawlers what they may access, and a sitemap lists every page. By contrast, an llms.txt file is a short, curated reading list that an agent can open on demand.
Will llms.txt become a real standard?
Honestly, I would not bet on it soon. Nobody has submitted it to a standards body as a working group item, and the big AI providers have not committed to reading it.
Still, there are signals pointing the other way. Chrome’s Lighthouse now has an experimental Agentic Browsing category with an llms.txt audit. However, that audit marks a missing file as Not Applicable, and Google describes the file as optional for now. My read is simple: it may grow inside developer tooling, while search and answer engines keep relying on normal crawling.
Are AI providers really adopting it?
This is where the hype and the evidence split apart. Publishing a file and reading one are two very different things.
Who publishes one
Many developer docs publish one. I fetched the files myself and confirmed live versions from Anthropic, OpenAI, Google’s Gemini API docs, Perplexity and Cloudflare. Mintlify has generated llms.txt automatically for hosted docs since November 2024, so thousands of SaaS docs sites got one without lifting a finger.
Who actually reads it
Here the picture gets thin. OpenAI, Anthropic and Perplexity document their crawlers and robots.txt controls, yet none of their crawler pages says the bots read llms.txt. Google goes further in its AI features guidance: “You don’t need to create new machine readable files, AI text files, or markup to appear in these features.”
John Mueller was blunt on Bluesky in June 2025: “FWIW no AI system currently uses llms.txt.” Later he added that the time to think about it is when an AI system sending you real traffic checks for it in your logs. I found no primary statement from Gary Illyes, so I am not quoting one.
The log studies agree with him. Ahrefs looked at 137,000 sites in June 2026 and found that 97% of llms.txt files received zero requests in a month. Moreover, no AI bot went looking for files that did not exist. Dries Buytaert saw 52 requests in a month on his own site, and every one came from SEO tools.
So, the plain answer is this. No major AI provider confirms that its search or answer engine uses llms.txt. The main readers today are coding agents pointed at docs, plus the tools that audit for it.
Does your website really need an llms.txt file?
My verdict depends on the type of site. Here is how I advise clients right now:
- Docs or SaaS: yes, add one. Developers paste docs into coding agents, and that is the real use case.
- E-commerce: optional. Product feeds, structured data and crawlable category pages matter far more.
- Local business: skip it. A complete Business Profile and a clear contact page do more.
- Blog: optional at best. If your SEO plugin creates one for free, leave it on, but expect nothing.
The cost is low. A hand written file takes me about an hour, and Yoast can generate one automatically. The real cost is keeping it accurate, because a stale reading list is worse than none.
# Example Studio
> Web shop builds and SEO audits for small businesses.
## Services
- [SEO audit](https://example.com/seo-audit/): what the audit covers
- [Pricing](https://example.com/pricing/): current packages
## Optional
- [Blog](https://example.com/blog/): articles and case studies
What I do instead of relying on llms.txt
First, I decide which AI crawlers a client wants and write that into robots.txt. The tokens come straight from the providers’ docs: GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot from Perplexity, and Google-Extended from Google. Blocking a search bot can cost you citations, so I block training bots only when a client asks for it.
Next, I make sure the content is crawlable without JavaScript tricks, which I covered in why content can be invisible despite a 200 status. Then I add structured data that matches the visible text. Finally, I keep crawl paths tidy, as explained in what actually matters for crawl budget.
That combination helps every engine, old and new. For the bigger picture, see my State of AI roundup for October 2026 and my take on whether SEO is dead. An AI crawler guide file is a nice extra, but it never replaces the basics.
Sources
- The /llms.txt file, v2, Jeremy Howard, llmstxt.org, September 3, 2024 (updated August 10, 2026): llmstxt.org
- llms.txt audit, Chrome for Developers, updated May 5, 2026: developer.chrome.com
- AI features and your website, Google Search Central, updated December 10, 2025: developers.google.com
- John Mueller on Bluesky, June 17, 2025: bsky.app
- We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read, Ahrefs, June 15, 2026: ahrefs.com
- Markdown, llms.txt and AI crawlers, Dries Buytaert, March 5, 2026: dri.es
- Overview of OpenAI crawlers, OpenAI, accessed October 14, 2026: developers.openai.com
