You’ve heard about robots.txt, but have you heard of llms.txt?
LLMs.txt is an emerging “standard” file that’s been designed to optimise website content for large language models (LLMs). Much in the same way that robots.txt steers the web’s crawlers, so LLMS.txt has been designed to handle how LLMs crawl your site’s content.
the new text file serves as a curated index for LLMs, providing concise summaries of a site’s purpose, critical contextual details, and prioritised links to machine-readable resources.
The key features of the new LLMs.txt include:
- Location: Typically hosted at a website’s root path (e.g., https://example.com/llms.txt ),
- Format: Uses markdown for readability by both humans and LLMs,
- Structure: Includes an H1 header for the site’s name, a blockquote summarising its purpose, freeform sections for context, and H2-delimited resource lists.
The purpose of LLMs.txt is to:
- Streamline how LLMs access, parse, and utilise web content during inference,
- Provide deterministic access to machine-friendly data, reducing latency and improving accuracy,
- Allow website owners to control and restrict AI access to their content.
Adoption of LLMs.txt seems to be growing, especially amongst those believing it may give them the slight edge when it comes to being included in AI search results. The new LLMs.txt file is certainly a complementary to long-established existing web protocols like robots.txt and sitemap.xml, by addressing LLMs’ specific need for pre-processed, hierarchical data.
However, I’m not yet convinced that it’s either officially recognised or will be widely adopted. That remains to be seen.
To implement LLMs.txt, webmasters should audit existing documentation, create the file following the specification, consider dual-format publishing (HTML and markdown), and test with LLM frameworks.
While LLMs.txt is not a binding rule and operates on a voluntary basis, it does seem to represent an early opportunity for websites to communicate more effectively with AI technologies in the future.
So watch this space and, if I hear of any updates to whether LLMS.txt is making a positive impact, or not, I’ll let you know. It’s well worth keeping an eye on, as I said, it may help with indexing and citations in AI results, but whilst I can still get better results in the SERPs, my main attention will remain there for now.
If you need any SEO, content, or any other digital marketing help, just call me on 01252 692 765.
1 thought on “LLMs.txt”