llms.txt Explained: What the llms txt File Does

llms.txt is a proposed web standard: a plain Markdown file at your site root that gives AI agents a curated, token-efficient map of your most important content.

For 30 years, the internet ran on robots.txt — a “Do Not Enter” sign for clumsy search spiders. In 2026, we’re dealing with readers, not just spiders. ChatGPT’s crawler, research agents, and RAG pipelines want to understand your content, and the modern web fights them with JavaScript bloat, popups, and cookie banners that waste tokens.

This guide covers what an llms txt file is, the exact structure the specification requires, a real example from Stripe, how to implement, generate, and validate one, and how it fits your Generative Engine Optimization (GEO) strategy.

Cluttered HTML wastes AI tokens and causes hallucinations

To see why llms.txt is needed, look at how LLMs actually “read.”

A traditional crawler like Googlebot scans for links and keywords and ignores the visual chrome. An AI agent (say, a RAG pipeline) is trying to ingest information, and the modern web is hostile to ingestion.

The token tax of the modern web breaks down into three buckets:

1. Boilerplate. Headers, footers, and navbars repeat on every page. An AI reading 10 pages reads your navbar 10 times. That wastes context-window space and money.
2. DOM noise. <div>, <span>, class names, and inline scripts are gibberish to an LLM trying to answer a question.
3. Visual vs. semantic content. Popups can obscure content, “Read More” buttons hide it, and an AI can’t really “click” anything.

The result is hallucination. When an AI scrapes a JavaScript-heavy page, it often gets a fragmented mess and fills in the blanks itself. That’s when it invents your pricing, your features, or your history.

llms.txt solves this by giving knowledge its own dedicated endpoint.

What is llms.txt exactly?

The llms.txt proposal (popularized by Jeremy Howard) is a convention for placing a file at the root of your domain, e.g. yourdomain.com/llms.txt.

It does two things:

1. Acts as a map. It tells AI agents where to find the “AI-ready” version of your site.
2. Provides context. A concise summary of who you are and what you do, injected straight into the model’s prompt.

Think of it as a sitemap for robots that read. You explicitly list the pages that matter, and you point the agent at clean Markdown instead of HTML.

The anatomy of an llms txt file

The format is defined in the official llms.txt specification, which Jeremy Howard published in September 2024. It is deliberately minimal, and it reuses Markdown so that both people and models can read the same file. The spec fixes the running order of the sections, yet only the very first heading is strictly required.

In practice, that means four ingredients in a fixed order: a name, a one-line summary a model can quote, optional context, then curated links to your best pages.

Example /llms.txt:

# cloro - AI Brand Monitoring Platform

> cloro is the leading platform for tracking brand visibility across Large Language Models (LLMs) like ChatGPT, Claude, and Perplexity.

## Key Pages

- [Pricing](/#pricing): current plans and query limits

The required order of sections

The order matters because it is the order a model reads: your name first, then the one-line summary it can quote back, then supporting context, then the links it might follow. The specification asks you to assemble the file in this precise sequence:

1. An H1 line with the name of the project or site. This is the only mandatory element in the entire file.
2. A blockquote that summarises the entity in one or two sentences. It behaves like a short system prompt for your brand.
3. Zero or more Markdown paragraphs of extra context, carrying no headings, that flesh out the summary.
4. One or more H2 sections, each holding a Markdown list of [name](url) links with optional notes after a colon.

The special “Optional” section

The spec reserves one H2 named Optional for links that a model can safely skip. When an agent is short on context-window budget, it drops the Optional block first and keeps the core pages. Treat that section as your overflow shelf rather than your main catalogue.

Why every link should point at Markdown

Each link in a file list should resolve to a clean .md or .txt version of the page, not the HTML original. The whole value of an llms txt file collapses when the links send an agent straight back into the boilerplate you were trying to avoid. Publish a Markdown twin for every page you list, and then point the link at that twin.

A real llms txt example: how Stripe structures its file

Stripe publishes a live file at docs.stripe.com/llms.txt that follows the spec closely. It opens with an H1 (# Stripe Documentation), then uses a blockquote to hand the model explicit integration guidance before any links appear.

That guidance is the interesting part. Stripe’s blockquote tells agents which APIs to prefer, such as the Checkout Sessions API over the older Charges API, and which endpoints to avoid. Below it, the file groups links under product-named H2 sections like Payments, Billing, Checkout, and Connect. Every entry is a Markdown link that points directly at a documentation page.

The lesson for your own llms txt file is that the blockquote is not throwaway marketing copy. It is the one place where you can steer how a model describes your product, so spend those words on the facts you most want repeated correctly.

How to implement llms.txt

You don’t need a site redesign. You’re creating a shadow site of text files alongside the existing one.

Step 1: Create your shadow content

Convert your key pages into Markdown to strip the HTML noise.

Your pricing.html might be 50kb of code. The equivalent pricing.md should be 2kb of text.

Example pricing.md:

# Pricing Plans

## Hobby Plan

- Cost: $29/month
- Features: 500 queries, Daily updates.

## Business Plan

- Cost: $99/month
- Features: 5,000 queries, Hourly updates.

Step 2: Consolidate into llms-full.txt

Several proposals suggest a single large text file (llms-full.txt) containing all your core documentation concatenated together. RAG systems prefer fetching one file: fewer HTTP requests, and the model gets the full context in a single pass.

Step 3: Deploy the root file

Agents check this location by convention, the same way crawlers check robots.txt, so the file only works if it sits in the right place with the right header. Place llms.txt at your root. Make sure your server returns text/markdown or text/plain headers.

Step 4: Advertise it

Auto-discovery is still evolving. In the meantime, feed the URL manually to custom GPTs, Claude Projects, and other agents to “train” them on your documentation.

Tools to generate llms.txt

If writing these files by hand feels tedious, a few llms txt generators will crawl your site and produce the Markdown structure for you.

• Keploy. One-click generator that scans a URL and builds the file. Fine for simple sites.
• Writesonic. Structured text generator aimed at LLM training and inference.
• Gushwork. More granular control over which site areas to include or exclude.
• Fibr AI. Generates a file with explicit permissions for bots like GPTBot and ClaudeBot.

These tools are useful for a first pass, but review the output by hand. The shadow content for your most critical pages needs to be accurate.

llms.txt vs llms-full.txt: which file should you ship?

People conflate these two files, yet they play different roles. The base llms.txt is an index: a short map of links that an agent can crawl selectively. A companion file, usually named llms-full.txt, concatenates the actual Markdown content of those pages into one long document.

The trade-off is context versus control. An llms-full.txt lets a RAG pipeline pull your entire documentation set in a single request, which is efficient when the model has a large context window to spare. The downside is size, because a full file for a big documentation site can run to hundreds of thousands of tokens that many agents will simply truncate.

This guide used to recommend shipping both as a safe default. The 2026 measurement data, covered in the next section, no longer supports that. When 97% of published llms.txt files go unread, a second and much larger file multiplies the maintenance surface for an audience the server logs cannot find.

Ship the index alone unless you have a named consumer for the full version. The one case with real users today is developer documentation: coding agents such as Cursor and Claude Code do fetch docs, and the documented pattern is to point them at llms.txt first and let them decide whether the context budget justifies pulling llms-full.txt. If that describes your product, most documentation platforms generate both files for you, which reduces the decision to a checkbox rather than a project.

For a marketing site, the calculation is different. Concatenating a blog archive produces a multi-megabyte file that no coding agent wants and no answer engine has been shown to read, while per-page Markdown, served through content negotiation, already gives any agent the same content on demand.

Who publishes an llms txt file — and do AI crawlers read it?

Adoption so far is led by developer-tools and documentation-heavy companies. Stripe, as shown above, ships a detailed file, and many API-first products have followed with their own documentation indexes. Documentation platforms have also begun generating the file automatically, which is why so many software sites now expose one at their root.

The consumption side is where the evidence turned. In June 2026 Ahrefs published server-log analysis of 137,210 domains: of the roughly 38,000 that served a valid llms.txt, 97% received zero requests for it during May 2026. Where requests did arrive, retrieval crawlers accounted for about 1% of them. The largest single share came from SEO audit tools checking whether the file existed, which is the industry inspecting itself rather than any system that generates answers.

Adoption itself is thinner than the discussion suggests. A separate SE Ranking study of nearly 300,000 domains found only 10.13% had an llms.txt file, and no citation benefit for the ones that did: dropping the variable from their model “actually improved its accuracy”, which led them to conclude the file “might currently introduce more noise than value”.

Google’s position matches the logs. In June 2026 John Mueller called the format “purely speculative for now”, noting that “the file has existed for years, yet none of the AI systems use it”. His recommendation was to wait for demand rather than pre-empt it: build one “when an AI platform that brings you clients complains that it needs the file for your site”. He named WebMCP, which exposes callable tools to in-browser agents, as the approach he finds more promising.

None of that makes the file harmful. It is small, it costs little to generate, and a format with no readers today is not the same as a format that will never have any. It does mean you should size the effort to the evidence: generate it automatically if you can, keep it accurate, and do not build a content programme around it or expect it to move citations on its own.

That is also why measurement matters more than the file. If you cannot prove a crawler reads your llms txt file, watch whether your brand facts appear more accurately in AI answers after you ship it, and treat that answer-level measurement as the real signal.

Maintain your llms txt file, or it goes stale and misleads agents

A stale or malformed file can mislead an agent just as badly as messy HTML does, so treat it like any other production asset.

We learned that the expensive way on this site. cloro’s llms.txt was hand-maintained, and when we audited it we found it listed 28 of 220 URLs, and 8 of those 28 pointed at slugs that now 301. An agent reading it got a stale, mostly-empty map of a site that had moved on. The file was deleted and replaced with one generated at build time from the sitemap, which currently emits 221 of 221 URLs and fails the build if it cannot.

Two details from that rebuild are worth stealing. Generate the list from your sitemap rather than by walking the built output, because the sitemap has already had noindex pages pruned from it, and a walk quietly hands agents your pagination and your unlisted pages. And fix the entry line shape as a contract, since anything parsing the file downstream depends on it; our own in-page search tool reads - [title](url): description, so an entry published without a description simply disappears from it.

The general rule: a hand-maintained llms.txt drifts to uselessness on any site that ships regularly, and the drift is invisible because nothing errors.

Validate the format before you deploy

Because the spec is strict about ordering, a small structural slip can break parsing. Run your file through an llms txt validator or checker, or simply confirm the sequence by hand: one H1, then a blockquote, then your H2 link lists. You should also verify that every link returns a real Markdown document, since a broken link is worse than a missing one.

Keep the shadow content in sync

Your Markdown twins can drift out of date the moment you change a price or rename a feature. Add the llms txt file and its linked Markdown pages to the same review process you already run for your main site. A quarterly audit is a reasonable baseline for most sites, and any change that touches pricing should trigger an immediate update.

Avoid the common mistakes

A handful of errors show up again and again in real files:

• Listing HTML pages instead of Markdown twins, which quietly defeats the whole purpose.
• Writing a vague blockquote that wastes the single prompt-like slot the format gives you.
• Dumping every URL on the site instead of curating the pages that genuinely matter.
• Forgetting the text/markdown content type, so agents receive the file as HTML anyway.

The business case for clean context

Four things justify the engineering hours.

The first is fewer hallucinations. Clean text drops the noise-to-signal ratio to near zero, so the AI doesn’t get confused by your cookie banner and decide you sell cookies. It reads your markdown and knows you sell software.

The second is citation authority. Perplexity and similar engines use RAG, and if their scraper can parse your content faster and cheaper than a competitor’s heavy React app, you get the citation.

Third, token economy: a 128k context window shouldn’t burn 50k tokens on HTML boilerplate. Serving Markdown packs more of your useful content into the model’s working memory.

The last one is the least certain. OpenAI, Anthropic, and Google are all looking for ways to cut web scraping costs, so crawlers that find an llms.txt will likely prioritize it because it saves them compute.

Robots.txt blocks crawlers, llms.txt guides them

These two files serve different masters.

Feature	robots.txt	llms.txt
Audience	Crawlers (Googlebot)	Agents (ChatGPT, Claude)
Function	Exclusion (Do not go here)	Inclusion (Read this first)
Format	Rules & Disallow paths	Markdown & Links
Goal	Indexing control	Context injection
Parsing	Machine logic	Semantic understanding

Don’t replace robots.txt. You still need it to block sensitive admin paths. llms.txt is an additive layer for the semantic web.

Monitoring agent behavior

Once llms.txt is live, how do you know it’s doing anything?

You need to track whether AI agents are hitting the file, and whether the data is showing up in their responses.

That’s where cloro fits in. Monitoring brand mentions lets you correlate the deployment of llms.txt with citation accuracy over time.

The feedback loop:

1. Deploy llms.txt.
2. Wait two weeks.
3. Check cloro for mention quality.
4. If hallucinations persist, refine the markdown descriptions.

Models are now reading the web as much as people are. llms.txt is how you make sure the version they read is accurate and clean.