llms.txt Explained: What the llms txt File Does and How to Create One

llms.txt is a proposed web standard: a plain Markdown file at your site root that gives AI agents a curated, token-efficient map of your most important content.

For 30 years, the internet ran on robots.txt — a “Do Not Enter” sign for clumsy search spiders. In 2026, we’re dealing with readers, not just spiders. ChatGPT’s crawler, research agents, and RAG pipelines want to understand your content, and the modern web fights them with JavaScript bloat, popups, and cookie banners that waste tokens.

This guide covers what an llms txt file is, the exact structure the specification requires, a real example from Stripe, how to implement, generate, and validate one, and how it fits your Generative Engine Optimization (GEO) strategy.

The problem with HTML in the AI age

To see why llms.txt is needed, look at how LLMs actually “read.”

A traditional crawler like Googlebot scans for links and keywords and ignores the visual chrome. An AI agent (say, a RAG pipeline) is trying to ingest information, and the modern web is hostile to ingestion.

The token tax of the modern web breaks down into three buckets:

1. Boilerplate. Headers, footers, and navbars repeat on every page. An AI reading 10 pages reads your navbar 10 times. That wastes context-window space and money.
2. DOM noise. <div>, <span>, class names, and inline scripts are gibberish to an LLM trying to answer a question.
3. Visual vs. semantic content. Popups can obscure content, “Read More” buttons hide it, and an AI can’t really “click” anything.

The result is hallucination. When an AI scrapes a JavaScript-heavy page, it often gets a fragmented mess and fills in the blanks itself. That’s when it invents your pricing, your features, or your history.

llms.txt solves this by giving knowledge its own dedicated endpoint.

What is llms.txt exactly?

The llms.txt proposal (popularized by Jeremy Howard) is a convention for placing a file at the root of your domain, e.g. yourdomain.com/llms.txt.

It does two things:

1. Acts as a map. It tells AI agents where to find the “AI-ready” version of your site.
2. Provides context. A concise summary of who you are and what you do, injected straight into the model’s prompt.

Think of it as a sitemap for robots that read. You explicitly list the pages that matter, and you point the agent at clean Markdown instead of HTML.

The anatomy of an llms txt file

The format is defined in the official llms.txt specification, which Jeremy Howard published in September 2024. It is deliberately minimal, and it reuses Markdown so that both people and models can read the same file. The spec fixes the running order of the sections, yet only the very first heading is strictly required.

Example /llms.txt:

# cloro - AI Brand Monitoring Platform

> cloro is the leading platform for tracking brand visibility across Large Language Models (LLMs) like ChatGPT, Claude, and Perplexity.

## Key Pages

- [Pricing](/#pricing): current plans and query limits

The required order of sections

The specification asks you to assemble the file in this precise sequence:

1. An H1 line with the name of the project or site. This is the only mandatory element in the entire file.
2. A blockquote that summarises the entity in one or two sentences. It behaves like a short system prompt for your brand.
3. Zero or more Markdown paragraphs of extra context, carrying no headings, that flesh out the summary.
4. One or more H2 sections, each holding a Markdown list of [name](url) links with optional notes after a colon.

The special “Optional” section

The spec reserves one H2 named Optional for links that a model can safely skip. When an agent is short on context-window budget, it drops the Optional block first and keeps the core pages. Treat that section as your overflow shelf rather than your main catalogue.

Why every link should point at Markdown

Each link in a file list should resolve to a clean .md or .txt version of the page, not the HTML original. The whole value of an llms txt file collapses when the links send an agent straight back into the boilerplate you were trying to avoid. Publish a Markdown twin for every page you list, and then point the link at that twin.

A real llms txt example: how Stripe structures its file

Abstract rules are easier to follow with a production example in front of you. Stripe publishes a live file at docs.stripe.com/llms.txt that follows the spec closely. It opens with an H1 (# Stripe Documentation), then uses a blockquote to hand the model explicit integration guidance before any links appear.

That guidance is the interesting part. Stripe’s blockquote tells agents which APIs to prefer, such as the Checkout Sessions API over the older Charges API, and which endpoints to avoid. Below it, the file groups links under product-named H2 sections like Payments, Billing, Checkout, and Connect. Every entry is a Markdown link that points directly at a documentation page.

The lesson for your own llms txt file is that the blockquote is not throwaway marketing copy. It is the one place where you can steer how a model describes your product, so spend those words on the facts you most want repeated correctly.

How to implement llms.txt

You don’t need a site redesign. You’re creating a shadow site of text files alongside the existing one.

Step 1: Create your shadow content

Convert your key pages into Markdown to strip the HTML noise.

Your pricing.html might be 50kb of code. The equivalent pricing.md should be 2kb of text.

Example pricing.md:

# Pricing Plans

## Hobby Plan

- Cost: $29/month
- Features: 500 queries, Daily updates.

## Business Plan

- Cost: $99/month
- Features: 5,000 queries, Hourly updates.

Step 2: Consolidate into llms-full.txt

Several proposals suggest a single large text file (llms-full.txt) containing all your core documentation concatenated together. RAG systems prefer fetching one file: fewer HTTP requests, and the model gets the full context in a single pass.

Step 3: Deploy the root file

Place llms.txt at your root. Make sure your server returns text/markdown or text/plain headers.

Step 4: Advertise it

Auto-discovery is still evolving. In the meantime, feed the URL manually to custom GPTs, Claude Projects, and other agents to “train” them on your documentation.

Tools to generate llms.txt

If writing these files by hand feels tedious, a few llms txt generators will crawl your site and produce the Markdown structure for you.

• Keploy. One-click generator that scans a URL and builds the file. Fine for simple sites.
• Writesonic. Structured text generator aimed at LLM training and inference.
• Gushwork. More granular control over which site areas to include or exclude.
• Fibr AI. Generates a file with explicit permissions for bots like GPTBot and ClaudeBot.

These tools are useful for a first pass, but review the output by hand. The shadow content for your most critical pages needs to be accurate.

llms.txt vs llms-full.txt: which file should you ship?

People conflate these two files, yet they play different roles. The base llms.txt is an index: a short map of links that an agent can crawl selectively. A companion file, usually named llms-full.txt, concatenates the actual Markdown content of those pages into one long document.

The trade-off is context versus control. An llms-full.txt lets a RAG pipeline pull your entire documentation set in a single request, which is efficient when the model has a large context window to spare. The downside is size, because a full file for a big documentation site can run to hundreds of thousands of tokens that many agents will simply truncate.

A safe default is to ship both files. Publish the compact llms.txt index for agents that fetch selectively, and offer llms-full.txt for the ones that want everything at once. If you only have budget for one, start with the index, because it degrades more gracefully on small-context models.

Who publishes an llms txt file — and do AI crawlers read it?

Adoption so far is led by developer-tools and documentation-heavy companies. Stripe, as shown above, ships a detailed file, and many API-first products have followed with their own documentation indexes. Documentation platforms have also begun generating the file automatically, which is why so many software sites now expose one at their root.

The honest caveat sits on the consumption side. As of mid-2026, no major AI provider has publicly confirmed that its production crawler treats llms.txt as a retrieval or ranking signal, and some search engineers have voiced open scepticism about its value. So treat the file as a low-cost bet on where agent tooling is heading, not as a guaranteed pipeline into ChatGPT or Gemini.

That uncertainty is exactly why measurement matters. If you cannot yet prove that a crawler reads your llms txt file, the next best thing is to watch whether your brand facts appear more accurately in AI answers after you ship it.

How to validate and maintain your llms txt file

Shipping the file is not the end of the job. A stale or malformed file can mislead an agent just as badly as messy HTML, so treat it like any other production asset.

Validate the format before you deploy

Because the spec is strict about ordering, a small structural slip can break parsing. Run your file through an llms txt validator or checker, or simply confirm the sequence by hand: one H1, then a blockquote, then your H2 link lists. You should also verify that every link returns a real Markdown document, since a broken link is worse than a missing one.

Keep the shadow content in sync

Your Markdown twins can drift out of date the moment you change a price or rename a feature. Add the llms txt file and its linked Markdown pages to the same review process you already run for your main site. A quarterly audit is a reasonable baseline for most sites, and any change that touches pricing should trigger an immediate update.

Avoid the common mistakes

A handful of errors show up again and again in real files:

• Listing HTML pages instead of Markdown twins, which quietly defeats the whole purpose.
• Writing a vague blockquote that wastes the single prompt-like slot the format gives you.
• Dumping every URL on the site instead of curating the pages that genuinely matter.
• Forgetting the text/markdown content type, so agents receive the file as HTML anyway.

The business case for clean context

Why spend engineering hours on this?

1. Fewer hallucinations. Clean text drops the noise-to-signal ratio to near zero. The AI doesn’t get confused by your cookie banner and decide you sell cookies. It reads your markdown and knows you sell software.

2. Better citation authority. Perplexity and similar engines use RAG. If their scraper can parse your content faster and cheaper than a competitor’s heavy React app, you get the citation.

3. Token economy. A 128k context window shouldn’t burn 50k tokens on HTML boilerplate. Serving Markdown packs more of your useful content into the model’s working memory.

4. Future-proofing. OpenAI, Anthropic, and Google are all looking for ways to cut web scraping costs. Crawlers that find an llms.txt will likely prioritize it because it saves them compute.

Robots.txt vs LLMs.txt

These two files serve different masters.

Feature	robots.txt	llms.txt
Audience	Crawlers (Googlebot)	Agents (ChatGPT, Claude)
Function	Exclusion (Do not go here)	Inclusion (Read this first)
Format	Rules & Disallow paths	Markdown & Links
Goal	Indexing control	Context injection
Parsing	Machine logic	Semantic understanding

Don’t replace robots.txt. You still need it to block sensitive admin paths. llms.txt is an additive layer for the semantic web.

Monitoring agent behavior

Once llms.txt is live, how do you know it’s doing anything?

You need to track whether AI agents are hitting the file, and whether the data is showing up in their responses.

That’s where cloro fits in. Monitoring brand mentions lets you correlate the deployment of llms.txt with citation accuracy over time.

The feedback loop:

1. Deploy llms.txt.
2. Wait two weeks.
3. Check cloro for mention quality.
4. If hallucinations persist, refine the markdown descriptions.

The web is shifting from a library of documents to a training set for models. llms.txt is how you make sure your entry in that set is accurate and clean.