Summary

llms.txt is a standard file served at /llms.txt that publishes a curated, token-efficient index of your site. Rather than agents crawling your entire documentation, llms.txt lets you declare upfront what matters most. The confirmed consumers are narrow: IDE and coding-agent workflows - Cursor, Claude Code, and similar tools - that index the file during project setup or on demand. General-purpose crawlers and inference-time browsing largely skip it; see Adoption and Consumption below.

Served at /llms.txt as text/plain; charset=utf-8
Required: H1 with product/site name
Recommended: blockquote summary, H2 sections with links and descriptions
Link format: - Title: description
Total size under 5,000 for efficiency
Use ## Optional section for lower-priority content
Worth shipping for documentation and developer-tool sites; not a general-site tactic

The llms.txt specification, proposed by Jeremy Howard in September 2024 at llmstxt.org, gives sites a standard location to publish a curated, token-efficient index of their content. Where a describes every URL for crawlers, llms.txt describes the most important content for - with enough context to navigate without crawling the full site.

The Problem It Solves

An AI agent trying to use your or understand your product faces a practical constraint: are finite. Crawling your entire documentation site could cost hundreds of thousands of tokens before the agent answers a single question. llms.txt moves the curation work to the publisher, letting you declare in advance what an AI needs to know and where to find it.

The confirmed value is narrow but real: IDE and coding-agent workflows - Cursor, Claude Code, and similar tools - fetch llms.txt on demand when indexing a documentation or developer-tool site during project setup. That is the use case this page's guidance is written for.

Canonical Format

llms.txt is served at the root path (/llms.txt) as text/plain; charset=utf-8. The format is a subset of with a specific structure:

# Product Name

> One-paragraph summary of what this product is and who it's for.

## Section Name

- [Page Title](https://example.com/page.md): Short description of what this page covers.
- [Page Title](https://example.com/other.md): Short description.

## Another Section

- [Page Title](https://example.com/more.md): Short description.

## Optional

- [Less Critical Page](https://example.com/optional.md): Lower-priority content.

The only required element is the H1 (#) with the product or site name. Everything else is recommended but optional.

H1 - Required. Site or product name. One per file.

Blockquote summary - Recommended. One paragraph describing the product and its intended users. Appears directly below the H1.

H2 sections - Group related links. Section names are free-form.

Link list items - Each item is [Title](URL): Description. The colon-and-description suffix is optional but strongly recommended. Descriptions help agents prioritize which links to fetch.

## Optional section - A specially named section that agents can skip when operating under context pressure. Put supplementary content - changelog entries, older tutorials, edge-case reference - in this section.

The .md URL Convention

Links in llms.txt should point to the Markdown version of each page, not the HTML version. The convention is appending .md to the URL path:

https://docs.example.com/authentication       → HTML page for browsers
https://docs.example.com/authentication.md    → Markdown for agents

If your site does not yet serve .md endpoints, link to the HTML pages as a fallback. See Content Negotiation for how to implement Markdown serving.

Token Budget

Keep the full llms.txt under 5,000 tokens. This fits comfortably in a single context window turn and ensures agents can load the entire index at once without truncation.

The Anthropic llms.txt is a useful reference point: their index file is approximately 8,364 tokens and their llms-full.txt is approximately 481,349 tokens. Both extremes demonstrate what is possible, but the index should be the compact version - agents fetch the full content when they need depth.

Real Examples

Anthropic publishes two files:

/llms.txt - 8,364-token index with sections for Claude models, APIs, and product features
/llms-full.txt - 481,349-token file with all documentation content inlined

Stripe publishes three files across their domains:

stripe.com/llms.txt - product overview and marketing content
docs.stripe.com/llms.txt - full API and integration documentation
support.stripe.com/llms.txt - help center and troubleshooting content

Stripe also uses an ## Instructions section (see below) to give behavioral guidance to AI agents about how to handle Stripe-specific concepts.

Vercel organizes sections by product area:

Deployment, Edge Functions, Storage, Observability, and each appear as separate H2 sections
Each link includes a description that differentiates it from adjacent pages
The file is kept under 3,000 tokens despite covering dozens of products

The Instructions Section

Stripe's pattern of adding an ## Instructions section is worth adopting. Unlike a pure index, this section gives behavioral guidance to AI agents:

## Instructions

- When discussing pricing, always refer users to the current pricing page rather than quoting figures directly.
- The API uses camelCase for JSON keys in requests and snake_case in webhook payloads - these are not inconsistencies.
- Authentication is always via Bearer token in the Authorization header, never via query parameters.
- Rate limit errors use HTTP 429 with a Retry-After header. Always surface this to the user.

Agents that read this section have the directives in context when answering; treat it as steering, not enforcement. This is particularly useful for:

Correcting common misconceptions agents develop from training data
Flagging which information goes stale quickly (pricing, feature availability)
Clarifying naming conventions and terminology that conflict with common usage
Directing agents to authoritative sources for specific topics

llms-full.txt

llms-full.txt is the companion to llms.txt. Where the index links to pages, the full file inlines all content:

# Product Name

> Summary of the product.

## Authentication

### API Keys

[Full content of the authentication page...]

### OAuth 2.0

[Full content of the OAuth page...]

llms-full.txt serves two distinct use cases:

pipelines - Teams building internal knowledge bases chunk and embed llms-full.txt rather than crawling and parsing HTML. A single well-structured file is easier to process than hundreds of HTML pages.

IDE indexing - Tools like Cursor and Claude Code download llms-full.txt once and index it locally. Subsequent queries against your documentation happen without network round trips.

Maintain llms-full.txt as a generated artifact. Inlining content manually does not scale. See Generation Tools below.

Serving Requirements

Serve both files as text/plain; charset=utf-8. Do not use text/html or text/markdown. Some agents and tools check the Content-Type header and will reject files served under the wrong type.

Content-Type: text/plain; charset=utf-8
Cache-Control: public, max-age=86400

A 24-hour cache is appropriate since llms.txt changes infrequently. If your documentation deploys frequently, use a shorter TTL or add a Last-Modified header so agents can revalidate efficiently.

What Lighthouse Checks

Lighthouse's llms-txt audit verifies the file is actually usable, not just present (source):

The file must be fetchable. A failed fetch or a 5xx response fails the audit outright; a 4xx response marks it not applicable.
The content must include a Markdown H1 (# Title).
The content must include at least one Markdown link.
The content must be at least 50 characters long.

These map directly onto the canonical format above: the required H1, and a link list with real entries rather than a placeholder file.

See Referencing llms.txt from robots.txt for the comment-line convention some crawlers recognize.

Generation Tools

Most documentation platforms now generate llms.txt automatically:

Mintlify - generates llms.txt and llms-full.txt from your docs config. No additional configuration required.

Fern - generates llms.txt alongside documentation. Configurable section grouping.

GitBook - generates llms.txt from your space's table of contents.

Docusaurus - community plugin docusaurus-plugin-llms-txt generates both files at build time.

WordPress - plugins available for major documentation themes.

Next.js - implement as a route handler:

// app/llms.txt/route.ts
import { source } from "@/lib/source";

export async function GET() {
  const pages = source.getPages();

  const sections = groupPagesBySection(pages);

  const lines = ["# Your Product Name", "", "> What your product does and who it serves.", ""];

  for (const [section, sectionPages] of Object.entries(sections)) {
    lines.push(`## ${section}`, "");
    for (const page of sectionPages) {
      lines.push(`- [${page.data.title}](${page.url}.md): ${page.data.description}`);
    }
    lines.push("");
  }

  return new Response(lines.join("\n"), {
    headers: {
      "Content-Type": "text/plain; charset=utf-8",
      "Cache-Control": "public, max-age=86400",
    },
  });
}

Directories

If you want to verify your file is discoverable:

llmstxthub.com - directory of submitted llms.txt files with category browsing
llms-txt-hub - GitHub-based registry with programmatic access
llms-txt.io - validation tool that checks format and token count

Submission to directories is optional. The primary value of llms.txt comes from agents discovering the file at the standard path, not from directory listings.

Adoption and Consumption

Few agents actually read llms.txt. General-purpose crawlers and inference-time browsing do not use it:

An Ahrefs study of ~137,000 sites found that roughly 28% publish an llms.txt file, but 97% of the valid files received zero requests during May 2026.
The major AI crawlers - GPTBot, ClaudeBot, PerplexityBot - skip llms.txt entirely.
Google's John Mueller compared llms.txt to the keywords meta tag, and Google's own AI-optimization guidance states that files like this are not needed or used for its systems.

So llms.txt is not an answer-engine-optimization (AEO) strategy for a general site. Treat any claim that it improves visibility in AI Overviews or chatbot answers with skepticism.

The genuine, confirmed value is narrow: coding agents and IDE assistants - Cursor, Claude Code, and similar tools - fetch llms.txt on demand when indexing a documentation or developer-tool site. That on-demand consumption during a developer's task is real, and it is the reason this site still ships an llms.txt.

Recommendation: worth shipping for documentation and developer-tool sites - it is cheap to generate and there is real coding-agent consumption. Do not ship it expecting an AEO benefit on a general marketing or content site; there is no evidence it changes how answer engines see you.

Modular, per-section indexes

For a large site, publish section-level llms.txt files alongside the root index - /docs/llms.txt for documentation, /api/llms.txt for API reference - so an agent working in one area loads a focused index instead of the whole site. Link each section file from the root index and verify both levels independently. Ora's modular-llms-txt check looks for this pattern by name: "Per-section llms.txt files (like /docs/llms.txt) so an agent working in one area gets a focused index instead of your whole site" (ora.ai/api/checks).

Keep authoritative prices in sync across visible pages and a pricing.md representation when useful. See Commercial and Entity Discovery for , , and validation.

On this page