Serving Markdown to Agents

Summary

HTML consumes 5-6x more tokens than Markdown for the same content. Content negotiation via Accept headers lets agents request Markdown, reducing their token budget by ~80%. An agent that reads five Markdown pages reads one HTML page for the same cost. Implementation varies by framework (Next.js middleware, Flask, etc.), but the pattern is consistent: check Accept header and return the best match.

• Markdown: ~3 tokens for "## Authentication", HTML: ~15 tokens
• HTTP Accept header: "Accept: text/markdown" vs "Accept: text/html"
• Fallback to HTML if Markdown not available
• Not all agents send Accept headers yet (optional optimization)
• Biggest impact on documentation-heavy systems

HTML costs tokens. A heading in HTML is <h2 class="text-2xl font-bold mt-8 mb-4">Authentication</h2> - roughly 15 tokens. The same heading in Markdown is ## Authentication - 3 tokens. Across a full documentation page, the reduction is consistent: serving Markdown instead of HTML cuts token consumption by approximately 80%.

For agents operating within context window budgets, this matters. An agent that can read five pages in Markdown reads one page in HTML for the same cost.

How Content Negotiation Works

HTTP content negotiation uses the Accept header. A browser sends:

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

An agent that prefers Markdown sends:

Accept: text/markdown, text/plain;q=0.9, */*;q=0.8

Your server reads the Accept header and returns the format with the highest priority that it can serve. If no matching format is available, it falls back to HTML.

Not all agents send Accept: text/markdown today. Many still accept */* and receive HTML. Content negotiation is an opt-in optimization - pages that serve it benefit agents that request it; all other clients receive HTML as before.

Next.js App Router Implementation

The cleanest approach in Next.js is a middleware rewrite that intercepts requests with Accept: text/markdown and routes them to a dedicated handler.

Middleware

// middleware.ts
import { NextRequest, NextResponse } from "next/server";

export function middleware(request: NextRequest) {
  const accept = request.headers.get("accept") ?? "";

  if (accept.includes("text/markdown")) {
    const url = request.nextUrl.clone();
    // Route to a markdown handler at /_md/[path]
    url.pathname = `/_md${request.nextUrl.pathname}`;
    return NextResponse.rewrite(url);
  }

  return NextResponse.next();
}

export const config = {
  matcher: ["/docs/:path*"],
};

Markdown Route Handler

// app/_md/docs/[...slug]/route.ts
import { source } from "@/lib/source";
import { notFound } from "next/navigation";

export async function GET(_request: Request, { params }: { params: { slug: string[] } }) {
  const page = source.getPage(params.slug);

  if (!page) {
    notFound();
  }

  // source.getPage returns the MDX source - strip frontmatter for clean Markdown
  const markdown = stripFrontmatter(page.data._raw.sourceFileContent);

  const tokenCount = estimateTokens(markdown);

  return new Response(markdown, {
    headers: {
      "Content-Type": "text/markdown; charset=utf-8",
      Vary: "Accept",
      "x-markdown-tokens": String(tokenCount),
      "Cache-Control": "public, max-age=3600",
    },
  });
}

function stripFrontmatter(content: string): string {
  return content.replace(/^---[\s\S]*?---\n/, "");
}

function estimateTokens(text: string): number {
  // Rough approximation: 1 token ≈ 4 characters for English prose
  return Math.ceil(text.length / 4);
}

Alternative: .md URL Suffix

A simpler approach that avoids middleware complexity: serve Markdown at .md-suffixed URLs. This is the convention used by llms.txt link lists.

// app/docs/[...slug]/route.ts - handles /docs/authentication.md
export async function GET(request: Request, { params }: { params: { slug: string[] } }) {
  const lastSegment = params.slug[params.slug.length - 1];

  if (!lastSegment.endsWith(".md")) {
    return new Response("Not found", { status: 404 });
  }

  const slug = [...params.slug.slice(0, -1), lastSegment.slice(0, -3)];

  const page = source.getPage(slug);

  if (!page) {
    return new Response("Not found", { status: 404 });
  }

  return new Response(page.data._raw.sourceFileContent, {
    headers: {
      "Content-Type": "text/markdown; charset=utf-8",
      Vary: "Accept",
    },
  });
}

Express / Node.js

import express from "express";
import { readDocPage } from "./docs";

const app = express();

app.get("/docs/:slug", async (req, res) => {
  const page = await readDocPage(req.params.slug);

  if (!page) {
    return res.status(404).json({ error: "Page not found" });
  }

  const accept = req.headers.accept ?? "";

  if (accept.includes("text/markdown")) {
    const markdown = page.markdown;
    const tokenCount = Math.ceil(markdown.length / 4);

    return res
      .status(200)
      .set({
        "Content-Type": "text/markdown; charset=utf-8",
        Vary: "Accept",
        "x-markdown-tokens": String(tokenCount),
      })
      .send(markdown);
  }

  return res.status(200).send(page.html);
});

Required Headers

Content-Type: text/markdown; charset=utf-8 - Identifies the response as Markdown. Use the registered MIME type. Do not use text/plain for Markdown responses - it loses semantic meaning.

Vary: Accept - Required for correct CDN caching. Without this header, a CDN may cache the Markdown response and serve it to browsers requesting HTML (or vice versa). Vary: Accept tells the CDN to cache separate copies for different Accept values.

x-markdown-tokens - An informal header carrying the estimated token count. Not standardized, but useful for agents that want to budget their context window before fetching. Omit if you do not compute it.

HTTP/2 200
Content-Type: text/markdown; charset=utf-8
Vary: Accept
x-markdown-tokens: 1847
Cache-Control: public, max-age=3600

Link Headers (RFC 8288)

The <link rel="alternate"> tag below only works when a client parses HTML. RFC 8288 defines an HTTP Link header carrying the same relation types, so any response - including the HTML page itself before a client has decided to fetch the Markdown - can advertise the alternate representation without a body to parse:

Link: </docs/authentication.md>; rel="alternate"; type="text/markdown"

Send the reverse relation on the Markdown response, so the HTML page stays the canonical URL:

Link: <https://docs.example.com/authentication>; rel="canonical"

Without a canonical link, a Markdown mirror looks like a duplicate page rather than an alternate representation of the HTML original - rel="canonical" tells search engines and agents which URL is authoritative when the same content is served at more than one path.

Locale and language negotiation

The same negotiation mechanics that pick a format also pick a language. Accept-Language (RFC 7231 §5.3.5) works exactly like Accept - a client sends its preferred locales with weights, and your server returns the closest match:

Accept-Language: fr-CA, fr;q=0.9, en;q=0.5

Three places carry locale for agent-facing content, and they need to agree:

• hreflang - <link rel="alternate" hreflang="fr-CA" href="https://example.com/fr-ca/docs/authentication"> on every localized page, pointing at every other locale of that same page plus an x-default fallback. This is how a client picks the right URL before fetching anything.
• Markdown routes - keep the locale segment in the path (/fr-ca/docs/authentication.md, not a query parameter or cookie), so a Markdown fetch resolves the same locale as the HTML page it mirrors, and so the URL alone (no session state) determines what's returned.
• llms.txt - publish a localized index per locale you support content in (/fr-ca/llms.txt) rather than one English index with mixed-language links, and cross-link each locale's index to the others.

Don't guess a locale from IP geolocation alone for agent traffic - many agents run from a single region's infrastructure regardless of the human they're acting for. Respect Accept-Language and an explicit path segment over inferred location.

Cloudflare Markdown for Agents

Cloudflare's "Markdown for Agents" feature performs automatic HTML-to-Markdown conversion at the edge. When enabled, Cloudflare intercepts requests with Accept: text/markdown, fetches the HTML origin response, converts it to Markdown, and returns the Markdown to the agent. Cloudflare docs also expose .md paths such as /index.md, which gives agents a URL fallback when they cannot set request headers.

This requires no changes to your origin server. Enable it in AI Crawl Control for the zone, or via the Cloudflare API by setting content_converter to on.

The response should include Content-Type: text/markdown; charset=utf-8, Vary: Accept, x-markdown-tokens, and a Content-Signal header. Cloudflare's default converted responses currently signal ai-train=yes, search=yes, ai-input=yes; override content-use policy where the platform allows it.

It only converts HTML origin responses, and Cloudflare documents a 2 MB origin response limit. Origin-served Markdown is preferable when you control the stack. Use edge conversion when modifying the origin is impractical. For arbitrary document conversion or dynamic pages, consider Workers AI AI.toMarkdown() or Browser Run's Markdown endpoint.

Agent-mode page view

A stripped-down structured view, served at a distinct URL or query parameter such as ?mode=agent, is a separate pattern from Markdown negotiation: plain content with direct links to the underlying endpoints, no navigation chrome, no client-side rendering dependency. There's no formal standard for it - it's a site-specific convention that Ora's agent-mode-view check looks for by name.

If you already serve Markdown via content negotiation, you usually don't need a second agent-mode view - it duplicates the same job through a different mechanism. Build one when the page's real value for an agent is a set of actions (direct links to sign up, get keys, or hit an endpoint) rather than prose content, since that's not something a Markdown conversion of the HTML expresses. Whichever mechanism you pick, keep it a single source of truth: an agent-mode view that quietly drifts from the canonical page's facts, prices, or links is worse than not having one, because nothing signals that it's stale.

HTML Fallback Signals

Even when you do not implement server-side content negotiation, you can signal Markdown availability in the HTML `<head>`:

<link
  rel="alternate"
  type="text/markdown"
  href="/docs/authentication.md"
  title="Authentication (Markdown)"
/>

This follows the same pattern as RSS feed discovery (<link rel="alternate" type="application/rss+xml">). Agents that look for alternate representations find the Markdown URL without having to guess or use Accept headers.

The "Copy for AI" Pattern

A growing pattern on developer documentation sites is a "Copy for AI" button that copies the page's Markdown source to the clipboard. Users paste it directly into their AI chat or IDE context.

// components/CopyForAI.tsx
'use client'

import { useState } from 'react'

interface CopyForAIProps {
  markdownUrl: string
}

export function CopyForAI({ markdownUrl }: CopyForAIProps) {
  const [state, setState] = useState<'idle' | 'loading' | 'copied' | 'error'>('idle')
  const [tokenCount, setTokenCount] = useState<number | null>(null)

  async function handleCopy() {
    setState('loading')

    const response = await fetch(markdownUrl, {
      headers: { Accept: 'text/markdown' },
    })

    if (!response.ok) {
      setState('error')
      return
    }

    const markdown = await response.text()
    const tokens = response.headers.get('x-markdown-tokens')

    if (tokens) {
      setTokenCount(Number(tokens))
    }

    await navigator.clipboard.writeText(markdown)
    setState('copied')
    setTimeout(() => setState('idle'), 2000)
  }

  return (
    <button onClick={handleCopy} disabled={state === 'loading'}>
      {state === 'copied'
        ? `Copied${tokenCount ? ` (~${tokenCount} tokens)` : ''}`
        : state === 'loading'
          ? 'Loading...'
          : 'Copy for AI'}
    </button>
  )
}

Displaying the token count alongside the copy confirmation sets expectations for the user. Someone pasting 25,000 tokens into a model with a 32K context window should know that before pasting.

What Not to Strip

When generating Markdown for agents, preserve:

• All code blocks with language identifiers
• Table content - agents parse Markdown tables directly
• Internal links - relative links are still useful for context
• Heading hierarchy - agents use this for navigation and summarization

Strip:

• Navigation menus and breadcrumbs
• Footer content (social links, legal notices)
• Cookie consent banners and modals
• Advertisement placeholders
• HTML-only interactive elements (accordions, tabs - flatten to headings and content)

Missing paths and representation parity

A missing page must return HTTP 404 rather than redirecting every unknown path to the homepage or returning a successful empty shell. For a Markdown request, provide a short explanation and links to the documentation index, search, or the nearest valid section. API failures should use the structured error contract. Verify status, media type, and recovery links on the deployed host.

If an explicit agent view or User-Agent-based Markdown variant exists, compare its facts, prices, links, and permissions with ordinary HTML. User-Agent strings are spoofable and must not grant authorization. Prefer explicit content negotiation where possible and test cache separation. Frontmatter on the Markdown response is optional but useful when the consuming client reads it: title, description, and a source-derived dateModified cover the common case. Never stamp dateModified as the current request time - derive it from the underlying content's actual last-modified date.