---
title: "Agent Discovery Checklist"
description: "A tiered checklist for making your site and API discoverable and consumable by AI agents"
url: "https://agentsurface.dev/docs/discovery/aeo-checklist"
lastVerified: 2026-09-25
lastModified: 2026-09-25T05:46:01.000Z
---



## Summary [#summary]

This tiered checklist organizes agent-discovery work from essential foundation items through comprehensive support. Tier 1 covers robots.txt, llms.txt, and OpenGraph tags - low effort, high impact. Later tiers add JSON-LD, Markdown content negotiation, Content Signals, OpenAPI links, API Catalog, MCP Server Cards, Agent Skills indexes, OAuth metadata, Web Bot Auth, and commerce protocols where relevant.

* Tier 1: robots.txt, llms.txt, OpenGraph meta tags
* Tier 2: OpenAPI discovery, JSON-LD markup
* Tier 3: Content negotiation, content structure, AGENTS.md
* Tier 4: Well-known endpoints, API Catalog, MCP Server Cards, Agent Skills, auth discovery
* Tier 5: AGENTS.md depth, llms-full.txt, Copy for AI, token count disclosure, analytics, changelog, status, agent terms of use
* Tier 6: NLWeb /ask endpoint, /mcp endpoint, agent-mode page view, Content Signals
* Start with Tier 1 before advancing

This checklist covers the full discovery surface organized by tier. Tier 1 items are the minimum viable foundation - ship these first. Later tiers add depth and capability for teams investing seriously in agent discoverability.

Each item links to the relevant section of this documentation where applicable. This is a checklist, not a scoring system - completing a tier means the items are done, not that you have reached a certified level.

## Tier 1 - Foundation [#tier-1---foundation]

These items cost little to implement and make later discovery work easier to verify. Complete these before anything else.

### robots.txt [#robotstxt]

* [ ] `robots.txt` exists at `/robots.txt` and is served with `Content-Type: text/plain`
* [ ] `User-agent: *` with `Allow: /` covers all legitimate bots
* [ ] Training crawlers explicitly disallowed: `GPTBot`, `ClaudeBot`, `Google-Extended`, `Applebot-Extended`, `cohere-ai`, `CCBot` (or explicitly allowed - the decision is made and documented)
* [ ] Retrieval bots allowed: `OAI-SearchBot`, `ChatGPT-User`, `Claude-SearchBot`, `Claude-User`, `PerplexityBot`, `Perplexity-User`, `Googlebot` are not blocked
* [ ] Search, Agent, and Training access are decided as three separate questions, not one blanket toggle - if the host is Cloudflare, its three-way classification and Bot Preference Sync are configured; otherwise the equivalent per-bot rules are hand-maintained
* [ ] `Sitemap:` directive points to your primary sitemap
* [ ] `/robots.txt` is tested with [Google Search Console's robots.txt report](https://support.google.com/webmasters/answer/6062598)

See [robots.txt for AI Agents](./robots-txt).

### llms.txt [#llmstxt]

* [ ] `/llms.txt` exists and is served as `Content-Type: text/plain; charset=utf-8`
* [ ] File begins with `# Product Name` (H1 required)
* [ ] Blockquote summary follows the H1 (`> Description here`)
* [ ] All significant documentation sections appear as H2 groups with link lists
* [ ] Each link includes a colon-separated description: `- [Title](url): Description`
* [ ] Links point to `.md` URLs where available, HTML otherwise
* [ ] Total file size is under 5,000 tokens (approximately 20,000 characters)
* [ ] `## Optional` section used for supplementary content

See [llms.txt](./llms-txt).

### OpenGraph Meta Tags [#opengraph-meta-tags]

* [ ] `og:title` is entity-first: "Product Name - Page Topic"
* [ ] `og:description` is 150–300 characters, factual, answer-dense
* [ ] `article:modified_time` is present and accurate (W3C datetime format)
* [ ] `article:published_time` is present
* [ ] `article:tag` values are present for topic categorization

### Sitemap [#sitemap]

* [ ] XML sitemap exists at `/sitemap.xml` or a sitemap index at `/sitemap.xml` referencing sub-sitemaps
* [ ] Every documentation page appears in the sitemap
* [ ] `<lastmod>` is present and accurate for every URL
* [ ] `<lastmod>` format is W3C datetime: `YYYY-MM-DD` or `YYYY-MM-DDTHH:MM:SSZ`
* [ ] Sitemap contains fewer than 50,000 URLs (use sitemap index if larger)
* [ ] Sitemap is referenced in `robots.txt`

### Semantic HTML [#semantic-html]

* [ ] Main content is wrapped in `<article>` or `<main>`
* [ ] Navigation is in `<nav>` with `aria-label`
* [ ] Headings use semantic tags (`h1`–`h6`), not styled `<div>` elements
* [ ] Page has exactly one `<h1>` matching the `<title>` tag
* [ ] `<time datetime="...">` used for dates

***

## Tier 2 - Content Quality [#tier-2---content-quality]

Content structure improvements that directly affect whether agents extract accurate answers.

### Answer-First Structure [#answer-first-structure]

* [ ] Every page puts the direct answer or key fact in the first 200 words
* [ ] Introductory paragraphs do not delay the answer with background or context
* [ ] Tables appear before prose explanations of the same data
* [ ] Rate limits, pricing, and other quantitative facts appear as early as possible

See [Content Structure for AI Consumption](./content-structure).

### Self-Contained Sections [#self-contained-sections]

* [ ] Each H2 section can be read and understood without context from preceding sections
* [ ] No section references "the previous step" or "as described above" without restating what was described
* [ ] Code examples include all necessary imports and context
* [ ] Error messages are quoted verbatim with exact capitalization and punctuation

### Markdown Availability [#markdown-availability]

* [ ] Documentation pages are available in Markdown format at `.md` URLs or via `Accept: text/markdown`
* [ ] `<link rel="alternate" type="text/markdown" href="...">` present in HTML `<head>`
* [ ] Markdown output strips navigation and footer, preserves content
* [ ] For localized content: `hreflang` alternates are present, the locale is a path segment on Markdown routes (not a cookie or query parameter), and each locale has its own `llms.txt`

See [Serving Markdown to Agents](./content-negotiation).

### Vary Header [#vary-header]

* [ ] All responses that differ by `Accept` header include `Vary: Accept`
* [ ] CDN configuration honors `Vary: Accept` (separate cache entries per format)
* [ ] `Content-Type: text/markdown; charset=utf-8` used for Markdown responses (not `text/plain`)

***

## Tier 3 - Structured Data [#tier-3---structured-data]

JSON-LD schema markup for AI Overview inclusion and retrieval system accuracy.

### FAQPage Schema [#faqpage-schema]

Google dropped FAQ rich results from Search in May 2026, so this schema no longer earns Google search presentation. Keep it only for the agents and other parsers that still read `FAQPage` JSON-LD directly.

* [ ] Pages with FAQ or Q\&A content include `FAQPage` JSON-LD
* [ ] Every `Question.name` exactly matches visible heading text
* [ ] Every `Answer.text` exactly matches visible answer text
* [ ] No schema content exists that is not visible on the page
* [ ] Schema validated with the [Schema.org Validator](https://validator.schema.org)

See [Structured Data for Agents](./structured-data).

### TechArticle Schema [#techarticle-schema]

* [ ] All technical documentation pages include `TechArticle` JSON-LD
* [ ] `dateModified` is accurate and kept current
* [ ] `author` and `publisher` are present
* [ ] `proficiencyLevel` is set (`Beginner` or `Expert`)

### HowTo Schema [#howto-schema]

Google dropped `HowTo` rich results from Search in September 2023. Keep the schema for parsers other than Google Search; do not expect it to earn a Google rich result.

* [ ] Step-by-step guides and tutorials include `HowTo` JSON-LD
* [ ] Each step includes `position`, `name`, `text`, and `url` (anchored to heading)
* [ ] `totalTime` is present in ISO 8601 duration format
* [ ] `tool` array lists prerequisites

### WebAPI Schema [#webapi-schema]

* [ ] API reference pages include `WebAPI` JSON-LD
* [ ] `documentation`, `endpointUrl`, and `provider` are present
* [ ] `version` reflects the current API version

### SoftwareApplication Schema [#softwareapplication-schema]

* [ ] Product and tool pages include `SoftwareApplication` JSON-LD
* [ ] `softwareVersion` is kept current
* [ ] `downloadUrl` and `codeRepository` present where applicable

***

## Tier 4 - Discovery Endpoints [#tier-4---discovery-endpoints]

Machine-readable capability declarations at standard paths.

### /.well-known/ai [#well-knownai]

* [ ] `/.well-known/ai` exists and is valid JSON
* [ ] `capabilities` array accurately reflects what you support
* [ ] `api.spec` points to a valid OpenAPI JSON file
* [ ] `mcp.endpoint` present if you run an MCP server
* [ ] `llms.index` points to your `llms.txt`

See [Well-Known Discovery Endpoints](./well-known-endpoints).

### OpenAPI Specification [#openapi-specification]

* [ ] OpenAPI spec available at `/openapi.json` or `/api/openapi.json`
* [ ] Spec validates against OpenAPI 3.1 schema
* [ ] Every operation has a `summary` and `description`
* [ ] Every parameter has a `description`
* [ ] Every response has a `description` and schema
* [ ] Spec is kept in sync with actual API behavior (tested in CI)

### API Catalog [#api-catalog]

* [ ] `/.well-known/api-catalog` exists if you expose public APIs
* [ ] Catalog links to canonical OpenAPI specs, documentation, status pages, and auth docs
* [ ] HTTP `Link: </.well-known/api-catalog>; rel="api-catalog"` header present where practical
* [ ] Catalog omits internal-only APIs that agents should not discover

### /.well-known/agent-card.json [#well-knownagent-cardjson]

* [ ] `agent-card.json` published if your service can be delegated to by other agents
* [ ] `skills` array describes what the agent can do with realistic `examples`
* [ ] `authentication` specifies how to obtain credentials
* [ ] `capabilities.streaming` accurately reflects whether you support streaming

### MCP Server Card [#mcp-server-card]

* [ ] `/.well-known/mcp/server-card.json` published if you run an HTTP MCP server
* [ ] `transport` correctly specifies `streamable-http` or `stdio`
* [ ] `authentication` section accurate and complete
* [ ] Card describes the server before connection: name, purpose, tools, transport, and auth
* [ ] `/.well-known/mcp.json` redirects or points to the server card if used for compatibility

### Agent Skills Index [#agent-skills-index]

* [ ] `/.well-known/agent-skills/index.json` exists when your site teaches task-specific agent workflows
* [ ] Index declares the Agent Skills discovery v0.2 schema
* [ ] Every entry has `name`, `type`, activation-oriented `description`, canonical `url`, and `sha256:` `digest`
* [ ] Each skill is narrow, procedural, and linked as a separate `SKILL.md`
* [ ] Linked skills follow the Agent Skills format: required `name` and `description`, optional `scripts/`, `references/`, and `assets/`
* [ ] Each digest matches the exact bytes returned by the corresponding skill URL
* [ ] Skill descriptions explain when to use the skill, not only what topic it covers
* [ ] Skills are generated from canonical docs or reviewed with the same rigor as public docs

### OAuth Protected Resource Metadata [#oauth-protected-resource-metadata]

* [ ] `/.well-known/oauth-protected-resource` exists for OAuth-protected APIs
* [ ] Metadata points to the correct authorization server, JWKS, supported scopes, and DPoP requirement
* [ ] Protected resource metadata is linked from API docs and MCP metadata where applicable

### Web Bot Auth [#web-bot-auth]

* [ ] `/.well-known/http-message-signatures-directory` exists if your service sends signed bot or agent requests
* [ ] Published keys are rotated and documented
* [ ] Receiving services can verify signed requests before granting high-trust access

***

## Tier 5 - Developer Experience [#tier-5---developer-experience]

Improvements for developers using AI coding assistants with your APIs and SDKs.

### AGENTS.md [#agentsmd]

* [ ] `AGENTS.md` exists at repository root
* [ ] `## Commands` section lists all essential commands (build, test, lint, dev server)
* [ ] Non-standard tooling (unusual package manager, monorepo setup) documented
* [ ] `## Agent Boundaries` section with three tiers: autonomous / ask first / never
* [ ] Testing rules stated explicitly: framework, location, required coverage
* [ ] File is under 500 lines
* [ ] No prose paragraphs - only lists, code blocks, tables

See [AGENTS.md and Context Files](./agents-md).

### llms-full.txt [#llms-fulltxt]

* [ ] `/llms-full.txt` exists with all documentation content inlined
* [ ] File is generated automatically from source content (not maintained by hand)
* [ ] `Content-Type: text/plain; charset=utf-8`
* [ ] `Cache-Control` TTL is appropriate for how frequently docs change
* [ ] Token count documented (in comments or referenced from `llms.txt`)

### Copy for AI Button [#copy-for-ai-button]

* [ ] Documentation pages include a "Copy for AI" button that copies Markdown to clipboard
* [ ] Button displays estimated token count after copy
* [ ] Markdown copy excludes navigation and footer content

### Token Count Disclosure [#token-count-disclosure]

* [ ] `x-markdown-tokens` header present on Markdown responses
* [ ] Token counts visible to developers (in UI, in response headers, or in `llms.txt`)
* [ ] Pages over their type's hard limit in [Content Structure](./content-structure) are flagged for splitting

### Analytics [#analytics]

* [ ] AI bot traffic tracked separately from human traffic (by user-agent)
* [ ] Requests carrying a verified Web Bot Auth signature are distinguished from requests that merely claim a matching user-agent string
* [ ] Which pages retrieval bots visit most frequently is visible
* [ ] `Retry-After` header compliance monitored (are bots respecting rate limits?)

### Changelog, Status, and Agent Terms of Use [#changelog-status-and-agent-terms-of-use]

* [ ] A dated changelog exists (page or RSS/Atom feed) so an agent already holding your docs in context can detect drift with one small fetch
* [ ] Changelog entries name what changed and why it matters, not just that a version shipped
* [ ] A public status page exists and is linked from the API Catalog entry, so an agent can tell an outage from its own error before deciding to retry
* [ ] Agent-facing terms of use are published as their own page - what automated or unattended use is actually permitted - and linked from `llms.txt` and API docs, not left implicit in `robots.txt` or buried in general Terms of Service

See [Content Structure](./content-structure) and [Commercial and Entity Discovery](./commercial-and-entity-discovery).

***

## Tier 6 - Advanced [#tier-6---advanced]

High-investment items for teams deeply committed to agent discoverability.

### NLWeb /ask Endpoint [#nlweb-ask-endpoint]

* [ ] `/ask` endpoint returns direct answers to natural-language questions
* [ ] Endpoint accessible as both HTTP GET and as an MCP server
* [ ] Response includes `sources` array with URLs and excerpts
* [ ] Indexed against current documentation content (not stale snapshots)

See [Well-Known Discovery Endpoints](./well-known-endpoints).

### /mcp Endpoint [#mcp-endpoint]

* [ ] MCP server running and accessible to agents
* [ ] All API operations exposed as MCP tools with accurate descriptions
* [ ] Resource endpoints serve documentation and configuration data
* [ ] Authentication works with the flow documented for your agent clients, usually OAuth 2.1 Client Credentials, token exchange, or host-managed delegated auth
* [ ] Tool definitions tested with MCP inspector
* [ ] Health check endpoint for uptime monitoring

See the [MCP Servers](/docs/mcp-servers) section.

### Agent-Mode Page View [#agent-mode-page-view]

* [ ] A stripped structured view exists only where it does a job Markdown negotiation doesn't - direct action links rather than prose content
* [ ] The view is served at a distinct, documented URL or parameter (e.g. `?mode=agent`), not guessed
* [ ] Facts, prices, and links in the agent-mode view are checked against the canonical page - it's not a second, silently drifting source of truth

See [Serving Markdown to Agents](./content-negotiation).

### Content Signals [#content-signals]

* [ ] `Content-Signal: search=yes, ai-input=yes, ai-train=no` declared in `robots.txt` or response headers
* [ ] Policy header overridden appropriately for gated content
* [ ] `robots.txt` and Content Signals policy are consistent with each other
* [ ] If a price or specific licence condition (not just yes/no) matters, RSL is attached via `robots.txt`, an HTTP header, HTML, or the RSS feed - Content Signals alone can't express a price
* [ ] IETF aipref (`draft-ietf-aipref-vocab`, `draft-ietf-aipref-attach`) is tracked, not implemented as a requirement - both are still pre-consensus drafts

See [robots.txt for AI Agents](./robots-txt).

### Instructions Section in llms.txt [#instructions-section-in-llmstxt]

* [ ] `## Instructions` section added to `llms.txt`
* [ ] Instructions correct known misconceptions agents have from training data
* [ ] Instructions flag rapidly-changing content (pricing, feature availability)
* [ ] Instructions clarify naming conventions that conflict with industry defaults
* [ ] Instructions point to authoritative sources for high-stakes questions

See [llms.txt](./llms-txt).

### agents.json / Wildcard AI Flows [#agentsjson--wildcard-ai-flows]

* [ ] `agents.json` at root describes common workflows with step-by-step actions
* [ ] Workflows cover the top 3–5 agent use cases for your product
* [ ] Each flow includes `success_output` and `failure_output` descriptions
* [ ] Flows tested with an agent to verify they execute correctly

See [Well-Known Discovery Endpoints](./well-known-endpoints).

### Agentic Commerce [#agentic-commerce]

* [ ] If agent-mediated commerce is a real product surface, see [Agentic Commerce](/docs/agentic-commerce) for product data, checkout protocols, payments, and trust and risk - not covered on this checklist

### Agentic Search Measurement [#agentic-search-measurement]

* [ ] A fixed set of brand-name queries is run across assistants (ChatGPT, Claude, Gemini, Perplexity) on a regular cadence, and your domain appears in the top results
* [ ] A fixed set of category or use-case queries ("a tool that does X") is run across the same assistants, tracking whether you are mentioned at all, not only your rank when named directly
* [ ] An assistant that already surfaces your brand can locate your developer resources (API docs, OpenAPI spec, MCP server, auth docs, SDK docs) through a follow-up search
* [ ] Exact queries, assistant, date, region, and returned URLs are recorded so a later run can be compared against this one
* [ ] Wrong facts are fixed at the source - structured data and entity pages - not by trying to prompt the assistant into a different answer

Ora's catalog names these checks `brand-search-accuracy`, `agentic-search-usecase`, and `agentic-search-specific`; see [Ora's checks catalog](https://ora.ai/api/checks). See [Commercial and Entity Discovery](/docs/discovery/commercial-and-entity-discovery) for the underlying measurement method.

***

## Where to stop [#where-to-stop]

Most production SaaS products targeting developer audiences should complete Tier 1 through Tier 4-5. Tier 6 items are appropriate for companies that want to be the primary source agents cite in their domain, or that build infrastructure other agents depend on. There is no certified score for finishing a tier - treat each tier as done when its checkboxes are, and move to the next.

## Verify the path from discovery to action [#verify-the-path-from-discovery-to-action]

* [ ] A caller starting from the intended public entry point can locate and use the advertised capability.
* [ ] Product identity, prices, availability, limits, and relevant terms agree across visible and machine-readable sources.
* [ ] Catalog entries validate against the claimed format and resolve to working resources.
* [ ] Missing links and pages return an accurate status with a useful recovery path.
* [ ] Any external identity, registry listing, or trust claim has supporting evidence.

Use [catalogs and feeds](/docs/discovery/catalogs-and-feeds), [commercial and entity discovery](/docs/discovery/commercial-and-entity-discovery), and [product journey evaluation](/docs/scoring/product-journeys) to verify these outcomes. Select additional formats for the clients and capabilities the product supports.