Summary

This tiered checklist organizes agent-discovery work from essential foundation items through comprehensive support. Tier 1 covers , , and OpenGraph tags - low effort, high impact. Later tiers add , , , links, , Server Cards, Agent Skills indexes, metadata, , and commerce protocols where relevant.

Tier 1: robots.txt, llms.txt, OpenGraph meta tags
Tier 2: OpenAPI discovery, JSON-LD markup
Tier 3: Content negotiation, content structure,
Tier 4: Well-known endpoints, API Catalog, MCP Server Cards, Agent Skills, auth discovery
Tier 5: AGENTS.md depth, llms-full.txt, Copy for AI, count disclosure, analytics, changelog, status, agent terms of use
Tier 6: /ask endpoint, /mcp endpoint, agent-mode page view, Content Signals
Start with Tier 1 before advancing

This checklist covers the full discovery surface organized by tier. Tier 1 items are the minimum viable foundation - ship these first. Later tiers add depth and capability for teams investing seriously in agent discoverability.

Each item links to the relevant section of this documentation where applicable. This is a checklist, not a scoring system - completing a tier means the items are done, not that you have reached a certified level.

Tier 1 - Foundation

These items cost little to implement and make later discovery work easier to verify. Complete these before anything else.

robots.txt

robots.txt exists at /robots.txt and is served with Content-Type: text/plain
User-agent: * with Allow: / covers all legitimate bots
Training crawlers explicitly disallowed: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, cohere-ai, CCBot (or explicitly allowed - the decision is made and documented)
Retrieval bots allowed: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot are not blocked
Search, Agent, and Training access are decided as three separate questions, not one blanket toggle - if the host is Cloudflare, its three-way classification and Bot Preference Sync are configured; otherwise the equivalent per-bot rules are hand-maintained
Sitemap: directive points to your primary
/robots.txt is tested with Google Search Console's robots.txt report

See robots.txt for AI Agents.

llms.txt

/llms.txt exists and is served as Content-Type: text/plain; charset=utf-8
File begins with # Product Name (H1 required)
Blockquote summary follows the H1 (> Description here)
All significant documentation sections appear as H2 groups with link lists
Each link includes a colon-separated description: - [Title](url): Description
Links point to .md URLs where available, HTML otherwise
Total file size is under 5,000 tokens (approximately 20,000 characters)
## Optional section used for supplementary content

See llms.txt.

OpenGraph Meta Tags

og:title is entity-first: "Product Name - Page Topic"
og:description is 150–300 characters, factual, answer-dense
article:modified_time is present and accurate (W3C datetime format)
article:published_time is present
article:tag values are present for topic categorization

Sitemap

XML sitemap exists at /sitemap.xml or a sitemap index at /sitemap.xml referencing sub-sitemaps
Every documentation page appears in the sitemap
<lastmod> is present and accurate for every URL
<lastmod> format is W3C datetime: YYYY-MM-DD or YYYY-MM-DDTHH:MM:SSZ
Sitemap contains fewer than 50,000 URLs (use sitemap index if larger)
Sitemap is referenced in robots.txt

Semantic HTML

Main content is wrapped in <article> or <main>
Navigation is in <nav> with aria-label
Headings use semantic tags (h1–h6), not styled <div> elements
Page has exactly one <h1> matching the <title> tag
<time datetime="..."> used for dates

Tier 2 - Content Quality

Content structure improvements that directly affect whether agents extract accurate answers.

Answer-First Structure

Every page puts the direct answer or key fact in the first 200 words
Introductory paragraphs do not delay the answer with background or context
Tables appear before prose explanations of the same data
Rate limits, pricing, and other quantitative facts appear as early as possible

See Content Structure for AI Consumption.

Self-Contained Sections

Each H2 section can be read and understood without context from preceding sections
No section references "the previous step" or "as described above" without restating what was described
Code examples include all necessary imports and context
Error messages are quoted verbatim with exact capitalization and punctuation

Markdown Availability

Documentation pages are available in Markdown format at .md URLs or via Accept: text/markdown
<link rel="alternate" type="text/markdown" href="..."> present in HTML <head>
Markdown output strips navigation and footer, preserves content
For localized content: hreflang alternates are present, the locale is a path segment on Markdown routes (not a cookie or query parameter), and each locale has its own llms.txt

See Serving Markdown to Agents.

Vary Header

All responses that differ by Accept header include Vary: Accept
CDN configuration honors Vary: Accept (separate cache entries per format)
Content-Type: text/markdown; charset=utf-8 used for Markdown responses (not text/plain)

Tier 3 - Structured Data

JSON-LD schema markup for AI Overview inclusion and retrieval system accuracy.

FAQPage Schema

Google dropped FAQ rich results from Search in May 2026, so this schema no longer earns Google search presentation. Keep it only for the agents and other parsers that still read FAQPage JSON-LD directly.

Pages with FAQ or Q&A content include FAQPage JSON-LD
Every Question.name exactly matches visible heading text
Every Answer.text exactly matches visible answer text
No schema content exists that is not visible on the page
Schema validated with the Schema.org Validator

See Structured Data for Agents.

TechArticle Schema

All technical documentation pages include TechArticle JSON-LD
dateModified is accurate and kept current
author and publisher are present
proficiencyLevel is set (Beginner or Expert)

HowTo Schema

Google dropped HowTo rich results from Search in September 2023. Keep the schema for parsers other than Google Search; do not expect it to earn a Google rich result.

Step-by-step guides and tutorials include HowTo JSON-LD
Each step includes position, name, text, and url (anchored to heading)
totalTime is present in ISO 8601 duration format
tool array lists prerequisites

WebAPI Schema

reference pages include WebAPI JSON-LD
documentation, endpointUrl, and provider are present
version reflects the current API version

SoftwareApplication Schema

Product and tool pages include SoftwareApplication JSON-LD
softwareVersion is kept current
downloadUrl and codeRepository present where applicable

Tier 4 - Discovery Endpoints

Machine-readable capability declarations at standard paths.

/.well-known/ai

/.well-known/ai exists and is valid JSON
capabilities array accurately reflects what you support
api.spec points to a valid OpenAPI JSON file
mcp.endpoint present if you run an MCP server
llms.index points to your llms.txt

See Well-Known Discovery Endpoints.

OpenAPI Specification

OpenAPI spec available at /openapi.json or /api/openapi.json
Spec validates against OpenAPI 3.1 schema
Every operation has a summary and description
Every parameter has a description
Every response has a description and schema
Spec is kept in sync with actual API behavior (tested in CI)

API Catalog

/.well-known/api-catalog exists if you expose public APIs
Catalog links to canonical OpenAPI specs, documentation, status pages, and auth docs
HTTP Link: </.well-known/api-catalog>; rel="api-catalog" header present where practical
Catalog omits internal-only APIs that agents should not discover

/.well-known/agent-card.json

agent-card.json published if your service can be delegated to by other agents
skills array describes what the agent can do with realistic examples
authentication specifies how to obtain credentials
capabilities.streaming accurately reflects whether you support streaming

MCP Server Card

/.well-known/mcp/server-card.json published if you run an HTTP MCP server
transport correctly specifies streamable-http or stdio
authentication section accurate and complete
Card describes the server before connection: name, purpose, tools, transport, and auth
/.well-known/mcp.json redirects or points to the server card if used for compatibility

Agent Skills Index

/.well-known/agent-skills/index.json exists when your site teaches task-specific agent workflows
Index declares the Agent Skills discovery v0.2 schema
Every entry has name, type, activation-oriented description, canonical url, and sha256: digest
Each skill is narrow, procedural, and linked as a separate SKILL.md
Linked skills follow the Agent Skills format: required name and description, optional scripts/, references/, and assets/
Each digest matches the exact bytes returned by the corresponding skill URL
Skill descriptions explain when to use the skill, not only what topic it covers
Skills are generated from canonical docs or reviewed with the same rigor as public docs

OAuth Protected Resource Metadata

/.well-known/oauth-protected-resource exists for OAuth-protected APIs
Metadata points to the correct authorization server, , supported , and requirement
Protected resource metadata is linked from API docs and MCP metadata where applicable

Web Bot Auth

/.well-known/http-message-signatures-directory exists if your service sends signed bot or agent requests
Published keys are rotated and documented
Receiving services can verify signed requests before granting high-trust access

Tier 5 - Developer Experience

Improvements for developers using AI coding assistants with your APIs and .

AGENTS.md

AGENTS.md exists at repository root
## Commands section lists all essential commands (build, test, lint, dev server)
Non-standard tooling (unusual package manager, monorepo setup) documented
## Agent Boundaries section with three tiers: autonomous / ask first / never
Testing rules stated explicitly: framework, location, required coverage
File is under 500 lines
No prose paragraphs - only lists, code blocks, tables

See AGENTS.md and Context Files.

llms-full.txt

/llms-full.txt exists with all documentation content inlined
File is generated automatically from source content (not maintained by hand)
Content-Type: text/plain; charset=utf-8
Cache-Control TTL is appropriate for how frequently docs change
Token count documented (in comments or referenced from llms.txt)

Copy for AI Button

Documentation pages include a "Copy for AI" button that copies Markdown to clipboard
Button displays estimated token count after copy
Markdown copy excludes navigation and footer content

Token Count Disclosure

x-markdown-tokens header present on Markdown responses
Token counts visible to developers (in UI, in response headers, or in llms.txt)
Pages over their type's hard limit in Content Structure are flagged for splitting

Analytics

AI bot traffic tracked separately from human traffic (by user-agent)
Requests carrying a verified Web Bot Auth signature are distinguished from requests that merely claim a matching user-agent string
Which pages retrieval bots visit most frequently is visible
Retry-After header compliance monitored (are bots respecting ?)

Changelog, Status, and Agent Terms of Use

A dated changelog exists (page or RSS/Atom feed) so an agent already holding your docs in context can detect drift with one small fetch
Changelog entries name what changed and why it matters, not just that a version shipped
A public status page exists and is linked from the API Catalog entry, so an agent can tell an outage from its own error before deciding to retry
Agent-facing terms of use are published as their own page - what automated or unattended use is actually permitted - and linked from llms.txt and API docs, not left implicit in robots.txt or buried in general Terms of Service

See Content Structure and Commercial and Entity Discovery.

Tier 6 - Advanced

High-investment items for teams deeply committed to agent discoverability.

NLWeb /ask Endpoint

/ask endpoint returns direct answers to natural-language questions
Endpoint accessible as both HTTP GET and as an MCP server
Response includes sources array with URLs and excerpts
Indexed against current documentation content (not stale snapshots)

See Well-Known Discovery Endpoints.

/mcp Endpoint

MCP server running and accessible to agents
All API operations exposed as MCP tools with accurate descriptions
Resource endpoints serve documentation and configuration data
Authentication works with the flow documented for your agent clients, usually OAuth 2.1 Client Credentials, , or host-managed delegated auth
Tool definitions tested with MCP inspector
Health check endpoint for uptime monitoring

See the MCP Servers section.

Agent-Mode Page View

A stripped structured view exists only where it does a job Markdown negotiation doesn't - direct action links rather than prose content
The view is served at a distinct, documented URL or parameter (e.g. ?mode=agent), not guessed
Facts, prices, and links in the agent-mode view are checked against the canonical page - it's not a second, silently drifting source of truth

See Serving Markdown to Agents.

Content Signals

Content-Signal: search=yes, ai-input=yes, ai-train=no declared in robots.txt or response headers
Policy header overridden appropriately for gated content
robots.txt and Content Signals policy are consistent with each other
If a price or specific licence condition (not just yes/no) matters, is attached via robots.txt, an HTTP header, HTML, or the RSS feed - Content Signals alone can't express a price
IETF aipref (draft-ietf-aipref-vocab, draft-ietf-aipref-attach) is tracked, not implemented as a requirement - both are still pre-consensus drafts

See robots.txt for AI Agents.

Instructions Section in llms.txt

## Instructions section added to llms.txt
Instructions correct known misconceptions agents have from training data
Instructions flag rapidly-changing content (pricing, feature availability)
Instructions clarify naming conventions that conflict with industry defaults
Instructions point to authoritative sources for high-stakes questions

See llms.txt.

agents.json / Wildcard AI Flows

agents.json at root describes common workflows with step-by-step actions
Workflows cover the top 3–5 agent use cases for your product
Each flow includes success_output and failure_output descriptions
Flows tested with an agent to verify they execute correctly

See Well-Known Discovery Endpoints.

Agentic Commerce

If agent-mediated commerce is a real product surface, see Agentic Commerce for product data, checkout protocols, payments, and trust and risk - not covered on this checklist

Agentic Search Measurement

A fixed set of brand-name queries is run across assistants (ChatGPT, Claude, Gemini, Perplexity) on a regular cadence, and your domain appears in the top results
A fixed set of category or use-case queries ("a tool that does X") is run across the same assistants, tracking whether you are mentioned at all, not only your rank when named directly
An assistant that already surfaces your brand can locate your developer resources (API docs, OpenAPI spec, MCP server, auth docs, SDK docs) through a follow-up search
Exact queries, assistant, date, region, and returned URLs are recorded so a later run can be compared against this one
Wrong facts are fixed at the source - structured data and entity pages - not by trying to prompt the assistant into a different answer

Ora's catalog names these checks brand-search-accuracy, agentic-search-usecase, and agentic-search-specific; see Ora's checks catalog. See Commercial and Entity Discovery for the underlying measurement method.

Where to stop

Most production SaaS products targeting developer audiences should complete Tier 1 through Tier 4-5. Tier 6 items are appropriate for companies that want to be the primary source agents cite in their domain, or that build infrastructure other agents depend on. There is no certified score for finishing a tier - treat each tier as done when its checkboxes are, and move to the next.

Verify the path from discovery to action

A caller starting from the intended public entry point can locate and use the advertised capability.
Product identity, prices, availability, limits, and relevant terms agree across visible and machine-readable sources.
Catalog entries validate against the claimed format and resolve to working resources.
Missing links and pages return an accurate status with a useful recovery path.
Any external identity, registry listing, or trust claim has supporting evidence.

Use catalogs and feeds, commercial and entity discovery, and product journey evaluation to verify these outcomes. Select additional formats for the clients and capabilities the product supports.

On this page