--- title: "Scoring Rubric" description: "Detailed 0/1/2/3 criteria for all 10 dimensions with detection patterns" url: "https://agentsurface.dev/docs/scoring/rubric" lastVerified: 2026-09-25 lastModified: 2026-09-25T08:23:18.000Z --- ## Summary [#summary] Authoritative 0–3 scoring criteria for all 10 dimensions with concrete evidence patterns. For each dimension: 0 = not implemented (blocker), 1 = basic/weak (exists but insufficient), 2 = good/functional (production use), 3 = excellent/optimized (best practices). Each level specifies detection patterns (file globs, grep patterns, command output) for evidence-based auditing. Covers API Surface, CLI Design, MCP Server, Discovery & AEO, Authentication, Error Handling, Tool Design, Context Files, Testing, Retrievability. ``` 0: Not implemented (blocker) 1: Basic/weak (exists, insufficient) 2: Good/functional (production-ready) 3: Excellent/optimized (best practices) ``` *** This is the authoritative rubric for scoring. For each dimension, the 0/1/2/3 scale defines what evidence you're looking for and what concrete signals indicate each level. *** ## Dimension 1: API Surface [#dimension-1-api-surface] **What it measures:** How well the project's HTTP API is described for AI agent tool generation. Includes OpenAPI quality, operationIds, agent-oriented descriptions, parameter structure, examples, and Arazzo workflow support. | Score | Criteria | Evidence Patterns | | ----- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **0** | No machine-readable API spec. Endpoints exist but no OpenAPI, Swagger, or formal schema. | No files matching `**/openapi.{json,yaml,yml}`, `**/swagger.{json,yaml}`. | | **1** | OpenAPI exists but descriptions are human-oriented. Missing operationIds, vague summaries, no examples, nested params. | OpenAPI present but: descriptions `` `<20` `` words or lack "use when" context; operationId missing on >30% of operations; no example values on schema properties; nested parameter structures (objects within objects). | | **2** | Agent-oriented descriptions with "use when" and disambiguation. Proper operationIds on all operations (verb\_noun style). Enums exhaustive. Examples on all parameters. Flat structures. | Descriptions include "Use this when…" and "Do not use for…"; operationId on 100% of operations; enum values fully enumerated on constrained strings; example property on all schema fields; parameters at root level, not deeply nested. | | **3** | Full agent optimization. Arazzo workflows for multi-step operations. Semantic extensions (x-speakeasy-mcp, x-action, x-agent-\*). MCP auto-generated from spec. LAPIS-style token efficiency. | Arazzo file present with workflow definitions; x-speakeasy-mcp or x-action extensions used; MCP server generated from OpenAPI spec; description token efficiency `` `<200` `` tokens per operation. | **Key files to examine:** `openapi.json`, `openapi.yaml`, `swagger.json`, API route files, Arazzo definitions, MCP generation code. *** ## Dimension 2: CLI Design [#dimension-2-cli-design] **What it measures:** How well the project's CLI tool works with agent automation. Based on Agent DX CLI Scale principles: JSON output, exit codes, schema introspection, input validation. | Score | Criteria | Evidence Patterns | | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **0** | Human-only output. Tables, color codes, prose. No structured format. Interactive prompts without bypass. | CLI exists but: no `--json` flag; no `--output json` option; interactive prompts with no `--yes` or `--force` bypass; no machine-readable output to stdout. | | **1** | JSON output exists but inconsistent across commands. Some support `--json`, others don't. Errors may be unstructured. | `--json` or `--output json` on some commands but not others; JSON shape differs between commands; non-zero exit code but no semantic distinction (all failures are exit 1); spinner output not suppressed when piped. | | **2** | Consistent JSON across all commands. Structured error responses. Semantic exit codes (0–5). Dry-run on mutations. TTY detection. | All commands produce consistent JSON schema; exit codes differentiate: 0=success, 1=usage error, 2=validation error, 3=not found, 4=permission, 5=conflict; `--dry-run` on all write operations; isatty() detection disables spinners when piped. | | **3** | NDJSON streaming for pagination. Schema introspection (--schema dumps params/types/required). Input hardening (path traversal, control chars). SKILL.md shipped. Agent knowledge packaging. | `--schema` or `--describe` command returns full machine-readable schema (params, types, required fields); NDJSON streaming for large result sets; input validation rejects `../`, `%2e`, control chars; SKILL.md or AGENTS.md ships with CLI package. | **Key files to examine:** CLI entry point, command definitions, middleware, error handlers, package.json `bin` field, SKILL.md. **N/A when:** Project has no CLI tool and is not primarily a CLI tool. *** ## Dimension 3: MCP Server [#dimension-3-mcp-server] **What it measures:** Whether and how well the project exposes an MCP server. Includes tool definitions, annotations, resources, error handling, auth, and testing. | Score | Criteria | Evidence Patterns | | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **0** | No MCP server. No .mcp.json. No MCP SDK imports. | No files importing `@modelcontextprotocol/sdk`, `mcp-handler`, `@mastra/mcp`, or similar MCP libraries. | | **1** | Basic MCP server exists but minimal. Few tools (`` `<5` ``). Weak descriptions. No annotations. No resources. Unstructured errors. | MCP server present but: `` `<5` `` tools defined; descriptions `` `<20` `` words or lack agent context; no annotations object or all empty; no resources or prompts exposed; errors not wrapped in `{ isError: true }` structure. | | **2** | Well-structured MCP with proper annotations. Agent-oriented descriptions. Structured error handling. outputSchema declared. Resources for static data. | Tools have annotations (`readOnlyHint`, `destructiveHint`, `idempotentHint`); descriptions include "use when" context; `isError: true` pattern on tool errors; `outputSchema` declared on tools returning structured data; resources expose configuration or reference data. | | **3** | Production MCP. RFC 9728 protected-resource metadata for protected HTTP servers. Pagination on list operations. Progress notifications. Multiple transports. Tested with InMemoryTransport. Tool count optimized (`<20`). Consent gates for destructive/authenticated/production tools. Roots, sampling, and elicitation explicitly gated where implemented. | RFC 9728 protected-resource metadata validates issuer, audience, resource, and scope; pagination implemented on tools returning arrays; progress notifications sent for long operations; both stdio and Streamable HTTP transports configured; test files using `InMemoryTransport.createLinkedPair()`; tool count under 20; a documented consent policy gates destructive, authenticated, or production-affecting tools; roots, sampling, and elicitation each have explicit gating where the server implements them. | **Key files to examine:** `.mcp.json`, MCP server implementation, tool definitions, test files, auth middleware. *** ## Dimension 4: Discovery & AEO [#dimension-4-discovery--aeo] **What it measures:** How discoverable, readable, governable, and callable the project is by AI agents. Covers llms.txt, AGENTS.md, JSON-LD schema, Markdown content negotiation, robots.txt, Content Signals, `.well-known` capability metadata, auth discovery, MCP discovery, and agent-commerce checks when relevant. | Score | Criteria | Evidence Patterns | | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **0** | No agent-specific discovery files. No llms.txt, no AGENTS.md, no structured data. robots.txt blocks AI bots or hides public docs from retrieval. | No `llms.txt` at web root; no `AGENTS.md` in repo root; no JSON-LD markup in HTML; robots.txt disallows retrieval or search crawlers (a matching `Disallow` under the relevant `User-agent` group, including AI-crawler groups). | | **1** | Basic discovery. AGENTS.md, llms.txt, robots.txt, or sitemap exists but is minimal. No capability discovery and no agent-specific content format. | AGENTS.md present but `` `<50` `` lines or auto-generated; OR llms.txt present but `` `<10` `` links, no descriptions; basic sitemap only; no JSON-LD; no Markdown response path. | | **2** | Good discovery. llms.txt with categorized sections. AGENTS.md with commands and conventions. JSON-LD on key pages. robots.txt allows intended retrieval/search bots. Sitemap with lastmod. OpenAPI is linked from docs or root. | llms.txt with H2 sections and link descriptions; AGENTS.md with exact commands, conventions, and boundaries; `FAQPage`, `TechArticle`, or `WebAPI` JSON-LD; robots.txt explicitly allows retrieval bots and references sitemap; OpenAPI discoverable at a stable URL. | | **3** | Full agent-readable web surface. Core discovery is semantically valid, not just present. Applicable capability signals are published and verified; unrelated optional protocols are recorded as not applicable. | `llms-full.txt` when useful; Markdown responses or `.md` routes; coherent Content Signals consistent with robots.txt access rules (and RSL where a price or licence condition applies); applicable API Catalog, MCP, Agent Skills, OAuth, Web Bot Auth, WebMCP, or commerce artifacts parse correctly, link to real content, and advertise working capabilities. | **Key files to examine:** `llms.txt`, `llms-full.txt`, `AGENTS.md`, `robots.txt`, `sitemap.xml`, HTML layout/templates, server middleware, `.well-known/` files, OpenAPI/API Catalog files, MCP metadata, Agent Skills indexes. **N/A when:** Project is a pure library or tool with no web presence and no documentation site. Within an applicable web surface, activate optional checks only when their capability is detected. Require MCP metadata for a remote MCP server, OAuth metadata for protected resources, Agent Skills discovery for published skills, WebMCP for page-scoped agent interactions, commerce manifests for commerce, and Web Bot Auth for signed bot identity. Record other signals as not applicable rather than passes or failures. Presence alone is not evidence of readiness. Parse each artifact, reject HTML/login/placeholder bodies, resolve advertised links, and safely test the capability. Hash linked Agent Skills against their published digests; when MCP resources are advertised, sample them with `resources/read` and confirm non-empty content with accurate MIME types. *** ## Dimension 5: Authentication [#dimension-5-authentication] **What it measures:** Whether clients can obtain and use credentials appropriate to the acting identity. User-delegated access uses Authorization Code with PKCE; service-owned M2M access can use Client Credentials. Interactive consent at connection time is valid. Routine protected requests need scoped tokens and structured authentication failures. | Score | Criteria | Evidence | | ----- | ------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 0 | No usable authorization path for the intended client and identity. | Protected requests return login HTML or require unsupported browser state; required consent or token acquisition cannot complete. | | 1 | Programmatic access works, but permissions, credential lifetime or validation are incomplete. | Broad or permanent credentials, incomplete scope enforcement, or unclear refresh and revocation behaviour. | | 2 | A supported grant fits the identity, with scoped, time-bounded access and server-side validation. | Authorization Code with PKCE for user delegation, or Client Credentials for service-owned M2M; validate token issuer, audience/resource, expiry and permissions using the token format's validation mechanism. | | 3 | Level 2 plus verified discovery, isolation and credential recovery for the supported flows. | Remote MCP protected-resource metadata where applicable; tested renewal, revocation and wrong-user/wrong-resource rejection; attributable actions. Token exchange or other extensions only when the use case requires them. | Do not penalize a consumer integration for lacking Client Credentials. Score only flows its advertised capabilities require. [MCP authorization extensions](https://modelcontextprotocol.io/extensions/auth/overview) distinguish user consent from optional M2M support. **Key files to examine:** Auth config, OAuth endpoints, middleware, JWT validation, `.well-known/oauth-*` files. *** ## Dimension 6: Error Handling [#dimension-6-error-handling] **What it measures:** Whether errors give agents enough information to recover. Covers RFC 9457 Problem Details, retriability hints, suggestions, and debugging context. | Score | Criteria | Evidence Patterns | | ----- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **0** | Generic HTTP status codes only. No structured error body. "400 Bad Request" with no detail. | Error responses return plain text or empty bodies. No consistent error schema. Errors lack operational context. | | **1** | Some structured errors but inconsistent. Some endpoints return JSON errors, others don't. | Partial error schema: some endpoints return `{ type, message }`, others return raw strings. No `is_retriable` field. Error shapes differ across endpoints. | | **2** | RFC 9457 Problem Details everywhere. type, title, status, detail, instance. is\_retriable boolean. suggestions array. trace\_id for debugging. | All error responses conform to RFC 9457 schema: `type` (URI), `title`, `status`, `detail`, `instance`. `is_retriable` boolean on all errors. `suggestions` array with recovery steps (e.g., "retry after 5s", "increase timeout"). `trace_id` header for request tracking. Rate limit 429 includes `Retry-After` header. | | **3** | Full agent error design. doc\_uri linking to docs. Intent trace on cancellation. Domain-specific codes. X-RateLimit headers on all responses. CLI semantic exit codes. | `doc_uri` in error responses (link to relevant docs); intent trace structure on cancel/abort operations; domain-specific error codes alongside HTTP (e.g., `code: "RATE_LIMITED_BY_UPSTREAM"`); `X-RateLimit-*` headers on all responses, including non-429 responses; CLI errors use semantic exit codes + JSON output. | **Key files to examine:** Error middleware, HTTP error handlers, API responses, CLI error output. *** ## Dimension 7: Tool Design [#dimension-7-tool-design] **What it measures:** Quality of tool definitions for AI agent consumption, across any framework (MCP, AI SDK, LangChain). | Score | Criteria | Evidence Patterns | | ----- | -------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **0** | No formal tool definitions. Functions exist but no schema, no description. | No `tool()` calls, no `@tool` decorators, no `createTool()`, no MCP tool registrations anywhere in codebase. | | **1** | Basic tool schemas exist but descriptions are terse or missing. No examples. | Tool definitions present but: descriptions `` `<20` `` words or lack context; no `.describe()` on Zod fields; no `inputExamples`; parameter names are single letters (`u`, `q`) or generic (`data`, `obj`). | | **2** | Good tool design. verb\_noun naming. Agent-oriented descriptions. Typed schemas with field descriptions. | Tool names follow `verb_noun` pattern (e.g., `search_users`, `create_document`); descriptions include "Use when…" and "Do not use for…"; all schema fields have `.describe()` docstrings; enum values fully specified; `` `<10` `` tools per agent context. | | **3** | Excellent tool design. toModelOutput reducing tokens. Annotations. Dynamic selection. Cross-framework. | `toModelOutput` defined to reduce token usage for responses; annotations object present (`readOnly`, `destructive`, `idempotent`); activeTools or `defer_loading` patterns for dynamic selection; tool definitions portable (same schema works in MCP, Claude SDK, OpenAI Agents SDK). | **Key files to examine:** Tool definitions, agent setup, MCP tool registrations, schema decorators. *** ## Dimension 8: Context Files [#dimension-8-context-files] **What it measures:** Quality of agent context files for AI coding assistants (AGENTS.md, CLAUDE.md, .cursor/rules, etc.). | Score | Criteria | Evidence Patterns | | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **0** | No AGENTS.md, CLAUDE.md, or equivalent context files. | No agent context files found in repo root or standard locations. | | **1** | Context file exists but generic or auto-generated. Prose paragraphs. No actionable commands. | AGENTS.md or CLAUDE.md present but: >500 lines of prose; auto-generated (shows no curation, mentions "auto-generated" or similar); lacks command section; architecture narrative without boundaries. | | **2** | Hand-curated context files. Commands with exact flags first. Permission boundaries. Testing expectations. | Commands section at top with exact invocations (e.g., `pnpm build`, `pnpm test -- --watch`); permission boundaries defined (always/ask-first/never); `` `<370` `` lines; non-obvious conventions documented with code examples; updated iteratively (git history shows refinements). | | **3** | Curated AGENTS.md as the single source. Tool-specific files (CLAUDE.md, .cursor/rules, Copilot instructions) exist only where a tool needs something AGENTS.md can't give it, and each defers to AGENTS.md rather than restating it. Progressive/nested disclosure in monorepos. Updated from real friction. | AGENTS.md present and hand-curated; any CLAUDE.md starts with a literal `@AGENTS.md` import and adds only Claude-only lines below it; .cursor/rules and Copilot instructions reference AGENTS.md rather than duplicating it; nested AGENTS.md for monorepo packages where relevant; permission boundaries enforced in text; files clearly evolved from usage friction (not auto-generated or one-shot). | **Key files to examine:** `AGENTS.md`, `CLAUDE.md` (if present), `.cursor/rules/*.mdc`, `.github/copilot-instructions.md`, `.windsurf/rules/`. *** ## Dimension 9: Testing & Evaluation [#dimension-9-testing--evaluation] **What it measures:** Whether agent interactions are tested and evaluated. Includes tool routing accuracy, error recovery, multi-step flows, and statistical metrics. | Score | Criteria | Evidence Patterns | | ----- | ------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **0** | No agent-specific tests. Standard unit/integration tests only. | No test files targeting tool selection, agent behavior, or MCP server testing. | | **1** | Basic tool routing tests. Some verification that tools are called correctly. | Test files verify tool selection or MCP tool responses. But: no error recovery testing; no multi-step flow verification; MCP tests don't use `InMemoryTransport`. | | **2** | Comprehensive tool testing. Selection accuracy, parameter correctness, error recovery. Multi-step flows. MCP tested with InMemoryTransport. | Tests cover: correct tool selection on various inputs, valid parameters, error → recovery flow, multi-step sequences. MCP tests use `InMemoryTransport.createLinkedPair()`. Test cases document failure scenarios. | | **3** | Full eval suite. pass\@k and pass^k metrics. Non-determinism handling. Regression detection. CI integration. Eval-driven development. | Statistical metrics (pass\@k: any of k runs succeed; pass^k: all k runs succeed); multiple runs per test case to handle non-determinism; baseline comparison for regression detection; eval suite runs in CI; test cases derived from real production failures. | **Key files to examine:** Test directories, eval suites, CI configuration (`.github/workflows/`, `package.json` test scripts). *** ## Dimension 10: Retrievability [#dimension-10-retrievability] **What it measures:** Whether an agent can query the product's own data - documents, records, catalog items, support content - through a documented search or retrieval capability and get back results it can trust: scoped correctly, current, and attributable to a source. This is the product side of retrieval; how an agent builds its own vector database or RAG pipeline is a separate concern, covered in [Agent Retrieval](/docs/agent-retrieval). | Score | Criteria | Evidence Patterns | | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **0** | No queryable retrieval surface. Content is reachable only by reading whole pages or files; no search endpoint, no filters, no structured result schema. | No `/search` route or OpenAPI path with query parameters; no sitemap or feed; no structured data markup. | | **1** | A search or lookup endpoint exists but is undocumented or unstructured: results are HTML fragments or free text, not typed records, with no pagination. | Endpoint returns rendered HTML or free text; no documented query parameters; no `limit`/`cursor` pagination; no result schema. | | **2** | Documented search or retrieval API with typed query and result schemas, pagination and filters scoped to retrieval, and freshness or provenance metadata on each result. | OpenAPI or tool schema documents the endpoint's parameters and response shape; results carry stable fields (`id`, `title`, `url`, `updated_at`); filters and pagination implemented and documented. | | **3** | Full retrieval contract. Citations or source metadata on results, permission-scoped results verified, freshness and versioning exposed, stable IDs for re-fetch, documented limits for bulk or paginated retrieval. | Results include a citation or source identifier; a test confirms a caller only sees results it is authorized to see; cursor-based pagination survives concurrent updates; bulk export or feed endpoint documented with size and rate limits. | **Key files to examine:** Search/retrieval endpoint implementations, OpenAPI paths with query parameters, MCP resource or tool definitions exposing search, sitemap/feed generators, structured-data (JSON-LD) templates. **N/A when:** the project exposes no searchable corpus, documents, or records to agents. *** ## Evidence from product use [#evidence-from-product-use] Evaluate a representative task through the intended client. Discovery evidence should show that a caller can find the right capability without private knowledge. Retrieval evidence should establish current, authoritative decision facts, including prices and availability when relevant. API and tool evidence should show usable continuation and lifecycle contracts. Testing evidence should include a meaningful failure, safe recovery, and the authoritative final result. Assign each defect to the dimension owning its root cause and explain effects on the rest of the journey. Avoid counting the same broken source or endpoint several times. Protocol files, catalog entries, and listings support the evaluation only when they lead to a capability needed by that product. Score definitions and totals above remain unchanged. See [Evaluating Product Journeys](/docs/scoring/product-journeys) for the full method. ## Scoring Notes [#scoring-notes] * Score based on **current state**, not intent or roadmap * Evidence must be **specific**: cite file paths and line numbers, quote grep results, specify glob patterns * When uncertain between two levels, score conservatively (lower) * Confidence levels: **High** (examined `` `>80%` ``), **Medium** (key files examined), **Low** (`` `<30%` `` coverage) * Mark N/A **only** when dimension is genuinely inapplicable to the project