Testing

Summary

Agent testing measures statistical performance under non-determinism. Covers the full lifecycle: evaluation frameworks and metrics (pass@k, tool-routing accuracy), LLM-as-judge calibration, specific platforms (Braintrust, Promptfoo, Vitest), red-teaming for injection/jailbreak, observability, and CI/CD integration. Start with framework definition, then pick a platform matching your workflow.

• Metrics: pass@k, pass^k, MTTR, tool-routing F1, schema conformance
• LLM-as-judge: bias calibration, when to use
• Platforms: Braintrust (managed), Promptfoo (red-team), Vitest (in-process)
• Red-teaming: injection, jailbreak, OWASP LLM Top 10
• Observability: OpenTelemetry GenAI conventions
• CI integration: PR-gated evals, budget guardrails

Traditional testing verifies deterministic behavior - given input X, does output Y occur every time? An agent may take different paths, call tools in different orders, or succeed on the fifth attempt when it failed on the first. A single success or failure tells you nothing.

Quick reference

• Metrics - /docs/testing/metrics covers pass@k, pass^k, MTTR, tool-routing F1, schema conformance, hallucination rates
• LLM-as-judge - /docs/testing/llm-as-judge explains when to use judge models, their biases, and calibration techniques
• Evaluation framework - /docs/testing/evaluation-framework - dataset, task, scorer, aggregation loop
• Braintrust - /docs/testing/braintrust - end-to-end eval platform with dataset versioning and trace integration
• Promptfoo - /docs/testing/promptfoo - YAML-first evals with strong red-teaming, acquisition announced by OpenAI, close unconfirmed
• Vitest harness - /docs/testing/vitest-harness - in-process evals, useful for fast iteration and local debugging
• Red-teaming - /docs/testing/red-teaming - prompt injection, jailbreak, data exfiltration attack templates; OWASP LLM Top 10 mapping
• Observability - /docs/testing/observability - OpenTelemetry GenAI semantic conventions for trace-driven debugging
• CI integration - /docs/testing/ci-integration - PR-gated evals, regression tests, statistical significance, budget guardrails

Start with the evaluation framework, then pick a platform that fits your workflow. If you're running Vitest-based agents, use the Vitest harness. If you want managed experiment tracking, use Braintrust. For quick red-teaming, Promptfoo is the fastest path.

A/B Testing

Evals answer whether a change passed against a dataset. A/B testing answers whether it performed better in front of real users, and complements evals: evals gate a merge cheaply before a change ever reaches traffic; A/B testing measures the thing evals can only approximate. It needs enough live traffic to reach statistical significance, so it becomes practical once volume is high enough to split without starving either arm - before that point, lean on offline and online evals instead. Online evals score output quality on sampled traffic; A/B testing compares variants on the product outcome.