--- title: "Evaluation and Observability" description: "AI evals, traces, monitoring, guardrails, policy, and security tooling" url: "https://agentsurface.dev/docs/tooling-catalog/evaluation-and-observability" lastVerified: 2026-09-24 lastModified: 2026-09-25T05:46:01.000Z --- Use these when agent behavior must be measurable, regressions must be caught, or untrusted inputs and sensitive data need explicit controls. ## Evaluation and Observability [#evaluation-and-observability] Keep this category AI-specific. General monitoring tools are only relevant as trace backends when paired with GenAI instrumentation. First check whether your agent framework already ships an exporter for the backend. Framework-native exporters mean tracing is configuration rather than instrumentation work; Mastra, for one, ships exporters for Arize, Arthur, Braintrust, Datadog, Laminar, Langfuse, LangSmith, PostHog, and Sentry alongside plain OpenTelemetry. Check your framework's exporter list before you shortlist on features alone. | Tool | Website | What it is for | Tier | Notes | | ----------------------------------------------------------------------------------------------- | ------------------------ | ------------------------------------------------------------------------------------------ | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | [Braintrust](https://www.braintrust.dev/docs) | braintrust.dev | AI observability, evals, datasets, experiments, prompts, and production monitoring | Core | Strong default for serious eval loops and regression testing. | | [Langfuse](https://langfuse.com/docs) | langfuse.com | LLM tracing, prompt management, evals, datasets, and analytics | Core | Strong open-source option with broad framework integrations. | | [LangSmith](https://docs.smith.langchain.com) | docs.smith.langchain.com | LangChain and LangGraph tracing, evals, datasets, and deployment feedback | Conditional | Strong when using LangChain or LangGraph. | | [Arize Phoenix](https://arize.com/docs/phoenix/) | arize.com | Open-source LLM observability, tracing, evals, and prompt experiments | Core | Strong OpenTelemetry/OpenInference-aligned option, and a common framework-native exporter target. | | [Laminar](https://laminar.sh/) | laminar.sh | Open-source agent observability with traces, failure clustering, evals, and a CLI/MCP path | Conditional | Apache-2.0 and self-hostable via Docker or Helm. Notable for turning error clusters into eval datasets, which closes the trace-to-regression-test loop rather than stopping at dashboards. | | [Arthur](https://www.arthur.ai/) | arthur.ai | Agentic tracing, automated evaluations, guardrails, and AI governance in one control plane | Conditional | Fits when governance, ownership, and risk scoring are the requirement rather than developer debugging alone; an open-source `arthur-engine` exists for the evaluation and guardrail layer. | | [Promptfoo](https://www.promptfoo.dev/docs) | promptfoo.dev | CLI-first prompt, model, RAG, agent, and red-team evaluations | Core | Good for CI-friendly evals and adversarial testing. | | [RAGAS](https://docs.ragas.io/en/stable/) | docs.ragas.io | RAG and agent evaluation metrics | Conditional | Useful for retrieval quality loops; avoid treating metrics as absolute truth. | | [DeepEval](https://deepeval.com/docs/getting-started) | deepeval.com | LLM eval framework for RAG, agents, chatbots, safety, and CI tests | Conditional | Good Python option for test-like evals and synthetic data. | | [Opik](https://www.comet.com/docs/opik) | comet.com | Open-source LLM observability, evaluation, prompt tracking, and agent optimization | Conditional | Good open-source option alongside Langfuse and Phoenix. | | [Weights & Biases Weave](https://weave-docs.wandb.ai/) | weave-docs.wandb.ai | LLM traces, evals, datasets, and experiment tracking | Conditional | Good if the team already uses W\&B. | | [Helicone](https://docs.helicone.ai) | docs.helicone.ai | LLM gateway observability, caching, costs, and logs | Conditional | Useful when lightweight model-call logging and cost visibility are enough. | | [PostHog LLM Observability](https://posthog.com/docs/llm-analytics) | posthog.com | LLM analytics, cost tracking, and product-level observability | Conditional | Relevant when LLM telemetry should sit next to product analytics and feature flags, and now a first-class exporter target in agent frameworks rather than a manual integration. | | [MLflow Tracing](https://mlflow.org/docs/latest/genai/tracing/) | mlflow\.org | OpenTelemetry-compatible tracing for LLM and agent applications | Conditional | Good if the team already uses MLflow or wants self-hosted GenAI traces that can feed eval datasets. | | [New Relic AI Monitoring](https://docs.newrelic.com/docs/ai-monitoring/intro-to-ai-monitoring/) | docs.newrelic.com | APM-backed AI monitoring for LLM calls, vector stores, latency, tokens, cost, and feedback | Conditional | Use when New Relic is already the production observability system; keep AI evals in a dedicated eval loop. | | [SigNoz LLM Observability](https://signoz.io/docs/llm-community-integrations) | signoz.io | OpenTelemetry-native LLM observability integrations and dashboards | Conditional | Good for open-source or self-hosted teams that want GenAI telemetry beside traces, logs, and metrics. | | [AgentOps](https://docs.agentops.ai) | docs.agentops.ai | Agent session tracing, replay, and analytics | Watch | Relevant for agent-specific debugging; evaluate ecosystem fit. | ## Guardrails, Security, and Policy Tools [#guardrails-security-and-policy-tools] Use these when agents read untrusted content, handle user data, call write tools, or operate in regulated environments. These tools do not replace product-level permission design; they help enforce and observe it. | Tool | Website | What it is for | Tier | Notes | | --------------------------------------------------------------------------------------- | ------------------- | ------------------------------------------------------------------------------------------- | ----------- | -------------------------------------------------------------------------------------------- | | [Guardrails AI](https://www.guardrailsai.com/docs) | guardrailsai.com | Validation, structured-output checks, and guardrail orchestration for LLM apps | Conditional | Good when output shape, policy checks, or validation failures need first-class handling. | | [NeMo Guardrails](https://docs.nvidia.com/nemo/guardrails/) | docs.nvidia.com | Programmable guardrails and dialog controls for LLM applications | Conditional | Useful in enterprise/NVIDIA-aligned stacks or policy-heavy conversational systems. | | [Lakera](https://docs.lakera.ai/) | docs.lakera.ai | Prompt-injection and LLM security protection | Conditional | Consider for agents browsing or ingesting hostile/untrusted web content. | | [Llama Guard](https://www.llama.com/docs/model-cards-and-prompt-formats/llama-guard-4/) | llama.com | Safety classification model family | Conditional | Useful as a self-hostable or model-based safety classifier, depending on deployment needs. | | [Microsoft Presidio](https://microsoft.github.io/presidio/) | microsoft.github.io | PII detection, redaction, and anonymization | Conditional | Practical for logs, traces, eval datasets, and tool outputs that may contain sensitive data. | | [Pangea AI Guard](https://pangea.cloud/docs/ai-guard/) | pangea.cloud | AI security guardrail API for prompt, response, and data protection | Watch | Track for teams already using Pangea security services. | | [Superagent](https://docs.superagent.sh) | docs.superagent.sh | Guardrails, PII, prompt-injection, and verification tooling surfaced by the AI SDK registry | Watch | Worth evaluating as the AI SDK ecosystem adds packaged security tools. | ## Public Readiness Scanners [#public-readiness-scanners] External, deployed-site scanners such as Is Agentic, Ora, Cloudflare Agent Readiness, Google Lighthouse's Agentic Browsing category, and Shopify's product-page audit check what an outside agent can observe right now; they complement rather than replace a repository-level Surface audit. See [Public Readiness Scanners](/docs/scoring/public-scanners) for coverage, APIs, and how to interpret each one's findings. Sources are linked inline.