Use these when agent behavior must be measurable, regressions must be caught, or untrusted inputs and sensitive data need explicit controls.

Evaluation and Observability

Keep this category AI-specific. General monitoring tools are only relevant as backends when paired with GenAI instrumentation.

First check whether your agent framework already ships an exporter for the backend. Framework-native exporters mean tracing is configuration rather than instrumentation work; Mastra, for one, ships exporters for Arize, Arthur, Braintrust, Datadog, Laminar, Langfuse, LangSmith, PostHog, and Sentry alongside plain OpenTelemetry. Check your framework's exporter list before you shortlist on features alone.

Tool	Website	What it is for	Tier	Notes
Braintrust	braintrust.dev	AI observability, , datasets, experiments, prompts, and production monitoring	Core	Strong default for serious eval loops and regression testing.
Langfuse	langfuse.com	tracing, prompt management, evals, datasets, and analytics	Core	Strong open-source option with broad framework integrations.
LangSmith	docs.smith.langchain.com	LangChain and LangGraph tracing, evals, datasets, and deployment feedback	Conditional	Strong when using LangChain or LangGraph.
Arize Phoenix	arize.com	Open-source LLM observability, tracing, evals, and prompt experiments	Core	Strong OpenTelemetry/OpenInference-aligned option, and a common framework-native exporter target.
Laminar	laminar.sh	Open-source agent observability with traces, failure clustering, evals, and a CLI/MCP path	Conditional	Apache-2.0 and self-hostable via Docker or Helm. Notable for turning error clusters into eval datasets, which closes the trace-to-regression-test loop rather than stopping at dashboards.
Arthur	arthur.ai	Agentic tracing, automated evaluations, , and AI governance in one control plane	Conditional	Fits when governance, ownership, and risk scoring are the requirement rather than developer debugging alone; an open-source `arthur-engine` exists for the evaluation and guardrail layer.
Promptfoo	promptfoo.dev	CLI-first prompt, model, , agent, and evaluations	Core	Good for CI-friendly evals and adversarial testing.
RAGAS	docs.ragas.io	RAG and agent evaluation metrics	Conditional	Useful for retrieval quality loops; avoid treating metrics as absolute truth.
DeepEval	deepeval.com	LLM eval framework for RAG, agents, chatbots, safety, and CI tests	Conditional	Good Python option for test-like evals and synthetic data.
Opik	comet.com	Open-source LLM observability, evaluation, prompt tracking, and agent optimization	Conditional	Good open-source option alongside Langfuse and Phoenix.
Weights & Biases Weave	weave-docs.wandb.ai	LLM traces, evals, datasets, and experiment tracking	Conditional	Good if the team already uses W&B.
Helicone	docs.helicone.ai	LLM gateway observability, caching, costs, and logs	Conditional	Useful when lightweight model-call logging and cost visibility are enough.
PostHog LLM Observability	posthog.com	LLM analytics, cost tracking, and product-level observability	Conditional	Relevant when LLM telemetry should sit next to product analytics and feature flags, and now a first-class exporter target in agent frameworks rather than a manual integration.
MLflow Tracing	mlflow.org	OpenTelemetry-compatible tracing for LLM and agent applications	Conditional	Good if the team already uses MLflow or wants self-hosted GenAI traces that can feed eval datasets.
New Relic AI Monitoring	docs.newrelic.com	APM-backed AI monitoring for LLM calls, , latency, , cost, and feedback	Conditional	Use when New Relic is already the production observability system; keep AI evals in a dedicated eval loop.
SigNoz LLM Observability	signoz.io	OpenTelemetry-native LLM observability integrations and dashboards	Conditional	Good for open-source or self-hosted teams that want GenAI telemetry beside traces, logs, and metrics.
AgentOps	docs.agentops.ai	Agent session tracing, replay, and analytics	Watch	Relevant for agent-specific debugging; evaluate ecosystem fit.

Guardrails, Security, and Policy Tools

Use these when agents read untrusted content, handle user data, call write tools, or operate in regulated environments. These tools do not replace product-level permission design; they help enforce and observe it.

Tool	Website	What it is for	Tier	Notes
Guardrails AI	guardrailsai.com	Validation, structured-output checks, and guardrail for LLM apps	Conditional	Good when output shape, policy checks, or validation failures need first-class handling.
NeMo Guardrails	docs.nvidia.com	Programmable guardrails and dialog controls for LLM applications	Conditional	Useful in enterprise/NVIDIA-aligned stacks or policy-heavy conversational systems.
Lakera	docs.lakera.ai	Prompt-injection and LLM security protection	Conditional	Consider for agents browsing or ingesting hostile/untrusted web content.
Llama Guard	llama.com	Safety classification model family	Conditional	Useful as a self-hostable or model-based safety classifier, depending on deployment needs.
Microsoft Presidio	microsoft.github.io	PII detection, redaction, and anonymization	Conditional	Practical for logs, traces, eval datasets, and tool outputs that may contain sensitive data.
Pangea AI Guard	pangea.cloud	AI security guardrail for prompt, response, and data protection	Watch	Track for teams already using Pangea security services.
Superagent	docs.superagent.sh	Guardrails, PII, prompt-injection, and verification tooling surfaced by the AI registry	Watch	Worth evaluating as the AI SDK ecosystem adds packaged security tools.

Public Readiness Scanners

External, deployed-site scanners such as Is Agentic, Ora, Cloudflare Agent Readiness, Google Lighthouse's Agentic Browsing category, and Shopify's product-page audit check what an outside agent can observe right now; they complement rather than replace a repository-level Surface audit. See Public Readiness Scanners for coverage, APIs, and how to interpret each one's findings.

Sources are linked inline.

Evaluation and Observability

Guardrails, Security, and Policy Tools

Public Readiness Scanners

On this page