Summary

Managed platform designed for agent workflows. Provides dataset versioning, experiment tracking with diffs showing improvements, human review UI, and integration. Fastest path for TypeScript agents wanting experiment tracking with minimal setup. Core concepts: datasets (versioned test cases), scorers (grade function), evals (run agent on all test cases, aggregate scores), experiments (versioned eval runs with metadata).

Dataset management: Versioned collections of task + input + expected output
Scorer functions: Async grading (0-1 score or pass/fail)
Eval runs: Execute agent on all cases, score, aggregate
Experiments: Versioned runs with metadata (model, prompt, etc.)
Trace integration: Link to observability for debugging
Human review: UI for validating and correcting eval scores

Braintrust (braintrust.dev) is a managed evaluation platform designed for agent workflows. It provides dataset management, version control, human review, and trace integration. If you are running TypeScript agents and want experiment tracking with minimal setup, Braintrust is the fastest path.

Core concepts

Dataset - versioned collection of test cases (task, inputs, expected output)
Scorer - async function that grades a single execution (returns number 0-1 or pass/fail)
Eval - runs your agent on every test case in a dataset, scores each, aggregates results
Experiment - versioned eval run with metadata (model, prompt, etc.). Braintrust diffs experiments to show improvement

Basic eval

import * as Braintrust from "braintrust";

// 1. Define a dataset
const dataset = await Braintrust.getDataset("summarize_tasks", {
  version: "2025-04-17",
});

await dataset.insert([
  {
    input: { text: "The quick brown fox..." },
    expected_output: "A story about a fox",
  },
  {
    input: { text: "Climate change causes..." },
    expected_output: "Climate impacts explained",
  },
]);

// 2. Define a scorer
async function scoreQuality(output: string, expected: string) {
  // Code-based checks
  if (output.length < 10) return 0;
  if (output.includes("ERROR")) return 0;

  // LLM-as-judge for quality
  const grade = await Braintrust.scoreCompletion({
    prompt: `
Is this summary coherent and complete?
Summary: ${output}
Expected tone: ${expected}
    `,
    model: "claude-opus-5-5",
  });

  return grade.score;
}

// 3. Run eval
async function runEval() {
  const results = await Braintrust.eval({
    projectName: "agent_summarize",
    data: () => dataset.fetch(),
    task: async (input) => {
      // Call your agent
      const agent = new YourAgent();
      return await agent.summarize(input.text);
    },
    scores: {
      quality: async (output, input, expected) => {
        return await scoreQuality(output, expected.expected_output);
      },
    },
  });

  console.log(`Eval complete. Pass rate: ${results.summary.scores.quality}`);
  // Braintrust CLI shows link: https://www.braintrust.dev/app/...
}

await runEval();

Dataset versioning

Datasets in Braintrust are versioned. Update a dataset and create a new version without breaking old eval runs.

// Create version 2025-04-18 with additional test cases
const dataset = await Braintrust.getDataset("summarize_tasks", {
  version: "2025-04-18",
});

// Insert adds to the dataset
await dataset.insert([
  { input: { text: "..." }, expected_output: "..." },
  { input: { text: "..." }, expected_output: "..." },
]);

// Old evals on version 2025-04-17 are unaffected
// New evals can target 2025-04-18

For how to decide what belongs in these versioned datasets - sourcing, labeling, leakage checks, and drift detection - see Datasets and experiments. This page covers the platform mechanics; that page covers the curation methodology.

Experiment tracking

Experiments are automatically tracked in Braintrust. Compare metrics across versions:

async function compareModels() {
  // Baseline: Claude Haiku
  const haikuResults = await Braintrust.eval({
    projectName: "model_comparison",
    metadata: {
      model: "claude-haiku-4-5",
      promptVersion: "v1",
    },
    data: () => dataset.fetch(),
    task: async (input) => {
      const response = await fetch("https://api.anthropic.com/v1/messages", {
        method: "POST",
        headers: {
          "content-type": "application/json",
          "x-api-key": process.env.ANTHROPIC_API_KEY!,
          "anthropic-version": "2023-06-01",
        },
        body: JSON.stringify({
          model: "claude-haiku-4-5",
          max_tokens: 1024,
          messages: [{ role: "user", content: input.text }],
        }),
      });
      return (await response.json()).content[0].text;
    },
    scores: { quality: scoreQuality },
  });

  // Variant: Claude Opus
  const opusResults = await Braintrust.eval({
    projectName: "model_comparison",
    metadata: {
      model: "claude-opus-5-5",
      promptVersion: "v1",
    },
    data: () => dataset.fetch(),
    task: async (input) => {
      const response = await fetch("https://api.anthropic.com/v1/messages", {
        method: "POST",
        headers: {
          "content-type": "application/json",
          "x-api-key": process.env.ANTHROPIC_API_KEY!,
          "anthropic-version": "2023-06-01",
        },
        body: JSON.stringify({
          model: "claude-opus-5-5",
          max_tokens: 1024,
          messages: [{ role: "user", content: input.text }],
        }),
      });
      return (await response.json()).content[0].text;
    },
    scores: { quality: scoreQuality },
  });

  // Braintrust web UI shows: Haiku 0.72 → Opus 0.89 (+17%)
}

Pinning experiments to a dataset version

A comparison between two experiments only means something if both ran against the same dataset version. Pin the version explicitly and record it on the experiment so a score delta reflects the model or prompt change, not a dataset that shifted underneath the runs.

async function comparePinnedToBaseline() {
  // Pin the exact dataset version both runs evaluate against
  const dataset = await Braintrust.getDataset("summarize_tasks", {
    version: "2025-04-18",
  });

  const candidate = await Braintrust.eval({
    projectName: "agent_summarize",
    experimentName: "opus-prompt-v2",
    // Diff this run against a named baseline experiment
    baseExperimentName: "opus-prompt-v1",
    metadata: {
      model: "claude-opus-5-5",
      promptVersion: "v2",
      datasetVersion: "2025-04-18", // provenance: reproducible later
    },
    data: () => dataset.fetch(),
    task: async (input) => agent.summarize(input.text),
    scores: { quality: scoreQuality },
  });

  // Braintrust diffs opus-prompt-v2 against opus-prompt-v1 case-by-case:
  // per-case regressions, improvements, and the aggregate delta
  console.log(candidate.summary.comparison);
}

If the baseline and candidate resolve to different dataset versions, the diff is meaningless - gate on a version match before trusting the delta.

Human review integration

After an eval run, manually review outputs that scored below a threshold or are ambiguous:

async function evalWithHumanReview() {
  const results = await Braintrust.eval({
    projectName: "summarize",
    data: () => dataset.fetch(),
    task: async (input) => agent.summarize(input.text),
    scores: { quality: scoreQuality },
  });

  // Export results for human review
  const lowScores = results.results.filter((r) => r.scores.quality < 0.5);

  await Braintrust.exportForHumanReview({
    experimentId: results.experimentId,
    cases: lowScores,
  });

  // In Braintrust UI: manually update scores, add comments
  // Re-run eval with human-corrected scores
  const revisedResults = await Braintrust.eval({
    projectName: "summarize",
    data: () => dataset.fetch(),
    task: async (input) => agent.summarize(input.text),
    scores: {
      quality: scoreQuality,
      // Import human review scores
      humanOverride: async (_, __, ___, resultId) => {
        const humanScore = await Braintrust.getHumanScore(results.experimentId, resultId);
        return humanScore ?? undefined;
      },
    },
  });
}

Tracing and debugging

Braintrust integrates with OpenTelemetry. Traces from your agent are captured and viewable in the UI:

import { trace } from "@braintrust/browser";

async function tracedTask(input) {
  return await trace(
    {
      name: "summarize",
    },
    async (span) => {
      span.setTag("model", "claude-opus-5-5");
      span.setTag("input_length", input.text.length);

      const summary = await agent.summarize(input.text);

      span.setTag("output_length", summary.length);
      return summary;
    },
  );
}

// In Braintrust UI: click eval result → inspect trace
// See which tools were called, latency, tokens used

CI integration

Run evals on every PR and comment results:

// .github/workflows/evals.yml
name: Agent Evals

on:
  pull_request:
    branches: [main]

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with:
          node-version: 22
      - run: npm ci
      - run: npm run evals
        env:
          BRAINTRUST_API_KEY: ${{ secrets.BRAINTRUST_API_KEY }}
      - uses: actions/github-script@v6
        if: always()
        with:
          script: |
            const fs = require('fs');
            const results = JSON.parse(fs.readFileSync('eval-results.json', 'utf8'));
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: `Eval results: ${results.summary.scores.quality} pass rate`
            });

Links and templates

Braintrust docs: https://braintrust.dev/docs
TypeScript : https://www.npmjs.com/package/braintrust
Template: /templates/cli-and-evals/eval-braintrust.ts - full working example

Braintrust Evals