Evals

An eval is a scored check that sends messages to your agent and grades what it did. Each eval lives in a .eval.ts file under evals/, and eve eval runs them. Use evals to catch regressions when you change a prompt or a tool.

Evals talk to a real agent server over the same HTTP API as your users, through the TypeScript client. A passing eval means the agent started, accepted a request, and produced the asserted result. To control what tools return, define tool stubs in the eval.

defineEval

The evals/ directory sits at the app root. Each .eval.ts file exports one eval, or an array of evals that runs once per dataset row. The file path is the eval's id, so evals/weather/brooklyn-forecast.eval.ts becomes weather/brooklyn-forecast. Don't add an id or name.

my-agent/
├── agent/
├── evals/
│   ├── evals.config.ts
│   ├── smoke.eval.ts
│   └── weather/
│       ├── brooklyn-forecast.eval.ts
│       └── no-tools-for-greetings.eval.ts
└── package.json

An eval is one async test(t) function. It sends messages and records assertions through t:

import { defineEval } from "eve/evals";
import { includes } from "eve/evals/expect";

export default defineEval({
  description: "Basic message and tool-usage coverage for the weather agent.",
  async test(t) {
    const turn = await t.send("What is the weather in Brooklyn?");
    t.succeeded();
    t.calledTool("get_weather");
    t.check(turn.reply, includes("Sunny"));
  },
});

test is the only required field; description, judge, tags, metadata, timeoutMs, and reporters are optional. The init template adds evals/**/*.ts to tsconfig.json, so eval code type-checks with the app.

evals.config.ts

Every evals/ directory needs exactly one evals.config.ts at its root. It sets the defaults that all evals share:

import { defineEvalConfig } from "eve/evals";
import { braintrustReporter } from "eve/evals/reporters";

export default defineEvalConfig({
  judge: { model: "typesafe-ai/jev" },
  reporters: [braintrustReporter({ projectName: "my-agent" })],
});

Every field is optional:

CLI flags (--max-concurrency, --timeout, --repeat) and per-eval values take precedence over config defaults.

Run setup and teardown

Use setup to start shared resources or run migrations before eve starts the local agent server. Return shared values from setup, and release resources in teardown:

import { defineEvalConfig } from "eve/evals";
import { startTestDatabase } from "./database.js";

export default defineEvalConfig({
  async setup() {
    const database = await startTestDatabase();
    return { database };
  },
  async teardown(context) {
    await context?.database.close();
  },
});

context is typed from the setup return value. To use that type in an eval file, pass the config's type to defineEval:

import { defineEval } from "eve/evals";
import { equals } from "eve/evals/expect";
import type config from "./evals.config.js";

export default defineEval<typeof config>({
  async test(t) {
    const result = await t.context.database.query("SELECT 1 AS value");
    await t.require(result.rows[0]?.value, equals(1));
  },
});

setup, every eval, and teardown share one context object in the eve eval process. The agent server never sees it, so class instances and functions work as values. Evals that run at the same time share these resources, so isolate any database records they change.

setup runs once per eve eval command, before the agent server starts. teardown runs after the local server and its sandboxes stop, even when setup, startup, or an eval fails. A teardown failure fails the command. The per-eval timeoutMs doesn't apply to either hook.

setup can return nothing, and teardown works without setup. teardown receives undefined when setup returns nothing or throws, so setup must clean up anything only it can reach.

Deterministic fixture models

mockModel gives a test agent fixed replies, so it runs without calling a model provider. A fixed reply is one line:

import { defineAgent } from "eve";
import { mockModel } from "eve/evals";

export default defineAgent({
  model: mockModel("A deterministic reply"),
});

Pass a callback when the reply depends on the conversation. It receives lastUserMessage, userMessages, userMessageCount, the available tools, and prior toolResults:

export default defineAgent({
  model: mockModel(
    ({ lastUserMessage, userMessageCount }) => `Turn ${userMessageCount}: ${lastUserMessage}`,
  ),
});

The callback can return { text, toolCalls, usage } for deterministic tool loops or explicit token counts. Use the options form when a fixture also needs a custom model identity:

model: mockModel({
  modelId: "weather-script",
  provider: "my-fixtures",
  respond: ({ toolResults }) =>
    toolResults.length === 0
      ? { toolCalls: [{ name: "get_weather", input: { city: "Brooklyn" } }] }
      : `Weather: ${JSON.stringify(toolResults[0]?.output)}`,
});

Omit the response to reply "Mock response". mockModel supports streamed and non-streamed replies and estimates token usage. Its context window is 1,000,000 tokens, so eve never compacts the conversation. To test compaction, set modelContextWindowTokens on the agent.

The model is part of the agent definition, so use mockModel in a separate test agent. That agent stays mocked locally and when you deploy it as an eval target.

The t context

t both sends messages and records assertions, so an eval is ordinary code that sends a message, then asserts on the reply.

• Send: t.send(...) starts a conversation with its first message and returns the turn. t.session() starts an empty conversation. Send follow-ups with turn.session.send(...), and read the reply from turn.reply. The full list of session methods is in the drive API.
• Assert: use the three surfaces in the next section.

Three assertion surfaces

t.judge grades the reply of the most recently finished turn. Pass { on: turn.session.transcript } to grade a whole conversation.

Gate vs soft

A failed gate marks the eval failed, and eve eval exits non-zero. A soft assertion appears in reports. A soft assertion below its threshold marks the eval scored, which fails the run only under --strict. A soft assertion with no threshold never fails.

t.judge(...), t.score(...), and similarity are soft by default; the rest are gates. Chain .gate(threshold?), .soft(threshold?), or .atLeast(threshold) to change one. See Severity.

t.succeeded(); // gate
t.calledTool("get_weather").soft(); // record as a metric, don't gate
t.judge("cites a source"); // soft, tracked (no threshold)
t.judge("The response is consistent with output.reference.", {
  on: { response: turn.reply ?? "", reference },
}).atLeast(0.7); // soft, gated under --strict at 0.7

Use await t.require(value, assertion) for a gate that must pass before the eval continues. Call t.skip(reason) first to skip an eval on a target that doesn't support it.

Run evals with eve eval

eve eval                       # run all discovered evals against a local dev server
eve eval weather               # run one eval, or every eval under evals/weather/
eve eval --url https://<app>   # target an existing server or deployment

Exit code 0 means every eval passed its gates. See Running evals for the full flag list, exit codes, and CI guidance.

A good baseline

Start with a few small smoke evals: t.succeeded() plus one or two content checks, with dataset files in evals/data/. Add a judge or Braintrust when you need fuzzy grading or a shared place to review results. In CI, run eve eval --strict so soft threshold misses fail the build.

What to read next

• Cases: single-turn evals, scripted multi-turn evals, and dataset fan-out
• Assertions: run-level methods and t.check value assertions, with matchers and severity
• Judge: LLM-as-judge grading and the judge model
• Targets: local vs remote targets for the same eval files
• Reporters: Braintrust experiments and JUnit XML
• Running evals: the eve eval CLI, exit codes, and artifacts
• Tools: the surface most evals assert on