Evaluation & Testing
Testing probabilistic systems: golden datasets, judges, deterministic evaluators, regression.
Evals are the test suite for systems whose output is a distribution, not a value: a fixed dataset, repeatable runs, evaluators that score traces, and a comparison against the last version.
A catalogue of agent metrics — outcome, trajectory, retrieval, efficiency and safety — with a precise definition and computation for each, so a number means the same thing across versions.
A golden dataset is the versioned set of inputs and expected outcomes every eval runs against; its coverage, hard cases and hygiene determine whether the numbers mean anything.
Using a model to score outputs scales evaluation to criteria code cannot express, but the judge has biases of its own and is only trustworthy after calibration against human labels.
Most of what matters about an agent run can be checked by code — schemas, tool calls, trajectory shape, budgets — and those checks are exact, free and never flaky; use them before reaching for a judge.
Offline evals gate changes in CI; online evaluation samples production traces, runs A/B or shadow comparisons and closes the feedback loop — and both need enough samples to separate signal from noise.