Evaluating Agents: Testing Probabilistic Systems
Evals are the test suite for systems whose output is a distribution, not a value: a fixed dataset, repeatable runs, evaluators that score traces, and a comparison against the last version.
Why unit tests are not enough
A deterministic function has one correct output per input; assert add(2, 2) == 4 is a complete test. An LLM-backed system has a distribution of outputs per input: the same prompt can produce a correct answer 92% of the time and a wrong one 8% of the time, and both runs are "the system working as designed". A single passing run is one sample from that distribution, which is why "it worked in the demo" is not evidence.
The consequence is that testing shifts from *exact assertions on single runs* to *statistics over many inputs*. You still write assertions — but the unit of truth becomes a pass rate over a dataset, and the question becomes "did this change move the rate up or down, and by more than noise?"
This is not optional for agents. Every prompt edit, model upgrade, tool-schema tweak and retrieval change can silently move behaviour. Without an eval suite, you find out from users.
- Deterministic software: one input, one expected output, binary pass/fail.
- Probabilistic software: one input, a distribution of outputs, a score aggregated over a dataset.
- The eval suite is what makes "version B is better than version A" a measurable claim instead of an opinion.
The eval pipeline
An eval run is a fixed pipeline. A test dataset (see Golden Datasets) supplies inputs and expected outcomes. The agent is executed on each input, producing traces — the full record of LLM calls, tool calls and final answer (see Tracing Agents). Evaluators read those traces and emit scores: some deterministic (Deterministic Evaluators), some model-based (LLM-as-Judge). Scores aggregate into metrics (Eval Metrics: What to Measure and How), and the metrics are compared against the previous version in a regression comparison (Regression Gates and Online Evaluation).
Keep the stages separate. Storing traces means you can re-score with a new evaluator without re-running the agent (which costs money and adds noise). Separating metrics from comparison means the same numbers feed a CI gate, a dashboard and a release note.
Unit-level vs end-to-end evals
Unit-level evals isolate one component: does the router pick the right branch for 200 labelled queries? Does the retriever return the gold chunk in its top 5? Does the SQL tool produce a valid statement for a given intent? They are cheap, fast and precise — when one fails you know which component regressed.
End-to-end evals run the whole agent on a task and score the final outcome: was the refund issued to the right customer with the right amount? They catch interaction bugs that unit evals miss (a correct retriever plus a correct prompt can still produce a wrong answer when the context is ordered badly), but they are slower, noisier and harder to attribute.
Run both. Unit evals in the inner loop while iterating on a component; end-to-end evals as the gate before release. The ratio in practice is often 5–10 unit evals per end-to-end eval, mirroring the test pyramid.
- Unit-level: one component, one metric, deterministic where possible, runs in seconds.
- End-to-end: whole trajectory, outcome-based, often needs a judge, runs in minutes and costs real tokens.
- Attribute end-to-end failures by reading the trace and adding a unit eval for the component that broke.
Minimum viable eval suite
You do not need a platform to start. A directory of 30–50 JSON cases, a script that runs the agent and stores traces, three or four deterministic evaluators and a pass-rate table per version is already more evidence than most teams have. Add a judge only for the metrics that resist deterministic checks (tone, summarisation quality), and calibrate it against a human-labelled sample before trusting it.
The failure mode to avoid is the opposite: building an elaborate eval harness with no cases that reflect real traffic. Cases come first; infrastructure follows.
Key points
- LLM systems produce distributions; tests must aggregate over a dataset, not assert on a single run.
- Pipeline: dataset → agent runs → traces → evaluators → metrics → regression comparison.
- Store traces so evaluators can be re-run without re-executing the agent.
- Unit-level evals localise regressions; end-to-end evals catch interaction bugs. Use both.
- Start with real cases and deterministic checks; add judges only where deterministic checks cannot express the criterion.
- A change is only an improvement if the metrics say so beyond noise.
Version A vs version B
| Metric | How | A | B | Read |
|---|---|---|---|---|
| Task success | LLM judge + rubric | 0.81 | 0.86 | better |
| Answer correctness | LLM judge vs reference | 0.78 | 0.8 | within noise at this n |
| Tool selection accuracy | deterministic (expected tool) | 0.92 | 0.9 | within noise at this n |
| Tool argument correctness | deterministic (schema + expected values) | 0.88 | 0.93 | better |
| Avg steps | deterministic | 5.2 | 6.9 | worse |
| Hallucination rate | LLM judge (faithfulness) | 0.06 | 0.04 | better |
| Retrieval recall@5 | deterministic (golden chunks) | 0.71 | 0.71 | no change |
| Latency p95 (s) | measured | 4.1 | 6.3 | worse |
| Tokens / task | measured | 4800 | 7900 | worse |
| Cost / task ($) | measured | 0.021 | 0.034 | worse |
| Safety violations | deterministic (policy checks) | 0 | 1 | worse |
When to use — and when not to
- Before any prompt, model, tool or retrieval change ships to users.
- When adopting a new model version — the vendor changed the distribution under you.
- When a bug is reported: turn the failing case into a dataset entry first.
- When comparing architectures (single agent vs workflow) with numbers instead of anecdotes.
- Do not build a heavy eval harness before you have a working prototype and real example inputs.
- Do not use end-to-end evals as the only signal — attribution becomes guesswork.
- Do not treat a green eval run as proof of production quality; it proves non-regression on the cases you thought of.
Failure modes
- Dataset drawn from the demo script rather than production traffic, so evals pass while users fail.
- Single-run evals: a flaky case flips pass/fail across runs and nobody trusts the suite (
flaky-evals). - Scores compared without confidence intervals; a 1-point difference on 40 cases is treated as a win.
- Evaluator drift: the judge prompt changes and every historical number becomes incomparable.
- Re-running the agent for every evaluator tweak, burning budget and adding noise.