The Agent Returned 200 OK and the Answer Was Wrong
Rate, errors and duration all look healthy while the agent confidently tells a customer something untrue. Traditional service metrics measure whether the machinery ran; agent systems need metrics for whether the task was actually accomplished.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
RED metrics cannot see the failure that matters
For a conventional service, "returned 200 in 180ms" is close to a definition of working. For an agent, it means the machinery ran — the model responded, the tools executed, the response serialized. Whether the answer was *correct*, whether the right tool was chosen, whether the retrieved documents were relevant: none of these appear anywhere in rate, errors or duration (RED: Rate, Errors, Duration).
This is a genuine gap rather than an instrumentation oversight, because correctness for an agent is frequently not decidable from the response alone. A confidently wrong answer is structurally identical to a confidently right one at the HTTP layer. That means agent observability needs a signal that conventional systems get for free from exceptions and status codes, and it has to be constructed deliberately (Evaluating Agents: Testing Probabilistic Systems).
The practical consequence: keep the RED metrics — they catch the machinery failures, which are real and common — and add an outcome layer on top. The two answer different questions. RED tells you the system is running; outcome metrics tell you it is working.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| Request rate | 1,240/hour, stable | Normal. Traffic is fine. | normal |
| HTTP error rate | 0.3% | Normal. The machinery is running. | normal |
| p95 duration | 11.2s | Within the usual range for this task type. | normal |
| Tool error rate | 1.1% | Normal. Tools are executing successfully. | normal |
| Task success rate (sampled + judged) | 71% (baseline 89%) | The actual failure. Invisible to every metric above it. | smoking gun |
| Retrieval relevance @5 | 0.42 (baseline 0.78) | The mechanism: retrieval degraded after an index rebuild, so the model reasoned over irrelevant documents. | smoking gun |
What to instrument beyond RED
The outcome layer has three parts. Task success — did the run accomplish what the user asked? Measured by sampling and judging, either with human review, deterministic checks where the task has a verifiable answer, or a model-based judge with its own known error rate (LLM-as-Judge, Deterministic Evaluators). Behavioural shape — step count, loop detection, tool-selection correctness, model fallback rate. Input quality — retrieval relevance, context size, whether the right documents were present at all (RAG Evaluation).
Behavioural metrics are the cheapest and most under-used. A run that takes fifteen steps when the task type usually takes four is very likely failing, and you know that without judging the output (Where an Agent Run Actually Spends Its Time). Repeated identical tool calls indicate a loop. A spike in fallback-model usage means the primary is degraded. These are all computable from the trace with no judgement required, and they catch a large share of quality problems early.
Retrieval quality deserves separate treatment because it is the most common upstream cause of silently wrong answers in RAG-backed agents. If the retriever returns irrelevant documents, the model reasons correctly over wrong inputs and produces a confident, well-formed, incorrect answer. Nothing downstream can detect that — the model did its job. Measuring relevance at the retrieval step is what makes this diagnosable rather than mysterious.
| Layer | Metrics | Catches | Blind to |
|---|---|---|---|
| Service (RED) | Rate, error rate, duration | Crashes, timeouts, provider outages, latency regressions | Every form of wrong answer |
| Behavioural | Step count, loops, tool selection, retry rate, fallback rate | Thrashing, tool confusion, degraded primary model | Wrong answers produced efficiently in the normal number of steps |
| Input quality | Retrieval relevance, context size, document recall | Bad retrieval producing confident wrong answers | Reasoning errors over good inputs |
| Outcome | Task success, judged quality, user feedback | The failure that actually matters to users | Slow to compute; sampled, not exhaustive; judges have their own error rate |
| Cost | Tokens per run, cost per successful task | Budget regressions, thrashing runs | Quality entirely (What One Agent Run Costs, and Which Term Dominates) |
Online and offline, and why you need both
Offline evaluation runs a fixed golden dataset against a candidate change before it ships, giving a controlled comparison with known expected answers (Golden Datasets). Online evaluation samples real production traffic and judges it, catching the distribution shift that no fixed dataset anticipates — new question types, changed user behavior, a degraded upstream index. Neither substitutes for the other: offline catches regressions from your changes, online catches regressions from the world's changes (Regression Gates and Online Evaluation).
The online path needs a sampling strategy, because judging every run is usually prohibitive. Sample uniformly for an unbiased baseline, and *additionally* sample the runs your behavioural metrics flag as suspicious — high step count, loops, fallback usage, low retrieval relevance. That stratification finds far more failures per judgement spent, provided you keep the uniform sample separate so your headline success rate stays unbiased.
Close the loop with user feedback where it is available, and treat it as a biased but real signal: users report confidently wrong answers much more readily than subtly incomplete ones, so feedback undercounts a specific failure mode. Combining thumbs-down rates with judged samples and behavioural anomalies gives three partial views, and the disagreements between them are frequently where the interesting problems are.
Key points
- Rate, errors and duration are all satisfied by a fast, successful, completely wrong answer — RED measures machinery, not outcomes.
- Behavioural metrics (step count, loops, tool selection, fallback rate) are computable from traces with no judgement and catch much of the problem early.
- Retrieval relevance is the most common upstream cause of confident wrong answers, and nothing downstream can detect bad inputs.
- Offline evals catch regressions you introduce; online evaluation catches regressions the world introduces. Both are needed.
- Stratify online sampling toward runs flagged as suspicious, while keeping a separate uniform sample so the headline success rate stays unbiased.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Upstream change → retrieval: an index rebuild, embedding-model change or content update degrades retrieval relevance.
- 2Retrieval → model: the model receives plausible but irrelevant documents and reasons over them correctly.
- 3Model → response: a well-formed, confident, incorrect answer is produced in the normal number of steps and the normal amount of time.
- 4Response → service metrics: rate, errors and duration are all nominal, because nothing in the machinery failed at any point.
- 5Users → support: complaints accumulate over days with no corresponding signal, and the incident is discovered by a human reading tickets rather than by monitoring.
- • "All metrics are green, the system is healthy" — RED metrics are green for confidently wrong answers; healthy machinery is not a healthy outcome.
- • "The model got worse" — check retrieval relevance and context first; the model reasoning correctly over bad inputs is far more common than the model degrading.
- • "Users are not complaining, so quality is fine" — feedback undercounts subtly incomplete answers dramatically, and users often cannot tell a wrong answer from a right one.
- • "Our offline evals pass, so we are safe" — a fixed dataset cannot detect distribution shift in real traffic; that is what online evaluation is for.
- • "An LLM judge said it was correct" — judges have their own error rate and their own biases, which must be measured against human labels before their output is trusted as a metric (LLM-as-Judge).
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Task success rate from a sampled, judged population, reported with its confidence interval since it is an estimate rather than a count.
- • Behavioural anomalies per run: step count versus the task-type baseline, repeated identical tool calls, retry and fallback rates.
- • Retrieval relevance at k and document recall for the retrieval step, tracked independently of downstream answer quality ([[rag-evaluation]]).
- • Tool-selection correctness — how often the chosen tool was the appropriate one — which is judgeable from the trace without evaluating the final answer.
- • User feedback rates alongside judged samples, treated as a biased signal that undercounts subtle failures.
- • Define task success concretely per task type, preferring deterministic checks wherever the task has a verifiable answer ([[deterministic-evaluators]]).
- • Instrument behavioural metrics from traces immediately — they are cheap, need no judgement, and catch thrashing and tool confusion.
- • Measure retrieval relevance as its own metric so bad inputs are attributable rather than presenting as model degradation.
- • Run offline evals as a pre-deploy gate and online sampled evaluation continuously, stratified toward flagged runs.
- • Alert on task success rate and retrieval relevance the way you alert on error rate, with an SLO attached ([[slo]]).
- • A deliberately injected regression (a worse retriever, a smaller model) is detected by the outcome layer within its sampling window — if it is not, the sampling rate is too low to be useful.
- • Judged success rate correlates with user feedback trends; persistent disagreement means one of the two is measuring something other than what you think.
- • Behavioural flags precede judged failures often enough to be a useful early signal, measured as their precision and recall against judged outcomes.
- • The judge itself is validated against human labels on a held-out set, with its error rate known and reported alongside every number it produces.
- • Judging costs money and time, so outcome metrics are sampled estimates with confidence intervals rather than exact counts — treat them accordingly.
- • Model-based judges are cheap and scalable and carry their own error rate, which must be measured or the metric is unreliable in unknown ways.
- • Stratified sampling finds more failures per judgement and biases the sample; keeping a separate uniform sample doubles the judging cost.
- • A quality SLO makes quality a first-class objective and creates pressure to game the metric — deterministic checks resist this better than judges do.
- • Gate deploys on offline eval results so a prompt, model or retrieval change cannot ship with a quality regression (Regression Gates and Online Evaluation).
- • Alert on online task success rate and retrieval relevance, with burn-rate style alerting against a quality SLO (Burn-Rate Alerts: How Fast Is the Budget Going?).
- • Alert on behavioural distribution shift — a step-count histogram moving right is an early warning that costs nothing to compute.
- • Re-validate the judge periodically against fresh human labels; judge drift silently corrupts every quality metric downstream of it.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe success rates, relevance scores and traffic figures are invented to show the relationship between layers. Real baselines are entirely task-specific and must be established empirically.
- WORKLOAD-SPECIFICHow task success can be measured depends on the task. Verifiable tasks (did the ticket get the right label?) support deterministic checks; open-ended generation requires human or model judgement with all the caveats that carries.