Choosing eval metrics
“How do you choose evaluation metrics for an agentic system, and what goes wrong when you pick the wrong ones?”
What this tests
- Mapping metrics to the task and to each component
- Preference for deterministic checks over subjective scores
- Understanding process metrics (steps, tools, cost) versus outcome metrics
- Awareness of metric gaming and Goodhart effects
Answers by level
Read the beginner answer first and notice what is missing.
Start from what the task needs to be true and write checks for that. Outcome metrics depend on the task: exact match for extraction, schema validity for structured output, task completion for agents (did the ticket get created with the right fields), groundedness and citation precision for RAG. Process metrics tell you why: tool-selection accuracy, argument validity rate, steps per task, cap-hit rate, retrieval recall@k. Operational metrics: cost per success, p95 latency, error rate. See Eval Metrics: What to Measure and How.
Prefer Deterministic Evaluators wherever a check can be coded (did the JSON validate, does the answer contain the required id, did the forbidden tool get called); use an LLM-as-Judge with a specific rubric only for qualities like tone or completeness, and calibrate it against humans.
Wrong metrics mislead: a 1–10 quality score hides whether the failure is retrieval or generation; average accuracy hides a regression on the critical 5% of cases; a single outcome metric cannot say whether cost or latency became unacceptable.
Green flags · Red flags
- Distinguishes outcome, process, and operational metrics
- Chooses deterministic checks first, judges second with rubrics and calibration
- Slices by category and tracks regressions on critical cases
- Pairs metrics with counter-metrics to avoid gaming
- Ties each metric to a decision and a threshold
- A single subjective judge score as the only metric
- No process metrics, cannot explain why a score moved
- Ignores cost, latency, safety
- Unaware of Goodhart effects