Evalsevalsdatasetscoverageversioning

Golden Datasets

A golden dataset is the versioned set of inputs and expected outcomes every eval runs against; its coverage, hard cases and hygiene determine whether the numbers mean anything.

Interview question
Progress

What a case contains

A golden case is more than input and expected answer. For an agent it typically records: the input (user message plus any starting state — account, documents, prior turns), the expected outcome (final answer or reference, required end state), the expected trajectory where it matters (tools and arguments that must or must not be called), metadata (category, difficulty, source, date added) and a rationale explaining why the expected value is correct so future editors can maintain it.

Expected values should be as deterministic as the task allows. "Refund of 42.00 EUR issued to order 8812" is checkable; "helpful response about the refund" needs a judge and is only as good as the rubric.

One golden case. Everything an evaluator needs, nothing that depends on the current prompt.
1type GoldenCase = {
2  id: string                       // 'refund-partial-003'
3  category: 'refund' | 'shipping' | 'account' | 'out-of-scope'
4  difficulty: 'easy' | 'medium' | 'hard'
5  input: { message: string; state: Record<string, unknown> }
6  expected: {
7    outcome: { refund_issued: boolean; amount?: number; order_id?: string }
8    tools: { name: string; args?: Record<string, unknown> }[]   // in order
9    forbiddenTools?: string[]
10    reference?: string             // for judge-based correctness
11  }
12  rationale: string
13  addedFrom: 'production' | 'synthetic' | 'incident'
14  version: number
15}

Coverage and hard negatives

A dataset is only as good as its coverage of the input space you actually serve. Sample from production traffic by category and stratify: if 60% of requests are shipping questions, the dataset should not be 60% refunds because refunds were more interesting to write. Add each incident as a case the day it happens — production failures are the highest-value examples you will ever get.

Hard negatives are inputs that look like they should trigger an action but must not: a refund request for an order outside the return window, a "delete my account" from an unverified session, a question that resembles a documented topic but is out of scope, a prompt-injection payload inside a pasted email. Without them, an agent that says yes to everything scores perfectly.

Include the boring cases too. A dataset made only of hard cases will show a 40% success rate and tell you nothing about the 90% of traffic that is easy.

  • Stratify by real traffic categories; oversample only the categories where failures are expensive.
  • Hard negatives: should-refuse, should-ask-for-clarification, should-not-call-tool, injection attempts.
  • Edge inputs: empty message, very long message, non-English, ambiguous entity references, multiple requests in one turn.
  • Every production incident becomes a case with addedFrom: "incident".

Size, versioning and contamination

Size guidance: 30–50 cases per category is enough to notice regressions of ~10 points; 200+ overall to detect a 3–5 point change with reasonable confidence; smaller than 20 per category and you are reading noise (see the statistics in Regression Gates and Online Evaluation). Judge-scored cases need more samples than deterministic ones because the judge adds its own variance.

Version the dataset like code: commit it, tag it, and record the dataset version in every eval result. When you add or fix cases, the old numbers are no longer comparable — re-run the baseline on the new version before comparing. Never edit an expected value to make a failing case pass without a written rationale.

Contamination happens when the dataset leaks into what is being tested. Few-shot examples in the system prompt copied from golden cases inflate scores. Cases scraped from documentation the RAG index also contains make retrieval look perfect. And if you tune the prompt against the full dataset for weeks, you have overfit to it — hold out a slice the tuning loop never sees, and refresh it periodically from production.

  • Split: a dev set for iterating and a held-out set for the release gate.
  • Record dataset_version, prompt_version, model and evaluator_version with every metric.
  • Retire cases that no longer reflect the product; do not let the dataset become a museum.

Synthetic cases

Generating cases with an LLM is a reasonable way to bootstrap coverage of rare categories, but synthetic inputs share the generator's biases and tend to be cleaner than real traffic. Use them to fill gaps, label them as synthetic, and make sure every synthetic expected value is reviewed by a person. Over time, replace synthetic cases with production samples of the same category.

Key points

  • A case records input, expected outcome, expected/forbidden tools, metadata and rationale.
  • Stratify by production traffic; incidents become cases immediately.
  • Hard negatives are what stop a say-yes-to-everything agent from scoring 100%.
  • Roughly 30–50 cases per category; 200+ overall to see 3–5 point changes.
  • Version the dataset and record the version with every metric.
  • Hold out a slice the prompt-tuning loop never sees; watch for contamination via few-shot examples and indexed docs.

When to use — and when not to

Use it when
  • As the first artefact of any eval effort — before evaluators, before tooling.
  • When onboarding a new category of traffic or a new tool.
  • After every incident, to prevent recurrence.
Avoid it when
  • Do not write expected values from the current agent's output — that freezes today's bugs as truth.
  • Do not rely on synthetic-only datasets for release gating.
  • Do not grow the dataset without stratification; 500 easy cases still hide the hard 5%.

Failure modes

  • Dataset built from the demo script; passes while production fails on real phrasing.
  • No hard negatives, so a permissive agent scores perfectly and issues refunds it should refuse.
  • Expected values silently edited to match new output; regressions become invisible.
  • Few-shot examples in the prompt copied from the dataset, inflating scores.
  • Dataset version not recorded; two runs are compared across incompatible case sets.
← Eval Metrics: What to Measure and HowLLM-as-Judge →