Correctnesstypical

Golden Tests

Record the emitted IR, assembly or diagnostics in a file and diff against it on every change. Excellent at catching what you did not mean to do, useless at telling you whether what you meant was right — and completely dependent on the compiler being deterministic.

The question

When is a recorded-output test worth its maintenance cost, and what can it never tell me?

Source→Lexing→Tokens→Parsing→AST→Semantics→Typed→IR→Optimize→Codegen→Machine code→Link→Execute

What the program is here

A pair of texts under comparison: the artifact the compiler emits now — an IR dump, an assembly listing, a diagnostic transcript — and a checked-in file recording what it emitted when the test was written. The test is the diff. What makes this a *representation* question rather than a testing convenience is that you get to choose which representation to record, and the choice decides what the test can detect.

What this phase may assume or do

A golden test is only meaningful if the compiler is deterministic under the flags being tested: the same source, same flags and same version must produce byte-identical output, including block order, symbol ordering, temporary names and any hash-derived identifier. If iteration order over a hash map leaks into the dump, the test fails at random and gets deleted. Determinism is therefore a precondition, not a nice property — the same precondition [[reproducible-compilation]] exists to establish, arrived at from a different direction.

Key points

A golden test asserts equality with a recording, and therefore encodes current behavior rather than correct behavior.
Its strength is catching unintended change; its blindness is to any bug that existed when the file was written.
The stage you snapshot decides what the test can see — AST goldens are blind to the middle-end, assembly goldens are blind to nothing and noisy about everything.
Directive-based matching such as FileCheck lets the test state its claim instead of transcribing an artifact.
The bless command is the central weakness: large diffs get blessed unread, which checks the regression in as expected output.
Golden tests require determinism, which is the same requirement reproducible builds impose for different reasons.

What a golden test actually asserts

A golden test asserts equality with a recording. That is all it asserts. It does not assert that the recording was correct, that the output is optimal, or that the program behaves properly — only that today's output matches the day the file was written or last blessed.

This makes it extremely good at one thing: detecting *unintended* change. A refactor in the register allocator that quietly changes instruction selection in forty unrelated files shows up immediately, and that is genuinely valuable, because unintended change is how most regressions enter a compiler. It also makes golden tests structurally incapable of catching a bug that was present when the file was recorded. They encode current behavior, not correct behavior.

The practical consequence is that golden tests are a *change detector* wired to a human reviewer. The test does not decide whether the new output is acceptable; a person reading the diff does. Every property of the technique follows from that, including its main failure mode.

What it catches: unintended change anywhere in the pipeline, including in code the author did not know they affected.
What it cannot catch: a bug that already existed when the file was recorded. The file records the bug.
What it does badly: anything heuristic. Register choice, block ordering and inlining decisions change for good reasons and produce enormous diffs.
What it does well: diagnostics. Error message text and span placement are exactly the kind of thing you want pinned and reviewed.
What it requires: determinism, and a review culture that actually reads the diff.

Choosing the representation to record

targetAn assembly golden is valid for exactly one target triple, one ABI and often one compiler version. The same test on AArch64 records different registers, a different argument order and different instruction mnemonics; on Windows x64 the argument registers differ again. Real suites either restrict such tests to one host or generate a golden per target, which multiplies the maintenance cost by the number of targets.

The stage you snapshot decides what the test is sensitive to, and this is the design decision people make by accident. Record the AST and you have a parser test that is blind to everything after it. Record optimized IR and you catch middle-end changes but not instruction selection. Record assembly and you catch everything, including every change you did not care about, which is why full assembly goldens are usually too noisy to live with.

The standard compromise is a *partial* golden test: match on the lines you care about and ignore the rest. LLVM's FileCheck is exactly this — a test file carries ; CHECK: directives naming the instructions that must appear in order, so a test can assert "this loop was vectorized" without pinning the register allocation around it. That is a substantially better tool than raw text equality, because it lets the author state which part of the output is the claim.

Where to take the snapshottypical

Recorded artifact	Sensitive to	Blind to	Diff noise
Token stream	Lexer changes, trivia handling	Everything after lexing	Very low
AST / typed AST	Parser shape, desugaring, inferred types	The entire middle-end and backend	Low
Unoptimized IR	Lowering decisions, ABI shape	Every optimization	Low
Optimized IR	Pass behavior and pass ordering	Instruction selection, scheduling, allocation	Medium
Assemblytarget	Everything, including target details	Nothing in the compiler; still blind to runtime behavior	Very high
Diagnostics	Message text, span placement, suggestion quality	Code generation entirely	Low, and the diffs are readable

The blessing problem

Every golden-test system needs a way to say "yes, the new output is correct, record it" — --bless, -u, UPDATE_SNAPSHOTS=1, whatever the local spelling is. That command is the technique's central weakness. When a change produces a two-thousand-line diff across ninety files, nobody reads it. They run the bless command, the diff goes green, and any real regression inside those two thousand lines is now checked in as the expected output and defended by CI forever.

This is not a discipline failure that better people would avoid; it is a property of the tool. The mitigations are structural. Keep goldens small and focused, so a real change touches few of them. Prefer directive-based matching over whole-file equality, so a test states its claim rather than transcribing an artifact. Keep behavior-asserting tests separate from goldens, so that blessing a formatting change cannot bless a semantic one. And treat a golden diff in review as code, because that is what it is.

The sharpest version of the rule: never let a golden test be the only test of a behavior you care about. If the claim is "this loop gets vectorized", a golden asserting the vector instruction is a fine tripwire, but the test that the loop still computes the right sum has to exist too, and it has to be a different test.

Determinism is a precondition, not a bonus

A golden test compares bytes, so anything nondeterministic in the output destroys it. The usual sources are mundane and all of them have bitten real compilers: iterating a hash map whose order depends on pointer values, naming temporaries from an address, embedding a timestamp or a build path, parallel passes appending diagnostics in completion order, and ASLR leaking into any of the above.

The fix is the same fix [[reproducible-compilation]] needs for entirely different reasons — deterministic containers or sorted iteration for anything that reaches output, stable naming derived from position rather than from address, and no environment leaking into the artifact. It is worth noticing that a compiler team usually gets pushed into determinism by its golden tests years before anyone asks for reproducible builds, and that the two requirements are the same requirement.

Our own compiler asserts this directly rather than assuming it: compile every example twice and require IR, SSA, optimized IR and assembly to be identical both times. Without that assertion, a flaky golden looks like a compiler bug, and a flaky compiler looks like a flaky test.

From `scripts/compilers-sim.test.ts` — determinism asserted before anything is snapshotted

1for (const ex of EXAMPLES) {
2  const a = compile(ex.source, ex.focus ? { focus: ex.focus } : {})
3  const b = compile(ex.source, ex.focus ? { focus: ex.focus } : {})
4  assert.deepEqual(a.irText, b.irText)
5  assert.deepEqual(a.ssaText, b.ssaText)
6  assert.deepEqual(a.optimizedText, b.optimizedText)
7  assert.deepEqual(
8    a.assembly.lines.map((l) => l.text),
9    b.assembly.lines.map((l) => l.text),
10  )
11}

Every stage is compared, not just the last one. A nondeterminism introduced in SSA construction and washed out by a later canonicalization would pass an assembly-only check and reappear the moment the canonicalization stopped applying.

How it works

The steps, in the order the compiler takes them.

Run the compiler with a dump flag that emits the chosen representation to text.
Normalize what is deliberately unstable — strip absolute paths, canonicalize temporary numbering, sort collections whose order carries no meaning.
Compare the normalized text against the checked-in expected file, or run a directive matcher such as FileCheck against the named patterns.
On mismatch, fail with a diff, and offer a blessing command that rewrites the expected file from the current output.
On review, read the diff as code: the question is whether the new output is correct, which the tool cannot answer.
Guard the whole scheme with a determinism test that compiles the same input twice and requires identical output at every stage.

How it breaks

What the engineer observes when it goes wrong — not what goes wrong internally.

A one-line change produces a two-thousand-line diff, it gets blessed unread, and a real regression is now the expected output with CI defending it.
A test fails intermittently because hash-map iteration order leaks into the dump; the team marks it flaky and disables it, losing the coverage permanently.
An assembly golden recorded on x86-64 fails on every contributor with an Arm laptop, and the project quietly restricts its test suite to one architecture.
A golden records a compiler bug as expected output. Two years later somebody fixes the bug and the test fails, and the fix is reverted because "it broke a test".
Normalization strips too much — temporaries renumbered away — and a real change in the number of temporaries becomes invisible.
The only test of a vectorization feature is a golden asserting a vpaddd instruction, so when the vectorizer starts emitting correct-looking but wrong-code vectors the test still passes.

When it helps

Diagnostics. Message wording, span placement and suggested fixes are exactly the kind of thing that should be pinned, reviewed and deliberately changed — Rust’s .stderr files are the model.
Locking in a hard-won optimization: a directive asserting that this loop still vectorizes catches the day somebody’s unrelated change perturbs the cost model.
Refactors with no intended behavior change, where an empty diff across a large golden suite is genuinely strong evidence.
Frontend work, where the AST or desugaring output is small, stable and readable, and the diff is a review artifact rather than a wall of noise.

When it hurts

Anything driven by a heuristic. Register allocation, block layout and inlining change output constantly for legitimate reasons, and pinning them converts every improvement into fifty test failures.
Multi-target projects, where the golden count multiplies by the target count and most contributors can only run one of them.
Fast-moving early development, where the expected files cost more to maintain than the regressions they catch.
As a substitute for execution tests. A suite of goldens can be entirely green while the compiler emits code that computes the wrong answer.

What it costs

Every one of these is paid by something.

Golden tests buy broad sensitivity to unintended change and cost a permanent maintenance stream of diffs to review and bless, which competes directly with the attention that review requires to be worth anything.
Snapshotting a late stage such as assembly buys end-to-end coverage and pays in target-specificity and noise, so the test becomes valid on one machine and irritating on all the others.
Directive matching buys a test that states its claim and costs the ability to notice changes outside the directives, which is precisely the unintended-change detection you adopted goldens for.
Normalizing unstable output buys stable tests and costs detection of any change inside the normalized region, so every normalization rule is a small deliberate blind spot.
Requiring determinism buys working golden tests and costs real implementation constraints — sorted iteration, position-derived names, no environment in the artifact — that a compiler otherwise would not pay.

What else you could do

What a different compiler or language does instead, and when that is better.

Assert behavior instead of shape: compile and run, and compare the program’s output. Immune to heuristic churn, blind to everything that does not change the answer, and the only layer that catches wrong-code bugs.
Assert invariants instead of text: no two live values share a register, every phi has one operand per predecessor. Survives every heuristic change and catches the class of bug goldens are worst at.
Track metrics rather than text for anything heuristic — instruction count, code size, spill count — with a threshold and a noise budget, so an improvement is a green number rather than a diff.
Compare against another implementation rather than against a recording, which is [[differential-testing]], and which answers "is this right" rather than "is this the same".

See it for yourself

The flag, dump or tool that shows you this directly.

LLVM: llvm-lit -v llvm/test/Transforms/LoopVectorize runs directive-based goldens; the ; CHECK:, ; CHECK-NEXT: and ; CHECK-NOT: lines are the assertions, and FileCheck --dump-input=fail shows why one did not match.
GCC: dg-final { scan-assembler "vpaddd" } and scan-tree-dump directives inside test files, run through DejaGnu with make check-gcc.
Rust: ./x.py test src/test/ui --bless regenerates the .stderr expected-diagnostic files. Read a .stderr file to see what a diagnostic golden looks like in practice.
Jest, Insta and similar snapshot libraries for your own tools: jest -u, cargo insta review — the latter is worth noting because it forces an interactive per-snapshot decision rather than a blanket bless.
To check determinism directly before relying on goldens: compile twice to separate files and cmp them, or diff <(clang -S -o - a.c) <(clang -S -o - a.c).

Plausible wrong readings

Stated the way a confident engineer states them.

"The golden tests pass, so the compiler is correct." They pass because the output matches a recording. If the recording was wrong, they are defending the bug.
"A big diff means a big change." It usually means you touched something heuristic. The size of a golden diff is a poor proxy for the size of a semantic change, which is exactly why blessing is dangerous.
"Snapshot tests are lazy tests." They are the right tool for diagnostics and for tripwires on hard-won optimizations. They are the wrong tool for anything a heuristic decides, and the mistake is choosing the target, not choosing the technique.
"We can add goldens now and worry about determinism later." Then the tests are flaky from day one and get disabled, and you have paid the cost and kept none of the benefit.

Misconceptions

The claim, and what is actually true.

A golden test is a regression test.

It is a change detector. It becomes a regression test only if a human confirmed the recorded output was correct, and it stops being one the moment somebody blesses a diff without reading it.

If a golden test fails, something is broken.

Something changed. Whether it broke is the reviewer’s judgement, and the tool has no opinion — which is the whole reason the blessing step exists.

Go deeper

The same idea at increasing depth. Stop wherever it stops being useful.

overview

A golden test records what the compiler emitted — IR, assembly, an error message — into a file, and fails when today’s output differs. It catches changes you did not intend, which is most regressions. It cannot tell you whether the output was ever right, because it only knows what it was told to expect. And it only works at all if the compiler produces byte-identical output for the same input every time.

practical

Snapshot diagnostics and small frontend artifacts freely; snapshot assembly only with directives that name the specific instructions you are claiming. Never let a golden be the only test of a behavior — pair every "this got vectorized" directive with a test that the loop still computes the right sum. Keep expected files small so a real change touches few of them, and prefer a review tool that walks snapshots one at a time over one that blesses everything that failed.

advanced

The deep point is that golden tests turn a compiler team into a determinism-enforcing organisation whether or not they intended it. Every source of nondeterminism that reaches output — pointer-ordered iteration, address-derived names, parallel diagnostic emission, environment leakage — has to be eliminated for the suite to be usable at all. That work is the same work [[reproducible-compilation]] and [[hermetic-compilation]] require, and teams that adopted goldens early usually find themselves most of the way to reproducible builds without having set out to be. The corollary is worth stating: if your goldens are flaky, your builds are not reproducible either, and the flakiness is the cheaper symptom to notice.

How much this depends on

Nothing in this domain is true of every compiler. These say how much.

typicalThe claim that heuristic output is too noisy to snapshot describes optimizing compilers such as LLVM and GCC, where allocation and layout change often. It describes a simple teaching compiler or a formatter far less well: gofmt’s output is deliberately stable by design, and whole-file goldens are the natural test there precisely because nothing about the output is heuristic.

targetAny golden recording assembly, ABI details or register names is valid for one target triple only. The same test on AArch64 or Windows x64 records different registers and a different argument order. Suites either pin the host or maintain one expected file per target, and the second option multiplies maintenance by the target count.

implementationBlessing commands differ and so does their granularity, which matters more than it sounds: ./x.py test --bless and jest -u rewrite everything that failed, while cargo insta review walks the snapshots one at a time and forces a decision on each. The interactive form materially reduces the odds of blessing a regression.

If you were asked this in an interview

When would you snapshot assembly, and when is that a mistake?
What does a passing golden suite prove, exactly?
Your golden test is flaky. What does that tell you about the compiler, independent of the test?

Connections

Domains that do not exist yet

Testing & Reliability Engineering — Snapshot testing and approval testing as general practice
The bless-the-diff dynamic and its failure mode are identical whether the artifact is compiler assembly or a rendered UI. What is compiler-specific is that the technique imposes a determinism requirement on the system under test, which most snapshot users never encounter.
DevOps / Production Engineering — Build reproducibility and hermetic build environments
Golden tests fail for exactly the reasons non-reproducible builds fail — paths, timestamps, iteration order, environment leakage — so a compiler that can support goldens has already solved most of the reproducibility problem, and one that cannot has not.