Reproducible Builds
The same source plus the same declared inputs yields the same artefact — which requires pinned dependencies, deterministic actions and an isolated environment, in that order.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
If I build this commit again next month on another machine, do I get the same artefact — and how would I know?
Without reproducibility you cannot tell whether a difference in behaviour came from a code change or from the environment the build happened to run in, which makes every incident investigation start from an unknown.
The build is deterministic — it is the same code, so it produces the same thing. Anything else would be a bug in the compiler.
Dependencies resolve at build time. An unpinned range that resolved to 4.17.2 in March resolves to 4.19.0 in June, and the source did not change (Dependency Pinning).
- Dependencies resolve at build time. An unpinned range that resolved to 4.17.2 in March resolves to 4.19.0 in June, and the source did not change (Dependency Pinning).
- Builds embed the current time, the hostname, the absolute build path, the user, and often a random build id. Those differ every run by construction.
- Archive formats record file order and metadata. Filesystem enumeration order is not guaranteed, so a tar or a jar built twice can differ byte for byte with identical contents.
- The toolchain drifts. A CI runner image update changes the compiler, the system libraries and the locale, none of which appear in the commit.
- Anything the build fetches from the network is an undeclared input with its own timeline — a base image tag, an
apt-get install, a script piped from a URL (Build Environments). - Parallel builds can produce different link orders or different generated-file interleavings, so the artefact depends on scheduling.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Reproducibility is a property of *declared inputs*, not of source. The claim is: given this source and this stated set of inputs, the output is determined.
- It comes in levels, and conflating them causes most of the confusion. Same-machine-same-day repeatability is nearly free. Cross-machine reproducibility needs environment isolation. Bit-identical-across-time reproducibility needs every source of nondeterminism eliminated.
- The sources of nondeterminism are a finite list: time, ordering, paths, locale, randomness, concurrency, network and toolchain. Each has a standard remedy, and most ecosystems provide one.
SOURCE_DATE_EPOCHis the cross-ecosystem convention for the time input: build tools that honour it use that value instead of the current clock, so timestamps become a declared input rather than an ambient one.- Verification is the part people skip. Reproducibility is not something you configure and believe; it is something you check by building twice and comparing digests.
- The payoff is not aesthetic. It is that an artefact digest becomes a meaningful identity — the thing signing, provenance and promotion all rest on (Tags Versus Digests, Build Provenance).
Three levels, three different claims
Most disagreements about whether a build is reproducible are two people describing different levels. Naming them separately makes the conversation short.
The middle level is where most of the operational value is. The third is where the security value is, and it is much more expensive.
| Level | Claim | Requires | What it buys |
|---|---|---|---|
| Repeatable | Same machine, same day, same result | A build system with correct staleness rules | Trustworthy incremental builds (What a Build System Actually Is) |
| Cross-machine | Any machine, any time, functionally identical artefact | Pinned dependencies, pinned toolchain, no network at build time | CI matches local; a release can be rebuilt; caches are safe to share |
| Bit-identical | Byte-for-byte equal artefact | All of the above plus normalised time, paths, ordering, locale and build ids | Independent verification: someone else can rebuild and confirm the artefact was not tampered with (Artifact and Build Integrity) |
| Not reproducible | Each build is a new thing | Nothing | Nothing — and every incident question about "what is running" starts from zero |
Where nondeterminism comes from
This is a finite list, which is what makes reproducibility tractable. Each row is an ambient input that should have been a declared one.
Work down the response column in order of how often each bites: dependencies first, then time, then paths, then ordering.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Rebuild a month later | Different dependency versions in the artefact | Version ranges resolved at build time | Committed lockfile with integrity hashes, covering the transitive closure (Dependency Pinning) |
| Build twice in a row | Digests differ, contents look identical | Embedded build timestamp or build id | Honour SOURCE_DATE_EPOCH; strip build ids with the toolchain's flag |
| Build in a different directory | Binaries differ; debug info contains the path | Absolute source paths embedded for debugging | Path remapping flags, or build at a fixed canonical path inside a container |
| Build on a different filesystem | Archives differ, files are the same | Directory enumeration order is not guaranteed | Sort file lists explicitly before archiving; normalise mtime, uid, gid and mode |
| Build on a machine with a different locale | Sorted output or formatted numbers differ | Locale-dependent collation and formatting | Pin LC_ALL and TZ as declared inputs |
| Runner image updated | Everything differs, no code change | Toolchain, system libraries and headers moved | Pin the base image by digest and the compiler by version (Build Environments) |
| Build with more parallelism | Different link order or different generated interleaving | Scheduling affects output | Deterministic ordering in the action, not a fixed thread count — a fixed count hides it rather than fixing it |
| Build with the network available | Reproducible on your machine, not on a clean one | The build fetched something it never declared | Run the build with the network disabled and fix what fails (How Networks Fail in Production) |
Verify it, do not assert it
unshare --net is Linux; on macOS the equivalent isolation comes from running the build in a container with networking disabled, and on Windows from a job object or a container. diffoscope is a Debian project tool and is the standard answer on Linux; elsewhere you fall back to comparing archive listings and per-file hashes, which finds most of the same problems more slowly.Reproducibility is a testable property and almost nobody tests it. The test is short, and its failures are always informative — a mismatch is a real gap in your input model, never noise.
Run it on a schedule against a recent release commit, not only when someone remembers.
1# 1. Build the same commit twice, in two clean environments.2# Different directories on purpose: absolute paths are a common leak.3git -C /tmp/a checkout "$SHA" && (cd /tmp/a && ./build.sh --out /tmp/out-a)4git -C /tmp/b checkout "$SHA" && (cd /tmp/b && ./build.sh --out /tmp/out-b)5 6# 2. Compare the artefacts, not the logs. Logs always differ.7sha256sum /tmp/out-a/app.tar /tmp/out-b/app.tar8 9# 3. If they differ, find out where. diffoscope decompresses archives,10# images and binaries recursively and names the differing bytes.11diffoscope /tmp/out-a/app.tar /tmp/out-b/app.tar12 13# 4. The strongest single check: does the build succeed with no network?14# Anything it fetched was an input it never declared.15unshare --net ./build.sh --out /tmp/out-cStep 4 is the one that finds the most in the least time. A build that fails without the network is telling you exactly which inputs are undeclared, and it names them in the error.
How to do it properly
Most important first.
- Pin dependencies to exact versions with integrity hashes, transitively, via a committed lockfile (Dependency Pinning).
- Pin the toolchain: compiler version, base image digest, and the CI runner image where the tool permits it.
- Set
SOURCE_DATE_EPOCHfrom the commit timestamp and use build flags that strip or normalise embedded paths and build ids. - Sort explicitly wherever a build enumerates files, and normalise archive metadata — fixed mtimes, fixed uid/gid, fixed permissions.
- Take the network out of the build. Vendor or pre-fetch dependencies into a declared input rather than fetching during the build (Build Environments).
- Verify continuously: a scheduled job that rebuilds a recent commit and compares the digest against the recorded one, with a real owner for its failures.
- When digests differ, find out why rather than shrugging.
diffoscopedecompresses and compares recursively, so it names the actual bytes.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A non-reproducible build means the artefact in production may not correspond to any commit you can rebuild. Nothing contains that directly; provenance records and immutable digests limit how long you are confused (Tags Versus Digests).
What can go wrong
- Reproducibility asserted from configuration alone, never tested, and quietly broken for months by a toolchain update.
- A build reproducible only inside one specific container image, which is worth having and is a narrower claim than it sounds — the image is now part of the input set and needs its own pinning.
- Chasing bit-identity in an ecosystem that makes it very expensive, spending effort that pinning and provenance would have bought more cheaply.
- Normalising timestamps to a fixed epoch and losing the ability to tell when something was built — the timestamp belongs in the provenance record, not in the artefact.
- Vendored dependencies that drift from the lockfile because someone edited the vendor directory.
- A verification job that compares logs rather than digests, which proves nothing about the artefact.
- "We build in Docker, so it is reproducible." The image pins the OS layer. It does not pin what the build downloads, what the compiler embeds, or which base image a mutable tag pointed at that day (Mutable Servers and Immutable Images).
- "Reproducible means deterministic output for the same source." For the same source *and the same declared inputs*. Source alone never determined the artefact.
- "Different digests mean the build is broken." They mean an input differed. Finding which one is the exercise, and often the answer is a timestamp — which is still a real gap in your input model.
- "This only matters for security." It matters most during incidents: without it you cannot rebuild what is running and compare it against what you think is running (Production Debugging).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Two builds of the same commit, on different machines, produce the same artefact digest. This is the only evidence that counts.
- A scheduled rebuild-and-compare job exists, has failed at least once, and someone investigated when it did.
- The set of declared inputs is written down, and every item on it is pinned.
- A build with the network disabled succeeds — the strongest practical test that inputs are declared.
- When a digest does differ, you can say which input changed within minutes because the input set is enumerable.
- Reproducibility work is additive and revertible: removing a normalisation flag restores the previous behaviour, and the artefact stays deployable throughout.
- If chasing bit-identity is blocking delivery, stop at the level you have. Cross-machine reproducibility with pinned inputs is most of the operational value; bit-identity is the last increment and the most expensive.
- A reproducibility check failing is not a reason to block a release on its own. It is a reason to investigate before the next one, because it means your recorded inputs are incomplete (Build Provenance).
- Automate the double-build comparison on a schedule; humans will not do it and the property decays silently.
- Automate pinning updates through a bot so that the pinned state is maintained rather than frozen (Dependency Management).
- Automate digest recording at build time so that later comparison has something to compare against.
- Do not automate ignoring a mismatch. A tolerated difference means the input set is wrong, and every downstream guarantee rests on it being right.
- Fully pinned inputs mean security updates arrive only when you fetch them, so you have taken on the update duty (Dependency Pinning).
- A network-free build needs a vendor directory or an internal mirror — real infrastructure with its own operational burden.
- Bit-identity is far harder in some ecosystems than others, and the last few percent of nondeterminism can cost more than everything before it.
- Normalised timestamps make artefacts comparable and make casual "when was this built" questions require the provenance record instead of the file.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- TOOL-SPECIFICGo builds are close to reproducible by default given a pinned toolchain and module set; Nix and Bazel make hermeticity the design centre; JVM jars embed timestamps and require explicit normalisation; OCI images embed created-at metadata unless the builder is told otherwise. The same phrase describes very different amounts of work per ecosystem.
- SIMPLIFIEDPresented as levels — repeatable, cross-machine, bit-identical — which is a teaching model. Real projects sit between levels and per-artefact rather than per-repository; a service image may be cross-machine reproducible while its documentation bundle is not, and that is a reasonable place to stop.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.