Build Once, Deploy Many
One artifact is built, then promoted unchanged through every environment, and environment differences arrive as configuration rather than as a rebuild.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
If staging and production need different settings, why not just build a separate artifact for each?
Environments differ, and there are exactly two places that difference can live: inside the artifact, or outside it. Putting it inside seems tidier and destroys the meaning of every test you ran.
Give each environment its own build. The staging pipeline builds with staging settings, the production pipeline builds with production settings. Each environment gets precisely what it needs and nothing it does not.
The artifact that passed staging is not the artifact that ships. Every test result, scan result and manual approval refers to a set of bytes that will never serve a user.
- The artifact that passed staging is not the artifact that ships. Every test result, scan result and manual approval refers to a set of bytes that will never serve a user.
- The two builds happen at different moments. Base images, transitive dependencies, compiler patch versions and CA bundles can all differ between them, so production runs code that was never assembled anywhere else (Reproducible Builds).
- A production deploy can now fail at build time. The change is approved, the window is open, and the pipeline is red because a dependency host is down — a delivery failure caused entirely by the decision to rebuild.
- Rollback rebuilds an old commit against today's inputs. That is a fresh, untested artifact being deployed during an incident, which is the worst possible moment to introduce a novel binary.
- The build-time difference is invisible at runtime. Nothing about a running process announces "this was compiled with the staging flag", so the class of bug that only exists in the production build is discovered by users.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The principle is a conservation rule: the artifact is the constant, the environment is the variable. One build produces one set of bytes with one identity, and that identity travels forward through every environment unchanged.
- Difference is reintroduced at start-up, from outside: environment variables, mounted config, a config service, a secret manager, workload identity. The process reads them and behaves differently; the bytes are the same (Build-Time and Runtime Configuration).
- This is what makes evidence accumulate. Because staging and production share one identity, a staging pass is a statement about the artifact in production. Rebuild and the statement is about something else, so the evidence chain is severed at the moment it would have been useful.
- Promotion, in this model, changes no bytes at all. It changes what is claimed about the identity — that this digest is now permitted in this environment (Promotion).
- The rule has a real edge: some differences genuinely cannot be deferred to run time, notably CPU architecture and, for some stacks, ahead-of-time-compiled asset URLs. Those are handled by building a multi-architecture set from one build invocation or by making the value runtime-resolvable — not by running the pipeline twice with a different flag.
Two pipelines, and where the evidence goes
Both shapes below ship the same commit to production. Only one of them can tell you that the thing which passed the tests is the thing serving traffic.
Notice what the left-hand shape does to a rollback. The previous production artifact is not a stored object; it is a build that would have to be performed again, from a commit whose dependency world has moved on.
commit a1b2c3d -> build(--env=staging) -> image A -> staging (tests pass on A) -> build(--env=prod) -> image B -> production (B never tested) rollback: build(--env=prod) of the previous commit -> image D (new)
commit a1b2c3d
-> build -> app@sha256:9f3e...
-> staging + staging config (tests pass on 9f3e)
-> production + production config (same 9f3e)
rollback: select the previous digest, already stored, already runOn the left, three artifacts exist for one change and only one of them was ever tested. Every gate downstream of the build is inspecting a binary that will not ship. On the right there is one identity, so a staging pass is a statement about production, and rollback is a lookup rather than a compile.
Where the difference is allowed to enter
The artifact crosses environment boundaries untouched. Everything environment-shaped joins at start-up, which is also where it can be validated and where a wrong value can fail fast and loudly.
Config, secrets and identity arrive by different mechanisms and have different blast radii, but they share the property that changing them does not change the artifact — which is exactly why config needs its own rollback story.
1# Per-environment build: two invocations, two sets of bytes2ARG APP_ENV=staging3RUN ./configure --env=${APP_ENV} # baked in, invisible at runtime4 5# Environment-agnostic build: one invocation, config joins later6ENV APP_ENV=""7CMD ["./server"] # reads APP_ENV, API_URL, DB_URL at start-upThe first form is not wrong because build arguments are bad — they are the right way to pass a version or a commit SHA. It is wrong because this particular argument makes the artifact belong to one environment, so the number of artifacts per change silently becomes the number of environments.
How the rule gets broken without anyone deciding to break it
Nobody argues for rebuilding per environment. It arrives as a small convenience under time pressure, and the failure it causes surfaces weeks later as a bug that "cannot be reproduced anywhere else".
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Promote job copied from the build job | Production deploy takes as long as a build and occasionally fails on dependency fetch | The promote stage still contains a checkout and a build step | Remove the build step; the promote job should have nothing to compile |
| Cross-registry copy done by rebuilding | Digest in the production registry differs from the one tested | The copy was implemented as build-and-push rather than as a manifest copy | Copy the manifest and layers; assert the digest is unchanged afterwards (Promotion) |
| Environment baked via build argument | A bug that exists only in production and only in production builds | Compile-time branch on the environment name | Move the branch to run time and validate the value at start-up (Validate at Startup, Fail Clearly) |
| Front-end bundle built per environment | Staging and production bundles have different asset hashes for one commit | API base URL inlined at bundle time | Serve a small runtime config document the bundle fetches, or inject at deploy time outside the hashed assets |
| Sidecar or migration job rebuilt | Application promoted correctly; the companion container is a fresh build | The rule applied only to the main image | Every container in the deployment is an artifact and is promoted by digest |
| Rollback triggers a rebuild | Recovery time includes a full pipeline run | Old artifacts were deleted, or the deploy path takes a ref rather than a digest | Retain artifacts that reached production, and deploy by digest (Artifact Retention) |
How to do it properly
Most important first.
- Build once, in one place, from a pinned source commit, and push exactly one artifact identity.
- Express every environment difference as configuration the process reads at start-up, and validate it there so a wrong value fails immediately instead of at 3am (Validate at Startup, Fail Clearly).
- Deploy by identity, not by label, so promotion cannot silently pick up a different build (Tags Versus Digests).
- Make the pipeline structurally incapable of rebuilding on promotion: the promote job should have no build step to run, and no source checkout to run it from.
- When a value truly must be baked in, make it a build input recorded in provenance rather than a branch in the pipeline, so at least the difference is visible (Build Provenance).
- Front-end bundles are the most common exception attempt. Prefer a runtime-fetched config document over per-environment bundles; if you must inline it, inline it at deploy time into a wrapper, leaving the hashed assets untouched.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Nothing downstream. The divergence is created at build time, so canaries, staged rollouts and manual approvals are all exercising a different binary than the one they are meant to be gating (Reducing Blast Radius).
What can go wrong
- A pipeline that says "promote" and quietly rebuilds because the promote job starts with a checkout and a build step copied from the original job.
- A registry-to-registry copy implemented as a rebuild, which produces new bytes under an old name — indistinguishable from a real promotion unless you compare identities.
- Config that is technically external but effectively baked: an image whose entrypoint script writes the environment-specific file itself, chosen by a build argument.
- The rule followed for the application and abandoned for its sidecars, agents or migration jobs, which are rebuilt per environment and carry the divergence instead.
- Build-once adopted without config validation, so the single artifact now fails in an environment nobody tested the config for — the failure moves rather than disappearing.
- "Build once means one artifact forever." It means one artifact per change. New commit, new artifact; the same commit is never built twice for two destinations.
- "We build once because we only run one pipeline." Running one pipeline that builds separately per stage is not building once. The test is whether the identities match.
- "Config in the image is fine if it is only defaults." Defaults are fine; defaults that differ per environment are not, and the difference is invisible from outside.
- "Docker gives us this automatically." Containers make it easy and do not enforce it. A
--build-arg ENV=prodpipeline is per-environment building with an image format.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- The artifact identity recorded by the staging deploy and the one recorded by the production deploy are the same string.
- The production pipeline's promote stage contains no build step, and its logs contain no compiler output.
- A rollback completes without producing a new artifact identity.
- Every environment reports the same build id from its version endpoint for a given release (From Developer to Users).
- This is what makes rollback cheap: the previous artifact still exists, still has an identity, and is already known to have served traffic. Selecting it is a deployment, not a build (Rollback: Only Useful If It Is Actually Safe).
- If the problem is configuration rather than code, the artifact stays and the config rolls back — a different, usually faster, reversal (A Config Change Is a Production Change).
- Reversing the *practice* is easy and worth naming: reintroducing a per-environment build is a one-line pipeline change, which is why it happens by accident during a deadline.
- Automate the enforcement, not just the intent: have the deploy step reject an artifact identity that no earlier environment recorded, so a rebuild cannot reach production even if someone adds one.
- Automate identity propagation so the digest deployed to staging is the input to the production deploy, rather than a name resolved again later.
- Keep the promotion decision human or evidence-gated. The mechanism should make the wrong artifact impossible; it should not decide that this artifact is ready (Promotion).
- The artifact must be built to be environment-agnostic, which is real design work: no compile-time endpoints, no baked credentials, and a start-up path that tolerates configuration arriving from several sources.
- Configuration becomes a first-class deployable with its own blast radius, and config changes are now capable of causing incidents that a rebuild would have caught at compile time (Configuration Drift).
- Some optimisations that depend on compile-time knowledge — dead-code elimination behind an environment flag, for instance — are unavailable. You trade a little efficiency for the ability to say what is running.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe rule holds for any artifact form. What varies is how hard the platform makes it to break: container and serverless platforms take configuration as environment by default, while system-package and machine-image workflows offer per-environment builds as a first-class feature.
- PLATFORM-SPECIFICMulti-architecture targets are the one honest exception. A container platform serving both amd64 and arm64 needs distinct per-architecture layers, which is why registries support a manifest list: one identity that resolves to the right architecture at pull time. That is one build invocation producing one addressable artifact, not two pipelines.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — the value of a test result is bounded by whether the thing tested is the thing shipped.