Promotion
Moving one artifact forward through environments by changing what is claimed about it, never by changing its bytes.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What is actually changing when an artifact is "promoted", and how do you tell a real promotion from a rebuild wearing its name?
Environments need a way to say "this build is now allowed here". If that mechanism produces new bytes, every piece of evidence collected in the previous environment stops applying, and nobody notices because the version number is unchanged.
Promotion means moving the artifact to the next environment: build it again from the release branch with production settings, or copy it into the production registry and give it a production tag.
Rebuilding from a release branch produces a different artifact. The staging evidence — tests, scans, soak time, a manual approval — attaches to bytes that will not ship (Build Once, Deploy Many).
- Rebuilding from a release branch produces a different artifact. The staging evidence — tests, scans, soak time, a manual approval — attaches to bytes that will not ship (Build Once, Deploy Many).
- A registry-to-registry "copy" implemented as build-and-push looks identical from the outside and produces a new digest. Unless someone compares digests, the difference is undetectable.
- Re-tagging on its own carries no evidence forward. The new tag says an artifact is production-approved; it does not say what it passed, when, or who decided.
- Promotion gates degrade into calendar gates: a change waits three days in staging because the process says three days, with no statement of what those days were supposed to demonstrate.
- Without a record tying the artifact identity to the decision, an incident cannot answer "what did we know when we promoted this", which is usually the most useful question in the postmortem (Postmortems).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Promotion changes nothing about the artifact. The bytes, the digest and the provenance are fixed at build time. What changes is the set of claims attached to that identity: this digest passed integration tests, this digest soaked in staging, this digest is approved for production.
- Mechanically there are three common implementations. Attach a label or attestation to the digest in one registry. Copy the manifest and its layers into another registry, preserving the digest. Or record the decision in a release record outside the registry and let the deploy read it (The Release Manifest).
- All three are equivalent in principle and differ in one property: whether the digest survives. A manifest copy preserves it; a rebuild does not. That comparison is the only reliable test of whether a promotion is real.
- Evidence accumulates on the identity as it travels. Because the identity is the same in every environment, a staging result is a statement about the production artifact — which is the entire return on the build-once discipline.
- A separate production registry is usually about trust boundaries rather than about promotion: it limits who can push into the environment that serves users. That is a security control, and it is compatible with promotion only if the copy preserves the digest (The Delivery Chain as Attack Surface).
One digest, gathering evidence
The diagram is deliberately boring in the middle: the same identity appears in every environment. That sameness is the product. Everything interesting is in what gets attached to it as it moves right.
If any arrow in this picture produces a new digest, everything to its left stops being evidence about what runs on its right.
What each gate must produce
A gate that does not produce evidence is a delay. The test for every stage is simple: name the thing production would not know if this stage did not exist.
- 1Build and publish
Produces the identity and its provenance.
fails by Publishing without provenance, so nothing downstream can be attributed.
evidence A digest with a recorded commit, builder and input set (Build Provenance).
- 2Automated checks
Runs tests, scans and policy checks against that digest.
fails by Checks run against the source rather than the artifact, so they say nothing about what was packaged.
evidence Results attached to the digest, not to a branch name (Scanning, and Why a Finding Is Not a Risk).
- 3Pre-production deploy
Runs the artifact with realistic configuration and dependencies.
fails by An environment different enough that passing means little (Parity That Is Worth Paying For).
evidence The artifact started, served, and was observed — with the observation window stated.
- 4Promotion decision
Records that this digest is approved for production, and why.
fails by An approval with no evidence in front of the approver.
evidence A record naming the digest, the evidence and the decider (The Audit Trail).
- 5Copy or attest
Makes the approval effective where production reads it.
fails by A copy that rebuilds, changing the digest.
evidence Digest before and after the copy are identical.
- 6Production deploy
Deploys the approved digest and verifies it.
fails by Deploying a tag, so the gate applied to a different artifact than the one that ran.
evidence The running workload reports the promoted digest (A Successful Deploy Is Not Evidence of a Healthy System).
The last two steps are where most real promotion bugs live, and both are caught by one assertion: compare the digest at every hop and fail loudly when it changes.
Where the promotion record lives
All three implementations below are real and defensible. The choice is mostly about where your trust boundary is and what your registry supports — not about which is conceptually correct.
Where does the statement that an artifact is approved for an environment actually live?
when One registry, low ceremony, small team.
cost Tags are mutable and carry no evidence. The approval is a name, so it can be moved by anyone who can push, and it says nothing about why.
when The registry supports signed attestations and the deploy path can verify them.
cost Requires a verification step at deploy time and key management to go with it; an unverified attestation is decoration (Signing and Verifying Artifacts).
when You want a hard trust boundary — nothing reaches the production registry except through promotion.
cost A copy step that can fail, a second store to secure and retain, and a digest assertion you must not forget.
when Promotion spans several artifacts plus config and migrations, which is the common case for a real release.
cost Another system on the delivery path; if it disagrees with the platform, you have two sources of truth during an incident (The Release Manifest).
when Never for production, though it is extremely common.
cost The postmortem question "what did we know when we promoted this" has no answer, and the same promotion cannot be repeated deliberately.
How to do it properly
Most important first.
- Define promotion as a decision recorded against a digest. Everything else is an implementation detail of where that record lives.
- Make the promote step structurally unable to build: no source checkout, no compiler, no build arguments (Build Once, Deploy Many).
- If you copy between registries, assert afterwards that the digest is unchanged, and fail the promotion if it is not.
- Gate on evidence rather than on elapsed time: what passed, what was observed, what a canary showed (Canary Analysis: Compared Against What?).
- Carry the accumulated evidence with the artifact so the production deploy can require it — a deploy that refuses an unattested digest is worth more than a checklist (Policy as Code).
- Keep the promotion decision visible and reversible: an environment should be able to say "this digest is no longer approved here" without a rebuild.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
The gate itself is the containment: a promotion that is enforced at deploy time stops an unqualified artifact from reaching production at all. When promotion is a convention rather than a check, nothing contains it and the first signal is production behaviour.
What can go wrong
- A promote job that rebuilds, usually because it was copied from the build job and the build step was never removed.
- A cross-registry copy that changes the digest, which is a rebuild by another name and defeats every identity check downstream.
- Promotion approved by a human who has no evidence in front of them, which is a signature rather than a gate (Change Management).
- Evidence attached to a tag rather than to a digest, so it silently transfers to whatever the tag points at next (Tags Versus Digests).
- A promotion pipeline so slow that people promote in batches, which enlarges every change and makes attribution harder (Change Size: Why Small Changes Are Safer, and When They Are Not).
- The mitigation failing: an attestation check at deploy time that is satisfied by any signature, so an artifact promoted for staging passes the production gate.
- "Promotion means moving the artifact." Nothing moves in the sense that matters. Bytes may be copied; identity and evidence are what promotion is about.
- "We promote by re-tagging, so we build once." Re-tagging can be a real promotion if the digest is unchanged and the evidence is attached to the digest. If the tag is the only record, the promotion is undocumented rather than absent.
- "More promotion stages means more safety." Each stage must produce evidence that production would not otherwise have. Stages that only add delay add batch size, which adds risk (What an Environment Is For).
- "Cloud registries handle promotion for us." Cloud & Infrastructure owns the copy and replication mechanics; the decision, its evidence, and the gate that enforces it are yours.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- The digest recorded at build, at staging deploy, at promotion and at production deploy is one string throughout.
- The promote job's logs contain no build output and its duration is dominated by network transfer rather than by compilation.
- Every production deploy can name the evidence that authorised it: which tests, which environment, which observation window, which person or policy.
- Attempting to deploy an unpromoted digest to production fails at the gate rather than succeeding quietly.
- Rolling back a promotion is unusual and worth supporting: revoking approval for a digest that turned out to be bad prevents it from being re-deployed by an autoscaling event or by someone re-running an old pipeline.
- Rolling back the deployment is separate and is a selection of the previously promoted digest, which by construction still exists and still carries its evidence (Rollback: Only Useful If It Is Actually Safe).
- If promotion was implemented as a rebuild, neither rollback is clean: the previously promoted artifact may not exist as an identity at all.
- Automate everything mechanical: the copy, the digest assertion, the attestation attachment, the record.
- Automate the gate that enforces the decision at deploy time, so an unpromoted artifact cannot reach production even by hand.
- Keep the judgement human or evidence-driven for high-blast-radius environments. Automatic promotion on green is continuous deployment and is a legitimate choice; it is a decision about risk appetite, not a default (Continuous Deployment).
- Promotion gates add latency between merge and production, which is exactly the delay that makes batches grow. Every gate must earn its delay with evidence it actually produces.
- A separate production registry improves the trust boundary and adds a copy step that can fail, plus a second store to keep, secure and retain (Artifact Retention).
- Recording evidence per digest is more infrastructure than a tag, and it is the part that turns promotion from a ritual into a claim you can check.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe principle — promotion changes claims, not bytes — holds for any artifact form and any environment topology. Only the implementation of the claim differs.
- TOOL-SPECIFICWhether a cross-registry copy preserves the digest depends on the tooling: a manifest-and-blob copy preserves it, while anything that re-packages or re-compresses layers does not. Some registries offer native replication that preserves digests; some CI plugins named "promote" quietly rebuild. Verify by comparing digests rather than by reading the step name.
- ORG-SPECIFICWho may approve a promotion into production, and whether approval can be automatic, is a governance decision. A regulated environment may require a named approver and a recorded justification; a team practising continuous deployment may require only that the canary analysis passed. Both are promotion; they differ in what counts as sufficient evidence.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — a promotion gate is only as strong as the evidence it requires, and evidence attached to the wrong identity is worse than none.