The question this answers
What exactly is running in production right now, and can you prove it came from your source?
The team ships several times a day. Every deploy replaces running code with something produced by an automated pipeline from a repository many people can write to, built on a base image maintained by strangers, using dependencies fetched from a public registry at build time. That entire chain has to be trustworthy, and right now nobody can say which image digest is serving traffic.
A verifiable path from a specific commit to the exact bytes running in production: a build performed by a known identity, an artifact addressed by content digest, stored in a registry with controlled write access, and pulled by production without room for substitution.
Five hops, five substitution points
The supply chain is a pipeline with an integrity question at each hop, and the useful exercise is to ask, at every stage, "who could put something else here". The answers are uncomfortable. At source: anyone with write access to any branch the pipeline builds, plus anyone who can modify the build configuration, which frequently is not the same list. At build: every third-party action, plugin or script the pipeline executes, each running with the pipeline's credentials. At dependency resolution: whoever controls each package version and each base image, including a maintainer whose account was taken over yesterday.
At registry: anyone who can push a tag. This is the hop that surprises people, because tags are *mutable*. Pushing a new image to app:v1.4.2 replaces what that name resolves to. Anything that deploys by tag will pick up the substitution on the next pull, and nothing about the deployment record changes. At deploy: whatever the deployment configuration says to pull, resolved at pull time, by a node that will happily accept whatever the registry now returns.
The lifecycle below names the risk at each state. Two controls remove most of the risk between them: build provenance that records which source and which builder produced an artifact, and deployment by content digest so the name of the thing you deploy *is* its content — see The Container Registry and Build Once, Promote the Same Bytes.
- 1Sourcecontinuous
A commit on a branch, in a repository, with a review history and a build configuration that lives alongside the code.
The build configuration is code too. Whoever can edit the pipeline file can change what gets built and often bypasses the review rules protecting the application code.
- 2Build, under an identityminutes
A runner assumes a build identity, checks out the commit, resolves dependencies, and produces an artifact. The identity and the source commit are what provenance records.
The pipeline executes third-party actions and build scripts with its own credentials. A compromised action is a compromised build identity, and it happens regularly.
- 3Dependencies and base imageseconds to minutes
Language packages plus a base image, both pulled from registries maintained by other people.
Typosquatting, maintainer account takeover, and base images carrying unpatched system packages. Pin with a lockfile and a digest, and scan what you actually shipped.
- 4Artifactinstant
An immutable set of bytes with a content digest. The digest is the only truly stable name it has.
None, if you refer to it by digest. Everything downstream that refers to it by tag reintroduces mutability.
- 5Registrypersistent
Storage addressed by digest, with human-friendly mutable tags pointing at digests.
Anyone with push access can repoint a tag. Broad push permissions on the production repository are one of the most common and least noticed supply-chain weaknesses.
- 6What production pullsseconds
The orchestrator or host resolves the reference in the deployment configuration and pulls those bytes.
A tag reference is resolved at pull time, so two nodes in the same deployment can end up running different code. A digest reference cannot.
- 7Admission and verificationmilliseconds
The cluster or host verifies a signature and provenance attestation before running the image.
Skipping this makes every earlier control advisory. Without verification at admission, provenance is documentation rather than enforcement.
The registry is a production dependency with write access
Draw the supply chain as infrastructure and the exposure becomes visible. The registry is not a build artifact store off to the side — it is a live production dependency. If it is unreachable, no node can start a new workload, which turns a registry outage into an inability to scale or recover during an incident. If its write permissions are broad, it is an unmonitored path into production that does not go through the deploy pipeline at all.
The topology marks the two flags. First, push access on the production repository extends far beyond the pipeline in most organizations — every engineer who ever debugged a build locally, plus a handful of long-lived service accounts. Second, the base image is pulled from a public registry at build time, which means an external party is in the transitive path to production and nobody is watching what changes there.
Note also what pulls what. If nodes pull through a NAT gateway, image pulls are metered per gigabyte, which is why image size shows up as both an operations concern and a cost concern — see Why Image Size Is an Infrastructure Problem and Egress: Moving Data Costs Money, Not Just Storing It. A pull-through cache or a private registry endpoint fixes the cost and the availability problem at the same time.
Tags are names; digests are identity
The single highest-value change in this whole lesson costs one line. Deploy by digest, not by tag. A tag is a mutable pointer maintained by whoever has push access. A digest is the cryptographic hash of the image content, so a reference by digest either resolves to exactly those bytes or fails to resolve at all. There is no third outcome, and that is the entire security property.
The operational payoff is as large as the security one. With tag-based deployment, "roll back to the previous version" means hoping the tag still points where you think, and a node that pulls at 14:00 and one that pulls at 14:20 can legitimately run different code with no record of it. With digest-based deployment, what you deployed and what is running are the same string, the deployment record is a complete answer to "what is in production", and rollback is a previous digest that cannot have changed underneath you.
The cost is real and small: your deployment manifests now contain unreadable hashes, and something has to update them — which is what the pipeline is for. Keep the human-readable tag as metadata alongside the digest so people can still tell which release it is. Add signature verification at admission when the environment warrants it, and you have moved from "we believe this came from our pipeline" to "the cluster refuses to run anything that did not".
spec:
containers:
- name: checkout-api
image: registry.internal/checkout-api:v1.4.2
imagePullPolicy: Always
# 'v1.4.2' is a mutable pointer. Anyone with push access can repoint it.
# Two nodes pulling ten minutes apart may run different bytes,
# and the deployment record looks identical in both cases.spec:
containers:
- name: checkout-api
image: registry.internal/checkout-api@sha256:9f31c0ab... # immutable
imagePullPolicy: IfNotPresent
# human-readable release kept as metadata, not as the resolution mechanism
metadata:
labels: { app.release: "v1.4.2" }
# plus, at admission:
# verify signature by the build identity
# verify provenance: this digest was built from repo X, commit Y, by builder ZA digest reference cannot be repointed, so what was reviewed, scanned and approved is provably what runs, and every node in the deployment runs identical bytes. Admission-time signature and provenance verification upgrades that from a convention the pipeline follows to a rule the cluster enforces. The cost is unreadable manifests, which is why the release tag stays as a label.
Key points
- Five hops — source, build, dependencies, registry, deploy — and at each one the question is who else could put something there.
- Tags are mutable pointers; digests are content identity. Deploying by digest is the cheapest large improvement available.
- The build identity executes third-party code with its own credentials, which makes a compromised build action a compromised production deployer.
- Registry push access is almost always broader than the pipeline, and it is a path into production that bypasses the pipeline entirely.
- The registry is a runtime dependency: if it is down, nothing new can start, exactly when you most need to scale or recover.
- Provenance without admission verification is documentation. Verification at admission is what makes it a control.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • A commit triggers a build; the runner assumes a short-lived identity bound to that repository and branch — see Roles vs Static Keys.
- • Dependencies resolve from lockfiles and a base image pinned by digest, so the inputs are reproducible rather than "whatever was latest today".
- • The build produces an artifact whose digest is computed from its content and is therefore a stable, unforgeable name.
- • The pipeline pushes the artifact plus a signature and a provenance attestation recording source repository, commit, builder identity and build parameters.
- • The deployment references the digest; the node pulls exactly those bytes; admission control verifies the signature and provenance before the workload starts.
- • Promotion between environments re-uses the same digest rather than rebuilding, so what was tested is what ships — the core idea of Build Once, Promote the Same Bytes.
- • Restrict registry push to the pipeline identity. Humans get read access; emergency push is a break-glass path that alarms.
- • Pin third-party build actions and plugins by commit hash, not by tag or branch, for exactly the same reason you pin images.
- • Rebuild and redeploy on a schedule even without code changes, because base-image vulnerabilities accumulate in artifacts that never change.
- • Scan the produced artifact rather than the dependency manifest — the manifest describes intent, the artifact is what ships.
- • Keep a queryable inventory of which digest is running where. "What is in production" should be one query, not an investigation.
- • Run a pull-through cache or private registry endpoint so pulls neither traverse a metered NAT path nor depend on a public registry's availability.
- • A repointed tag: production silently runs different code with no deployment event and no diff to review.
- • Registry unavailable during an incident: no node can pull, so scaling and recovery both stall while the running instances keep serving.
- • A compromised third-party build action exfiltrating the build identity's credentials, which are production-privileged.
- • A dependency or base image compromised upstream, shipped into every artifact built afterwards, invisible until someone scans the artifact.
- • Drift between environments because staging and production were built separately from the same tag rather than promoting one digest.
- • Image bloat: a large base image multiplies pull time on every scale-out event and, if pulls cross a metered path, multiplies cost too.
- • Pull volume grows with instance count × deploy frequency × image size, so image size becomes a startup-latency and bandwidth problem well before it becomes a storage problem.
- • Registry storage grows monotonically unless retention policies delete untagged and superseded digests, and the deletion policy must not remove a digest something still runs.
- • Scanning cost grows with artifacts × frequency; scanning every build of every branch is a common and avoidable expense.
- • The dimension that runs out first is usually registry throughput during a large simultaneous rollout, which is what pull-through caches exist to absorb.
- • The chain is only as strong as its weakest write permission, and that is normally registry push or the pipeline definition file.
- • The build identity is a production-privileged identity that runs untrusted code by design. Scope it, make it short-lived, and never let it read production secrets — see Infrastructure Trust Boundaries.
- • Signature verification at admission converts every upstream control from convention into enforcement.
- • Provenance attestations answer the incident-response question directly: which commit, which builder, which inputs produced the artifact now running.
- • Cross-link the Security domain's supply-chain material for dependency compromise, artifact signing and malicious-package tradecraft; this lesson covers the infrastructure path only.
- • Registry storage and data transfer, both driven by image size and retention depth.
- • Pull bandwidth on every scale-out, replacement and deploy — metered if it crosses a NAT or leaves the provider network.
- • Scanning and signing tooling, typically priced per artifact or per repository.
- • Build minutes, which rise with rebuild frequency and fall sharply with layer caching.
- • The counterfactual: a supply-chain incident means auditing every artifact built in the affected window, which is why provenance metadata pays for itself the first time.
- • Which digest is running in each environment, as a first-class inventory rather than an inference from tags.
- • Registry push events by identity — any push not from the pipeline is worth a question.
- • Image pull failures and pull duration, which is the leading indicator of a registry problem and of image bloat.
- • Artifact age in production: an image not rebuilt in months carries every base-image vulnerability published since.
- • Admission rejections, which prove verification is switched on and actually evaluating.
- • The signal that lies: a green scan at build time. It describes the artifact on the day it was built, not the vulnerabilities disclosed since, and not the artifact currently running if the tag moved.
- • Start with digest pinning alone. It is one line, needs no new tooling, and removes the largest single substitution risk.
- • A lockfile plus a pinned base image gets most of the dependency benefit without a signing infrastructure.
- • For a small team, restricting registry push to the pipeline and rebuilding weekly beats an elaborate attestation pipeline nobody maintains.
- • A managed build service with built-in provenance is usually less work than assembling signing, attestation and verification yourself.
- • If you deploy VM images or plain artifacts rather than containers, the same reasoning applies to image identifiers and checksums — the mechanism is not container-specific.
- • Digest pinning buys reproducibility and removes tag substitution; costs readability in manifests and a pipeline step that keeps them updated.
- • Admission verification buys enforcement; costs a component in the critical path of every workload start, which will one day block a legitimate deploy at an inconvenient moment.
- • Signing and attestation add real assurance and real key-management work, including the question of what protects the signing key.
- • Frequent rebuilds keep vulnerabilities low and change what ships more often, which trades one class of risk for another.
What people believe, and what is true
A version tag identifies a specific build.
A tag is a mutable pointer. Anyone with push access can repoint it, and nothing in your deployment record changes when they do.
Our code is reviewed, so the supply chain is fine.
The pipeline definition, third-party build actions, dependencies and the base image are all in the path to production, and most of them are not reviewed by anyone on your team.
The scan passed, so the image is safe.
A scan is a snapshot against a vulnerability database on the build date. An image untouched for six months accumulates every disclosure made since.
The registry is part of CI, not production.
Every node pull, every scale-out and every replacement reads from it. A registry outage is a production incident that presents as an inability to recover.