The Delivery Chain as Attack Surface
Every hop between a line of source and a running process is something that can be substituted, and each hop needs a control and a way to verify it held.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What exactly is between my source code and the process serving traffic, and which of those things am I trusting without checking?
A running artifact is the product of source you wrote, dependencies you did not, a build environment, a registry, a deployment system and a base image. You review one of those six in a pull request.
The code is reviewed, the repository has branch protection, and the pipeline is ours. The remaining risk is in the application code, which is where the security effort goes.
The reviewed code is a small minority of what ships. Most of the bytes in a typical artifact came from dependencies and a base image that nobody read.
- The reviewed code is a small minority of what ships. Most of the bytes in a typical artifact came from dependencies and a base image that nobody read.
- The pipeline has more privilege than any engineer: it can write to the registry, read every secret, and deploy to production. It is the highest-value target in the delivery path and is usually configured with the least scrutiny (CI Security).
- Tags are mutable. "Deploy version 1.4.2" resolves to whatever currently carries that tag, which is not necessarily what was tested (Tags Versus Digests).
- A dependency is resolved at build time from a public index. Without a lockfile and integrity hashes, two builds of the same commit can contain different code (Dependency Pinning).
- When a widely used library turns out to be compromised, the question is "which of our artifacts contain it, at which version, and are they deployed" — and without an inventory that question takes days (Software Bill of Materials).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Model the path as a chain of custody. At each hop something produces an output and something else consumes it, and the consumer either verifies what it received or trusts it.
- Each hop has two questions: what could be substituted here, and what would detect the substitution. Where the second answer is "nothing", you have a trust assumption rather than a control.
- The controls compose into a chain that is only as strong as its verification: signing an artifact that nothing verifies at deploy time is a ritual, not a control (Signing and Verifying Artifacts).
- The build environment is inside the trust boundary, not outside it. Provenance generated by a build step is a claim the build step makes about itself, which is worth exactly as much as the build step's integrity (The Builder Is Inside the Trust Boundary).
- This is a defensive discipline: the work is inventory, pinning, verification and response time. How the substitution is actually performed is Security Engineering's subject, and the depth is there.
The chain, hop by hop
Draw it once and the shape of the work becomes obvious: there are more hops than there are reviews, and the ones with the most privilege sit in the middle.
What each hop is trusting, and what would catch it
The useful exercise is to fill in the third column for your own pipeline. Every blank is a trust assumption, which is fine as long as it is a decision rather than an oversight.
| Hop | What could be substituted | Control | How you verify the control works |
|---|---|---|---|
| Source | A commit that no reviewer approved | Protected branches, required reviews, signed commits (Protected Branches) | Try to push directly to the release branch; it is refused |
| Dependencies | A different version, or a same-version package with different contents | Lockfile with integrity hashes; resolution fails on mismatch (Dependency Pinning) | Alter a hash locally; the install fails rather than warns |
| Base image | A rebuilt tag with different contents | Pin by digest, rebuild deliberately | The same commit built twice produces the same layers (Reproducible Builds) |
| Build environment | A step that alters the output | Ephemeral isolated builders, build definition from the reviewed source (The Builder Is Inside the Trust Boundary) | Builds cannot be reached interactively and leave no reusable state |
| Registry | A tag repointed after testing | Deploy by digest; immutable tags where the registry supports them | Attempt to overwrite a released tag; it is rejected |
| Deploy | An artifact that never went through the pipeline | Verify signature and provenance at admission (Signing and Verifying Artifacts) | An unsigned image is refused, observed rather than assumed |
| Runtime | Nothing substituted — but the blast radius if it was | Least privilege for the workload identity (Least Privilege in Production) | The workload cannot reach anything it does not need |
The day a widely used library is disclosed
This is the scenario the whole module is designed around, and it is a good test of whether the controls are real. Nothing here requires the library to have been maliciously altered — an ordinary critical vulnerability produces the same sequence.
- T+0hsignalCritical vulnerability disclosed in a library used transitively by most of the industry
- T+0h15actionTeam A queries stored SBOMs by package name; gets a list of eleven artifacts, four of them deployed
- T+0h15actionTeam B starts grepping repositories, which finds direct dependencies and misses transitive ones
- T+1hchangeTeam A has patched versions building; deploy by digest, verification unchanged
- T+4hrecoveryTeam A completes rollout and confirms via SBOM query that no deployed artifact contains the affected range
- T+2dactionTeam B is still finding artifacts, because the inventory is being built during the incident rather than before it
The difference is not tooling sophistication. It is that one team generated an inventory on every build when nothing was happening, and the other tried to reconstruct one under pressure (Software Bill of Materials).
How to do it properly
Most important first.
- Pin by digest everywhere a digest exists: base images, build actions, deployed artifacts. Tags name things; digests identify them.
- Commit lockfiles with integrity hashes and make the build fail if resolution would change them (Dependency Management).
- Give the pipeline short-lived, narrowly scoped credentials rather than long-lived keys, and separate the identity that builds from the identity that deploys (Securing the Pipeline Itself, Workload Identity).
- Produce an SBOM at build time and store it against the artifact digest so the "are we affected" question is a query rather than an investigation.
- Sign at the trusted builder and verify at admission, so an unsigned or unrecognised artifact cannot run.
- Scan, and prioritise the output by exploitability and context rather than by severity alone (Scanning, and Why a Finding Is Not a Risk).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Verification at admission and least privilege at runtime. Nothing else contains a compromised artifact, because by the time it runs it has already passed every gate that trusted it.
What can go wrong
- Controls that produce evidence nobody consumes: signatures never verified, SBOMs never queried, scan reports auto-closed.
- Pinning applied to the application's direct dependencies and not to the base image, the build tooling or the CI actions — the three places with the most privilege.
- A break-glass path that bypasses verification, left permanently enabled after the incident that justified it (Break-Glass Access).
- A registry that permits tag mutation, so an artifact that passed every gate is replaced afterwards (Artifact Registries).
- Security controls concentrated at the build and absent at deploy, so anything that reaches the registry by another route runs unchecked.
- "We use a private registry, so the supply chain is internal." The artifacts in it were built from public dependencies and public base images. The registry is a boundary for distribution, not for provenance.
- "Signing proves the artifact is safe." It proves who produced it. A compromised builder produces perfectly valid signatures (The Builder Is Inside the Trust Boundary).
- "This is a security team problem." Every control here lives in the delivery pipeline and is operated by whoever owns delivery. Security owns the threat model; delivery owns the mechanism.
- "Zero vulnerabilities is the goal." The goal is knowing what you have, shipping patches quickly, and being able to answer an industry-wide disclosure the same day (Scanning, and Why a Finding Is Not a Risk).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- A deliberately unsigned artifact was pushed and the deployment refused it. Until that has been observed, verification is a configuration, not a control.
- Given a package name and version, you can list every deployed artifact containing it in minutes.
- Every running artifact resolves to a digest, that digest to a build, and that build to a commit (Build Provenance).
- Rolling back to a previous artifact is only safe if you can say what is in it. A rollback to an older image often moves you to an artifact with more known vulnerabilities, not fewer.
- Rolling back a supply chain control — turning off verification to unblock a deploy — should require the same approval as any production change, because it silently widens the boundary for everything that ships afterwards.
- Automate: pinning enforcement, lockfile drift detection, SBOM generation, signing, verification at admission, and scanning on a schedule as well as at build.
- Keep human: the decision to accept a specific unpatched vulnerability, and any decision to bypass verification. Both are risk acceptances and need a name attached and an expiry date (Change Management).
- Every verification step is a place a deploy can fail for reasons unrelated to the change, which costs delivery speed and patience. Make failures precise or people will route around them.
- Pinning by digest means dependency updates become explicit work rather than something that happens by itself, which is the point and is a real ongoing cost.
- Strict admission control means the emergency path needs designing in advance, or it gets designed at 03:00 by someone with production access.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe chain-of-custody framing holds for any language and any platform. What differs is which hops have verification available: container ecosystems have digests and signatures throughout, while some language package ecosystems still have weaker integrity guarantees at install time.
- TOOL-SPECIFICDescribed in terms of container images, registries and a CI system. On a platform that deploys source directly — a build-on-push platform-as-a-service — several of these hops are inside the provider and the verification you can perform is whatever they expose.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — what it costs to rebuild and revalidate every artifact on short notice, which is the capability this module is really buying.