The Pipeline as Infrastructure

The pipeline is usually drawn as a row of boxes, which hides the two things that matter at each hop: what concrete thing now exists, and which identity was allowed to make it. A commit becomes a test result, becomes a build output, becomes an image with a content digest, becomes a running revision behind a load balancer. Each transition is performed by an identity with a specific permission, and the permissions get more dangerous as you move right.

The last hop is the one worth staring at. Something has to be able to change production. If that something is a CI runner, then the CI system can change production — and so can anything that can run a job on it, which includes any pull request from a fork if the configuration is careless, and any dependency that executes code at install time. A build system with deploy credentials is a production system whose access control is a YAML file in a repository.

The mitigations are structural. Separate the identity that builds from the identity that deploys; the build identity needs to push to a registry and nothing else. Use short-lived, workload-identity-federated credentials rather than long-lived keys stored as CI secrets. Scope the deploy identity to the specific resources it updates, not to the account. And gate the deploy stage on the protected branch, so a job triggered by an arbitrary pull request cannot reach it. See Roles vs Static Keys and Human vs Workload Identity.

Ask "what is the most privileged identity in this system?" and the honest answer in most organisations is the deploy role, not any human. It can replace the running code of every service. A supply-chain attacker does not need to break your application; they need one dependency that runs at build time on a runner that holds that role.

This is what makes build-time execution such a sharp edge. A package install script, a test helper, a linter plugin — all of them execute with whatever the runner has. If the runner has production deploy rights, then so does every transitive dependency in the build. Splitting build and deploy into separate jobs with separate identities is not bureaucracy; it is the control that makes that sentence false.

The rest is conventional least privilege applied to a place people forget to apply it: no wildcard resources, no long-lived static keys in CI secret stores, an approval gate on the production deploy job, and audit logging that ties every production change to a pipeline run and through it to a commit and a reviewer.

Pipeline failures cluster into three groups and they want different responses. Correctness failures — a test caught something — are the pipeline working. Infrastructure failures — runner out of disk, registry throttling, an expired credential — are yours to operate, and they present as flakiness that erodes trust in the first group. Path failures are the interesting ones: the deploy succeeded, the pipeline is green, and production is broken, because the pipeline verified the deploy mechanism and nothing verified the workload.

The runner fleet is real infrastructure with real capacity limits. A pipeline whose queue depth grows on Monday morning is a capacity problem, and the symptom — "CI is slow" — gets treated as a tooling complaint rather than as saturation. Runners also need egress to pull dependencies, which puts them behind the same NAT and the same rate limits as everything else. A build failing with connection timeouts is often a network story, not a build story.

The single most valuable pipeline metric is not build duration. It is the time from a merged commit to that change serving traffic, tracked as a distribution — because the long tail is where the manual steps are hiding.