Supply ChainTOOL-SPECIFICCLOUD-SPECIFICORG-SPECIFIC

Securing the Pipeline Itself

CI is the most privileged system in the delivery path and the least reviewed — it can read every secret, write to the registry, and deploy to production.

What actually happens How to do it properly Evidence and rollback

The question, the obvious approach, and why it breaks

Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.

The production question

What can our CI system reach, and what would it take for something that runs in it to reach further?

The problem

A build job executes code — yours, your dependencies', and your build tooling's — with credentials that no individual engineer holds, on infrastructure whose configuration is rarely reviewed with the same care as application code.

What teams do first

CI runs our code on our runners with our secrets. Access to the repository is controlled, so the pipeline is as secure as the repository.

How it breaks

The pipeline's credentials are usually broader than any human's: registry write, cloud deployment rights, and read access to the whole secret store (Secrets in CI).

How it breaks in production

The pipeline's credentials are usually broader than any human's: registry write, cloud deployment rights, and read access to the whole secret store (Secrets in CI).
A build runs third-party code by construction — dependency install scripts, build plugins, reusable pipeline steps — all with the job's privileges.
Reusable pipeline steps referenced by a floating tag can change content without any change in your repository (Dependency Pinning).
Workflows triggered by pull requests from outside the organisation are, by default in several systems, running proposed code. Whether that job has access to secrets is a configuration detail with very different consequences depending on the answer.
Long-lived cloud keys stored as CI secrets are valid from anywhere, do not expire, and are frequently the actual crown jewels (Workload Identity).
Runners reused between jobs carry state — caches, checkouts, credentials in memory or on disk — from one job to the next.

Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve

What is actually happening

Underneath the tooling, which is the part that survives a change of tool.

Think of the pipeline as a production system with an unusual property: it deliberately executes code that has not been fully reviewed, with high privilege. Every control follows from that.
Separate the identities. The identity that builds does not need to deploy; the identity that deploys does not need to build. Splitting them means a compromise of one does not automatically produce a production change (Privilege Separation in Security Engineering).
Prefer short-lived credentials issued to the job through a workload identity federation over long-lived stored keys. A token that expires in minutes and is scoped to one repository is a fundamentally smaller thing to lose (Short-Lived Credentials).
Isolate the execution: ephemeral runners that are destroyed after each job, so state cannot cross between jobs.
Pin everything the pipeline consumes by digest or commit — base images, container images used as build steps, reusable actions — so the pipeline definition fully determines what runs.
The privileged parts of the pipeline should be triggered only from a protected branch, not from a proposed change (Protected Branches, Required Checks).

Least privilege, expressed per job

TOOL-SPECIFICShown in one CI system's syntax. The concepts — deny-by-default permissions, a token exchanged for a short-lived cloud credential, immutable step references, an approval-gated deployment target — exist in most systems under different names. The specific field names do not transfer; the four properties do.

The single highest-value change in most pipelines is narrowing default permissions and pinning what the pipeline consumes. Both are visible in the definition, which means both can be linted.

A build job that cannot deploy and a step that cannot change under it

1permissions: {}          # deny by default at workflow level
2 
3jobs:
4  build:
5    permissions:
6      contents: read         # read the source
7      id-token: write        # request a short-lived cloud token
8      packages: write        # push the artifact
9    steps:
10      # pinned by commit, not by tag: the step cannot change
11      # without a commit in this repository
12      - uses: actions/checkout@8f4b7f84864484a7bf31766abe9204da3cbe65b3
13      - run: make build
14 
15  deploy:
16    needs: build
17    permissions:
18      contents: read
19      id-token: write        # a different, deploy-scoped identity
20    environment: production  # requires approval before it runs
21    steps:
22      - run: make deploy

Two properties are doing the work: the build job has no deploy capability, and the deploy job runs behind an approval. Neither costs anything to add and both have to be added deliberately.

Which trigger, which privilege

Most pipeline compromises are not clever. They come from a privileged job being reachable by a trigger that should not have been able to reach it.

Trigger	Code being run	Secrets it should have	What it should be able to do
Push to the release branch	Reviewed and merged	Build and signing identity	Build, sign, publish, deploy behind an approval
Pull request, same repository	Proposed by a member, not yet reviewed	None that can write anywhere	Build and test into a throwaway location
Pull request from a fork	Proposed by anyone	None	Build and test only, on an isolated runner
Scheduled run	Reviewed, but running unattended	Narrowly scoped to its task	Only what the task needs — often read-only scanning
Manual dispatch	Reviewed, run by a named person	As per the target environment	Whatever it does, logged against that person (The Audit Trail)

How privilege escapes a pipeline

Each row is a boundary that exists on paper and is crossed by an ordinary configuration choice. None of them require an exotic technique — the depth on how these are exploited is Security Engineering's (CI/CD Security).

Boundary crossings in CI

Trigger	Symptom	Cause	Response
Fork pull request runs the main workflow	Nothing visible	The workflow is not separated by trust level, and secrets are available to all jobs	Split into a low-trust validation workflow and a privileged one triggered only from the protected branch
Reusable step referenced by tag	Pipeline behaviour changes with no commit in your repository	Tags are mutable references	Pin by commit hash and lint for it; update via a reviewed pull request
Shared self-hosted runner	Occasional cross-job weirdness, stale caches	State persists between jobs of different trust levels	Ephemeral runners, isolated per trust level
Cache restored across trust levels	A privileged build consumes an artifact a low-trust job wrote	Cache scoping is broader than the trust boundary	Scope caches to the branch and trust level; never restore an unprivileged cache into a release build (Caching in CI)
One identity builds and deploys	Convenient; nothing looks wrong	No separation between producing an artifact and releasing it	Split identities, and require the deploy job to verify the artifact signature (Signing and Verifying Artifacts)
Static cloud key stored as a CI secret	Works from anywhere, forever	No federation configured, or it was never migrated	Move to short-lived federated credentials; where impossible, scope narrowly and rotate on a schedule (Rotation That Applications Survive)

How to do it properly

Most important first.

Default every job to the minimum permissions and grant upward explicitly, per job. Broad default permissions are the most common finding and the easiest to fix.
Pin reusable steps by commit hash, not by tag. A tag is a name the author can repoint.
Run untrusted-trigger jobs — anything from a fork or an external contributor — without secrets and without deployment rights, in a separate workflow from the privileged one.
Use ephemeral runners; if self-hosted runners are needed for network access, isolate them per trust level and never share one between public and private triggers.
Scope each secret to the jobs that need it, and prefer federated short-lived credentials over stored keys wherever the provider supports it.
Protect the branch that triggers releases with required review and required checks, and treat pipeline definition changes as changes to production infrastructure, because that is what they are.
Log what the pipeline did with its privileges, into a store the pipeline cannot rewrite (The Audit Trail).

How much can this affect

Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.

Blast radius if this is wrongEveryone

One testEveryone

What contains it

Split identities, short-lived scoped credentials, and admission verification at deploy. A single pipeline identity that can build, sign and deploy is contained by nothing.

What can go wrong

Failure modes, including of the mitigation

Secrets available to every job in the repository, including the ones that run proposed changes.
A self-hosted runner shared between a public-facing repository and an internal one, so state and credentials cross a trust boundary.
Pipeline definition changes merged without review because the change was "just CI".
Reusable steps pinned by tag, so the pipeline's behaviour changes without any commit in your repository.
Cache poisoning: a cache written by a low-trust job and restored by a high-trust one (Caching in CI).
Deploy credentials in the same job as the build, so anything that influences the build can also deploy.
Debug logging that prints an environment containing a secret into a log with wide read access (Secrets in Logs in Backend covers the same failure in services).

Misreads this invites

"CI is a developer tool, not production." It has production credentials and produces the artifacts production runs. It is production infrastructure with a different user interface.
"Our repository is private, so the pipeline is safe." Privacy limits who can propose changes. It does not limit what the dependencies you install can do inside the job.
"Secrets are encrypted, so they are safe in CI." They are decrypted for the job that uses them. The question is which jobs can ask for them.
"We reviewed the application code." The pipeline definition, the base images and the reusable steps are all executable inputs, and they are typically reviewed with less care than a one-line application change.

Operating it

Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.

How you know it worked

An inventory of what each pipeline identity can reach, reviewed on a schedule like any other access (Access Review).
A test workflow from a fork demonstrably cannot read any secret.
Cloud credentials used by CI are short-lived: the provider audit log shows federated sessions rather than static key usage.
Every reusable step in the pipeline is pinned to an immutable reference, verified by a check rather than by convention.

How you get back

Tightening pipeline permissions breaks jobs that were silently relying on the excess. Roll out per repository, watch for failures, and expect to discover undocumented dependencies.
If a pipeline credential is suspected compromised, rotation is the rollback — and it is only fast if you know which systems accept that credential (Rotation That Applications Survive).

What to automate, and what stays human

Automate: permission linting on pipeline definitions, pin checking, detection of secret access from untrusted triggers, and credential expiry.
Keep human: approving any grant of deployment privilege to a new pipeline, and any exception that gives an untrusted-trigger job access to a secret (Human Approval for High-Risk Agent Actions in Security Engineering).

What this costs

Ephemeral runners cost startup time on every job and lose warm caches, which is a real slowdown on large builds (Caching in CI).
Splitting build and deploy identities adds a handoff and makes the pipeline harder to follow.
Pinning by commit hash makes updates manual and produces a steady stream of update pull requests — which is the visible cost of a pipeline that cannot change underneath you.

Where this applies

This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.

TOOL-SPECIFICPermission models differ substantially: some CI systems default a job to broad repository write and require you to narrow it, others default to nothing and require explicit grants. Fork-trigger behaviour differs too — some withhold secrets from fork-triggered runs by default, some do not. Read your system's defaults rather than assuming the safe one.
CLOUD-SPECIFICWhether short-lived federated credentials are available depends on the provider and on your CI system supporting the federation. Where they are not, the fallback is a stored key with a narrow scope and a rotation schedule, which is meaningfully weaker and should be recorded as such.
ORG-SPECIFICHow much separation is worth it depends on team size and regulatory obligation. A three-person team with one pipeline identity is making a defensible trade; the same setup with fifty contributors and external pull requests is not.

Where the depth lives

This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.

Backendsecrets-in-logs

Cloudcicd-infrastructure human-vs-workload-identity roles-vs-static-keys

Securitycicd-security secrets-management short-lived-credentials least-privilege privilege-separation human-approval

Domains that do not exist yet

— Testing & Reliability Engineering — running proposed changes safely is a testing requirement and a trust boundary at the same time, and the two pull in opposite directions.