SecretsTOOL-SPECIFICCLOUD-SPECIFIC
Secrets in CI
CI holds credentials for everything and executes code that anyone can propose — which makes it the highest-value target in the delivery path and the one most often protected by conventions rather than controls.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
The production question
How does a build system get the credentials it needs without becoming the easiest route to production?
The problem
A pipeline needs to push artifacts, deploy, run integration tests and call third parties — so it accumulates credentials for all of them, while running arbitrary code from branches, forks and a dependency tree nobody has read.
What teams do first
Store the credentials as encrypted variables in the CI system. They are encrypted at rest and masked in logs, so they are protected.
How it breaks
Masking is log post-processing, not a control. It matches known values in output; it does not stop a build step from sending a credential somewhere, and it misses transformed forms — base64, JSON-escaped, split across lines (Secrets in Logs).
How it breaks in production
- Masking is log post-processing, not a control. It matches known values in output; it does not stop a build step from sending a credential somewhere, and it misses transformed forms — base64, JSON-escaped, split across lines (Secrets in Logs).
- Any step in the job can read every variable available to that job, including a dependency's install script and a tool nobody audited (The Delivery Chain as Attack Surface).
- Credentials accumulate and are rarely removed. A pipeline ends up holding production deploy credentials to run a lint job.
- Pull requests execute code. If workflows from forks or untrusted branches receive the same credentials as trusted ones, the credential is available to whoever opens a pull request (CI Security).
- Build caches and artifacts persist. A credential written to a cache directory or an uploaded artifact outlives the job that created it (Caching in CI).
- The credentials are long-lived, so a single disclosure from any job at any time in the past remains exploitable.
Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- CI is a privilege concentrator: it holds credentials for the registry, the cloud, the deployment target, the package repository and several third parties, and it runs code proposed by contributors and pulled from a dependency tree.
- The two properties that make it dangerous — broad credentials and untrusted execution — are both intrinsic to what CI is for. So the design has to reduce the value of what a job can obtain rather than try to make execution trustworthy.
- The structural fix is the same primitive as everywhere else: no stored credentials at all. The CI system presents a signed job token, a trust policy checks its claims — repository, branch, environment, workflow — and issues a short-lived, narrowly scoped credential (Workload Identity).
- What remains is segmentation: split pipelines so untrusted code and privileged credentials never run in the same job. A build from a pull request needs no deploy credential; a deploy needs no untrusted code.
- Secrets should be scoped to the step that needs them, not the job, and certainly not the pipeline. Most CI systems support per-step or per-environment scoping and most teams use a single global scope.
Why CI is the target
Stated as properties rather than as a warning, the reason is obvious: CI has the credentials of everything and runs code proposed by anyone. No other component in the delivery path combines those two.
| Property of CI | Why it exists | What it means for an attacker |
|---|---|---|
| Holds deploy credentials | It has to deploy | Compromising CI is compromising production (Production Access) |
| Holds registry write credentials | It has to publish artifacts | A malicious artifact can be published under a trusted name (Artifact Registries) |
| Runs code from branches and forks | That is what continuous integration is | Code execution is available to anyone who can open a pull request |
| Executes a full dependency tree | Builds install dependencies | Any transitive dependency's install script runs with job credentials |
| Runs third-party plugins and actions | Reuse is the point | An unpinned action updated upstream runs new code in a privileged job |
| Retains logs, caches and artifacts | Debuggability and speed | Anything written there outlives the job and is often broadly readable |
| Often uses one shared identity | It was simpler at setup | Audit cannot attribute actions; every pipeline has every permission |
Splitting untrusted execution from privilege
TOOL-SPECIFICHow the split is expressed differs: separate workflows with different triggers, environment-scoped secrets with required reviewers, or separate pipelines in separate projects. What matters is the invariant — no job that executes untrusted code holds a credential that reaches production — and whether your CI system can actually enforce it rather than merely document it (The Builder Is Inside the Trust Boundary).
The structural control is segmentation. Untrusted code runs with nothing worth stealing; privileged steps run only trusted code, on a reviewed artifact, behind a gate.
Note that the artifact crosses the boundary and the credentials do not — and that the boundary needs its own integrity story, because otherwise the untrusted side simply hands the trusted side a malicious artifact.
UserLLMAgentToolDataDecisionHumanGuardrail
How CI credentials actually leak
None of these require a sophisticated attacker. Each is a default setting, a convenience, or an unexamined piece of configuration, and every one has a control that is cheaper than the incident.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Workflow trigger runs fork code with repository secrets | Nothing visible; credentials are readable by anyone who opens a pull request | A trigger chosen for convenience whose secret semantics differ from the safe one | Separate untrusted build from privileged deploy; verify by attempting it from a fork (CI Security) |
| Debug output prints a configuration object | A credential in a build log, possibly in a form masking did not match | Masking matches known literal values, not transformed ones | Redact at the serialisation boundary in tooling; treat any occurrence as a rotation event |
| Credential file lands in a cached directory | The credential is restored into later jobs, including other branches | Broad cache paths that include the whole workspace (Caching in CI) | Explicit cache path allowlists; inspect what the cache actually contains |
| Unpinned third-party action updated upstream | New code runs inside a privileged job with no change on your side | A mutable tag reference rather than a digest | Pin by digest and update deliberately (Dependency Pinning) |
| Dependency install script runs in the build job | Job credentials readable by a transitive dependency | Builds execute dependency lifecycle scripts by default in several ecosystems | Disable install scripts where possible; give build jobs no credential worth taking |
| Self-hosted runner reused between jobs | Credentials or checked-out code from one job visible to the next | No isolation between job executions on a persistent host (Process Isolation: One Kernel, Many PID 1s) | Ephemeral runners, or full workspace and credential cleanup enforced between jobs |
| Federation trust policy accepts any repository | Any workflow in the CI system can assume the deploy role | A wildcard subject condition added to make setup work (Workload Identity) | Pin repository and branch or environment; test the exchange from an unauthorised context |
How to do it properly
Most important first.
- Replace stored CI credentials with federated short-lived credentials wherever the target supports it. This removes the object rather than protecting it (Roles vs Static Keys).
- Write the trust policy to require the repository and the branch or deployment environment. A policy that accepts any workflow from the organisation is a standing grant (Protected Branches).
- Never expose deployment credentials to a job triggered by a fork or by an unreviewed branch. Split into an untrusted build job and a privileged deploy job with a review gate between them.
- Scope each secret to the smallest unit the CI system supports — the step, or an environment gated on approval.
- Treat masking as a safety net, not a control: assume anything a job can read can leave the job.
- Exclude credentials from caches and artifacts explicitly, and check what your cache paths actually contain.
- Pin actions, plugins and container images by digest so a moved tag cannot introduce new code into a privileged job (Dependency Pinning, Tags Versus Digests).
- Audit what each pipeline can reach on a schedule, and remove credentials that no current step uses (Access Review).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Blast radius if this is wrongEveryone
One testEveryone
What contains it
Contained by credential lifetime and scope, and by nothing else. A long-lived CI credential with deploy rights is unbounded in both time and reach: whoever holds it can deploy, which is a path to every user. Short-lived, narrowly scoped credentials are the only mechanism that meaningfully bounds this.
What can go wrong
Failure modes, including of the mitigation
- A workflow trigger that runs fork-proposed code with access to the repository's secrets, which is the single highest-impact CI misconfiguration.
- A credential printed by a debug step, or by a tool that echoes its configuration, in a form masking does not match.
- A build cache containing a credential file, restored into later jobs — including jobs from other branches.
- An unpinned third-party action updated by its author, executing new code inside a privileged job (Securing the Pipeline Itself).
- One service account used by every pipeline, so audit cannot attribute an action and least privilege is impossible.
- A self-hosted runner shared between jobs without isolation, where one job leaves credentials on disk for the next (Process Isolation: One Kernel, Many PID 1s).
- Test fixtures containing real credentials for a third-party sandbox that shares an account with production.
Misreads this invites
- "CI masks secrets, so they cannot leak." Masking matches known strings in output. It does not prevent a job from sending a value elsewhere, and it does not match transformed forms.
- "Only maintainers can run the pipeline." Depending on the trigger, pull requests run pipeline code. The question is which triggers receive which credentials, and the defaults are not always the safe answer.
- "The secrets are encrypted at rest." So is everything. The exposure is what the job can read at runtime, which is the whole point of storing them there.
- "We rotate CI credentials annually." A long-lived CI credential is exploitable from the moment it leaks until it is rotated. This is the strongest case in the domain for short-lived credentials instead (Short-Lived Credentials).
- "Self-hosted runners are more secure because they are ours." They are ours and often shared, persistent and less isolated between jobs than a fresh hosted runner, which makes cross-job residue a real concern.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
How you know it worked
- The list of stored long-lived secrets per repository is short and shrinking, ideally empty.
- A deliberate attempt to obtain deploy credentials from a fork-triggered workflow fails.
- Cloud audit logs attribute deploy actions to a per-pipeline identity, not to one shared account (Audit Logs for Privileged Actions).
- Credentials observed in CI have short expiry, verified by inspection.
- Cache and artifact contents have been examined for credential files at least once, deliberately.
How you get back
- A credential exposed in CI cannot be un-exposed. Logs are retained, artifacts are downloadable, and caches are shared. Rotate at the issuer and treat the value as compromised (Rotation That Applications Survive).
- Deleting a job log is not remediation. It may have been read, forwarded to a log vendor, or captured in a status integration before deletion.
- The recoverable part is the configuration — the trigger, the scope, the pinning — and it should be fixed in the same change as the rotation, not instead of it.
What to automate, and what stays human
- Automate federation setup so a new repository gets short-lived credentials by default rather than a stored key (Service Templates).
- Automate a check that fails any workflow granting privileged secrets to an untrusted trigger.
- Automate scanning of build logs and artifacts for credential patterns, accepting that it is a detector rather than a preventer (Scanning, and Why a Finding Is Not a Risk).
- Do not automate approval of a deploy job on an untrusted branch. The gate between untrusted code and privileged credentials is exactly where a human belongs (Guardrails, Not Gates).
What this costs
- Split pipelines are more complex and slower, and the handoff between an untrusted build and a privileged deploy needs its own integrity story — usually artifact signing (Signing and Verifying Artifacts).
- Federation ties CI to a provider's identity model and takes real setup effort per target.
- Fine-grained per-step scoping is more configuration to maintain, and drift in that configuration is itself a risk.
- Refusing credentials to fork pull requests means some integration tests cannot run on external contributions, which is a genuine cost to open collaboration.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- TOOL-SPECIFICCI systems differ in ways that decide whether a configuration is safe: which triggers expose secrets to fork-proposed code, whether secrets can be scoped per step or per environment, whether approval gates exist, and whether job tokens can be federated to a cloud identity. Read your system's specific trigger-and-secret semantics — these differ between products and change between versions, and a pattern copied from another CI system is exactly how a fork gains credentials.
- CLOUD-SPECIFICFederation from CI exists on all major providers and the trust policy syntax and claim names differ. The claim that identifies the branch or environment is provider- and CI-specific, and a policy translated without re-reading claim semantics commonly ends up accepting any workflow from any repository in the CI system (Anatomy of a Policy).
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
Backendsecrets-in-logs
OS & Networkingprocess-isolation
Domains that do not exist yet
- — Testing & Reliability Engineering — running the pipeline's own security invariants as tests, so a workflow that grants privileged secrets to an untrusted trigger fails a build.