SecretsGENERALTOOL-SPECIFIC
What Counts as a Secret, and Where It Must Not Be
Credentials, API keys, private keys, certificates and tokens must not live in source control or in images — because both are copied, cached and retained far beyond the systems you control.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
The production question
What actually counts as a secret, and which places must it never reach?
The problem
A running service needs credentials, and the two most convenient places to put them — next to the code and inside the image — are also the two that copy themselves everywhere and keep history forever.
What teams do first
Keep credentials in a configuration file in the repository, or set them as build arguments so the image is self-contained. Access to the repository is controlled, so the secrets are controlled.
How it breaks
Version control keeps history. Removing a value in a later commit leaves it in every clone, every fork, every mirror, every CI cache and every backup of the repository (Git Workflows).
How it breaks in production
- Version control keeps history. Removing a value in a later commit leaves it in every clone, every fork, every mirror, every CI cache and every backup of the repository (Git Workflows).
- Repository access is far broader than production access: every engineer, every contractor, every CI job, every code-scanning integration and every developer laptop (Least Privilege in Production).
- Image layers are immutable and additive. A value present in an early layer persists in the image even if a later layer deletes the file, and anyone who can pull the image can read it (Layers and the Build Cache).
- Build arguments are recorded in image metadata by default on common builders, so "it was only a build argument" is not a control.
- Registries retain and replicate. An image pushed once may be cached on dozens of nodes and mirrored into other registries (Artifact Registries).
- Deleting is not revoking. Removing a credential from a repository or an image does nothing to the credential itself, which remains valid until it is rotated at the issuer (Rotation That Applications Survive).
Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A secret is any value whose disclosure lets someone act as you: passwords, API keys, private keys, session and bearer tokens, signing keys, client secrets, database credentials, webhook signing secrets, and the private half of a certificate (Certificate Trust Chains).
- The useful test is not "is this sensitive" but "if someone else had this value, could they do something as us?" That test correctly includes things people forget — a webhook signing secret, a database read replica password, a third-party sandbox key that shares an account with production.
- Some values are adjacent and not secret: a public key, a certificate's public half, a resource identifier, a hostname. Treating those as secret costs operability for no benefit, and the confusion usually runs the other way — encoding is mistaken for protection (Hashing vs Encryption vs Encoding).
- The delivery-side rule follows from durability. Source control and images are replicated, cached and retained artifacts. A secret placed in either has been distributed to systems and people you do not enumerate, at a time you cannot reconstruct.
- That leaves one shape: the secret lives in a store, and the running workload fetches it under its own identity at startup. Nothing in the delivery path ever holds the value (Workload Identity).
The test for "is this a secret"
GENERALThe classification holds anywhere. The handling of the key-store row differs by provider: some key services never release key material at all and perform cryptographic operations on your behalf, others issue data keys to the application — which changes what the application is holding and therefore what can leak (Symmetric Encryption).
Sensitivity is the wrong axis, because it produces both over- and under-classification. The right question is capability: if someone else held this value, what could they do as us?
| Value | Secret? | Why | Commonly mishandled as |
|---|---|---|---|
| Database password | Yes | Direct data access | A configuration file in the repository |
| API key for a third party | Yes | Acts as your account, often billed to you | A build argument |
| TLS private key | Yes | Impersonation of your service (Certificates as an Operational Object) | A file copied into an image |
| TLS certificate (public half) | No | Published to every client during a handshake | Treated as secret, complicating renewal |
| Webhook signing secret | Yes | Lets someone forge inbound events you trust | Forgotten entirely — rarely in the inventory |
| Signing key for artifacts or tokens | Yes | Forged releases or forged sessions (Signing and Verifying Artifacts) | Kept on a build machine rather than in a key store |
| Session or bearer token | Yes, and short-lived by design | Acts as an authenticated user | Logged in request traces (Secrets in Logs) |
| Cloud account or project identifier | No | An identifier, not a capability | Treated as secret, which hinders debugging |
| Internal hostname | No | Not a credential; do not rely on it being unknown | Treated as a control in itself |
| Encryption key for data at rest | Yes, and key-store managed | Decrypts everything it protects (Key Management and Encryption at Rest) | Stored beside the data it protects |
Why source control and images are the two forbidden places
Both are designed to be copied, cached and kept. That is what makes them good at their job and disqualifying for secrets. The property that matters is that you cannot enumerate the copies afterwards.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Credential committed, removed in a later commit | Nothing visible; the value stays valid | History persists in clones, forks, mirrors and CI caches | Rotate at the issuer immediately; treat history rewriting as optional cleanup, never as the fix |
| Credential passed as a build argument | Anyone who can pull the image can read it from metadata | Build arguments are recorded in image metadata by common builders | Use a build-time secret mount that is not persisted, and verify by inspecting the built image |
| Secret file copied in, deleted in a later layer | The file is absent at runtime and present in the image | Layers are additive; deletion adds a layer, it does not remove data (Layers and the Build Cache) | Never introduce the value into any layer; multi-stage builds do not fix a secret in an early stage that is copied forward |
| Image pushed to a shared or public registry | Distribution beyond your control, possibly permanent | Registries replicate, mirror and cache on every node that pulls | Rotate; assume disclosure; audit registry access and retention (Artifact Retention) |
| Configuration object logged at startup | Credentials in every log sink and forwarding vendor | Redaction applied at call sites rather than at the serialisation boundary | A redacting serialiser on the type itself, tested by attempting to log it |
| Secret pasted into a chat channel during an incident | Credential in a searchable retained system with different access controls | Urgency plus no sanctioned fast path | A break-glass path that grants access without transferring values (Break-Glass Access) |
The only shape that works
CLOUD-SPECIFICEvery major provider offers this shape and the pieces are not interchangeable. They differ in how the workload identity is attested, in whether the secret is delivered by an SDK call, a mounted file or an injected environment variable, in whether versions are addressable and rollback-able, and in how access is audited. Treat a design that works on one provider as needing re-verification on another rather than as portable (Secrets in Infrastructure).
Everything in this module is a variation on the diagram below: the value lives in one managed place, the workload proves who it is, and the credential arrives in memory at startup. No human handles it, and no artifact in the delivery path contains it.
Note what is absent. The repository holds a reference. The image holds nothing. CI never sees the production value at all (Secrets in CI).
UserLLMAgentToolDataDecisionHumanGuardrail
How to do it properly
Most important first.
- Fetch secrets at runtime under a workload identity. Configuration holds a reference — a path or a name — and never a value (Artifact Plus Configuration).
- Scan for committed secrets both at commit time and across full history, and treat any hit as a rotation event rather than a deletion task.
- Keep secrets out of build arguments, image layers, base images and cached build context; use build-time secret mounts that are not persisted where the builder supports them (Multi-Stage Builds).
- Redact at the logging boundary rather than at call sites: a redacting serialiser applied to the whole configuration object, not a promise to be careful (Secrets in Logs).
- Prefer credentials that cannot be reused: short-lived tokens over static keys, and scoped tokens over account-wide ones (Short-Lived Credentials).
- Keep an inventory of what secrets exist, who issued them, what they grant and when they were last rotated. A secret nobody knows exists is never rotated and never revoked.
- Make the safe path the easy path — a template that wires up secret fetching correctly, so nobody has to invent it under time pressure (Service Templates).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Blast radius if this is wrongEveryone
One testEveryone
What contains it
Nothing contains a disclosed credential except its own scope and lifetime. That is the entire argument for least privilege and short lifetimes: they are the only mechanisms that bound this, because deletion cannot (Least Privilege).
What can go wrong
Failure modes, including of the mitigation
- A secret removed in a commit but not rotated, so it stays valid in history indefinitely — the most common form of this mistake.
- Secrets in an image passed to a vendor, a customer or a public registry, which cannot be recalled.
- Secrets in environment variables that reach crash dumps, error reporting payloads, process listings and child processes (The Anatomy of a Process).
- Whole configuration objects serialised into logs at startup, putting every credential into every log sink and forwarding vendor.
- A
.envfile mounted for convenience and then baked into an image by a broad copy instruction in the build. - A scanner producing so many false positives that real findings are ignored — scanner output is not the same as risk (Scanning, and Why a Finding Is Not a Risk).
- Secrets shared over chat during an incident, which puts them into a searchable retained system with a different access model.
Misreads this invites
- "The repository is private, so it is fine." Private means the current access list. It does not cover forks, clones, laptops, CI caches, backups, or whoever joins the organisation next year.
- "I deleted the commit." History persists in clones and mirrors you do not control. Deletion is not revocation; only rotation at the issuer is.
- "It is base64 encoded." Encoding is not encryption. Base64 is reversible by anyone, and this misunderstanding is common enough that some platforms' base64-encoded secret objects are routinely mistaken for encrypted ones (ConfigMaps and Secrets).
- "It is only a staging credential." Staging credentials often reach the same third-party account as production, and staging is the environment with the weakest controls (Production Data in Lower Environments).
- "We store secrets in environment variables, so they are safe." Environment variables keep secrets out of the image, which is progress. They are still visible to child processes, crash dumps and anything that serialises the environment.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
How you know it worked
- A full-history scan of every repository returns no live credentials, and every historical hit has a recorded rotation.
- Inspecting a production image's layers and metadata reveals no credential values.
- A deliberate test — logging the configuration object — shows redaction working rather than a promise that it would.
- The secret inventory exists, names an owner per secret, and includes a last-rotated date (Rotation That Applications Survive).
How you get back
- There is no rollback for disclosure. Once a value has been in a repository, an image or a log, the only response is to rotate it at the issuer and treat the old value as compromised (The Secret Lifecycle).
- Rewriting history does not undo distribution: clones, forks and caches already exist. Rewrite if you like, but rotate first and count the rotation as the fix.
- What can be rolled back is the change that introduced the path — the build step, the mount, the logging call — and that should be done alongside the rotation, not instead of it.
What to automate, and what stays human
- Automate pre-commit and full-history scanning, image-layer scanning, and redaction at the logging boundary. All three are mechanical and all three catch the common cases.
- Automate the inventory: derive it from the store rather than maintaining a document, which goes stale immediately.
- Automate rotation triggering on a scanner hit, so the response is rotation by default rather than a discussion (Rotation That Applications Survive).
- Do not automate the judgement of whether a leaked credential was used. That requires reading access logs with context, and it is the part that decides whether this is a cleanup or an incident (What Happens Between the Page and the Postmortem).
What this costs
- Runtime fetching adds a startup dependency on the secret store, which becomes a fleet-wide availability concern (When Secrets Fail).
- Short-lived credentials require the application to refresh, which is real work and is the change most likely to be skipped.
- Aggressive scanning generates false positives, and the cost of triaging them lands on the same people who would fix the real ones.
- Strict handling makes local development harder, and that friction is precisely what drives people to commit a
.envfile.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe rule — never in source control, never in an image — holds on every stack. What differs is the substitute: mounted files and injected environment on container platforms, instance metadata and agent-fetched values on virtual machines, platform-managed environment on serverless.
- TOOL-SPECIFICWhether a build argument persists in the final image depends on the builder and the instruction used. Some builders record build arguments in image metadata by default; dedicated build-time secret mounts exist specifically to avoid that and are not the default path. Verify by inspecting a built image's layers and metadata rather than trusting the intent of the build file.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
Securitysecrets-managementsecret-lifecycleinfrastructure-secretsleast-privilegehashing-encryption-encoding
OS & Networkingprocess-anatomy
Domains that do not exist yet
- — Testing & Reliability Engineering — asserting the absence of credentials in artifacts as a build-failing check rather than a review item.