ConfigMaps and Secrets
Two objects that inject configuration into pods, one of which is named after a security property it does not, on its own, provide.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
How does configuration reach a container, and what does calling something a Secret actually buy you?
An artifact plus configuration equals a running service. The configuration differs per environment, changes on a different cadence from the code, and some of it is credentials that must not be in the image or the repository.
Put non-sensitive values in a ConfigMap, put passwords in a Secret, mount both, and consider the secret-handling problem solved because the object type says Secret.
A Secret is base64-encoded, not encrypted. Base64 is an encoding, and anyone who can read the object can read the value with one command. The name promises a property the object does not deliver by itself.
- A Secret is base64-encoded, not encrypted. Base64 is an encoding, and anyone who can read the object can read the value with one command. The name promises a property the object does not deliver by itself.
- Whether Secret values are encrypted where they are stored depends on the cluster: encryption at rest for Secrets is a control plane configuration decision, and it is not universally enabled (Encryption at Rest vs in Transit).
- Access is governed by RBAC, and RBAC is easy to get wrong in the permissive direction. A role granting
geton secrets in a namespace grants every secret in it, and broad wildcards in cluster roles are common (Least Privilege in Production). - Secrets injected as environment variables leak readily: crash dumps, error reporters that serialise the environment, debug endpoints, and any child process that inherits it (Secrets in Logs).
- Rotation is not part of the model. Environment variables are read once at process start, so changing a Secret does nothing until pods restart — and nothing tells you they have not (Rotation That Applications Survive).
- Manifests containing base64 values get committed. It looks encoded, so it does not look like a password, and it ends up in git history forever.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Both objects are namespaced key–value stores. ConfigMap is plain data; Secret is the same shape with base64-encoded values, a type field, and slightly different handling — it is not written to a node's disk in the same way, and tooling avoids printing it.
- Both can be consumed two ways: as environment variables, or mounted as files in a volume. The choice matters more than which object you used.
- A mounted ConfigMap or Secret is updated in place by the kubelet after the object changes, with a propagation delay. An environment variable is fixed for the life of the process. That difference decides whether rotation requires a restart.
- Neither object is versioned in a way a rollout can reason about. Changing one changes what pods read without any rollout, any gate and any record of the change (A Config Change Is a Production Change).
- Marking either object
immutable: trueblocks updates and lets the kubelet stop watching it — a performance and safety property that turns config change into create-new-and-repoint, which is closer to how you want it anyway. - The real security boundary is who can read the object, which is RBAC, plus whether the store beneath it is encrypted, plus who can create a pod in the namespace — because a pod can mount any Secret in its namespace (Role-Based Access Control).
What "Secret" does and does not mean
This table exists because the object's name does most of the misleading. Read the middle column as what you get for free and the right column as what you have to do yourself.
| Property | A Secret object gives you | What actually provides it |
|---|---|---|
| Not in the image | Yes — injected at runtime | Using the object at all |
| Not in the repository | No | External secret manager or an encrypted-at-rest manifest form |
| Encrypted where stored | Only if the control plane is configured for it | Cluster encryption at rest, a control plane setting |
| Restricted readers | Only what RBAC says | Namespace-scoped, name-scoped roles (Role-Based Access Control) |
| Hidden from tooling output | Partly — tools avoid printing it | Nothing prevents a decode by anyone with read access |
| Audited reads | Only if audit logging is on | Control plane audit configuration (Audit Logs for Privileged Actions) |
| Rotation | No | A rotation procedure plus a restart or reload (Rotation That Applications Survive) |
| Not in crash dumps | Only if mounted as a file | Choosing volume mount over environment variable |
Environment variable or file: the choice that matters
People agonise over ConfigMap versus Secret and then pick the consumption method by habit. The consumption method is the one that decides whether rotation needs a restart and whether the value shows up in a crash report.
1apiVersion: v12kind: Secret3metadata:4 name: checkout-db5type: Opaque6stringData: # written as plain text, stored base64-encoded7 password: "not-a-real-password"8---9apiVersion: v110kind: Pod11metadata:12 name: checkout13spec:14 containers:15 - name: app16 image: registry.example.com/checkout@sha256:9f3e1c...17 env:18 - name: DB_PASSWORD # read once at process start19 valueFrom: # visible in the environment, and in crash dumps20 secretKeyRef:21 name: checkout-db22 key: password23 volumeMounts:24 - name: db25 mountPath: /etc/creds # updated in place after the Secret changes26 readOnly: true # the app must re-read the file to benefit27 volumes:28 - name: db29 secret:30 secretName: checkout-dbBoth forms are shown for contrast; a real pod picks one. stringData is a write-only convenience — the stored object holds base64 under data, which is an encoding and not protection. The environment form cannot be rotated without a restart; the file form can, if the application re-reads it.
A rotation that half-worked
This is the most common secret incident in a cluster, and it is not a security failure — it is a propagation failure. The Secret changed and the pods did not.
The shape to notice is that nothing was wrong until the old credential was revoked, which is a step teams often perform hours or days later, long after the rotation was marked done.
- T+0changeNew credential created in the database; Secret object updated
- T+1msignalSecret shows the new value; every running pod still holds the old one in its environment
- T+5mactionRotation marked complete — the object is correct, so it looks done
- T+2hchangeAn unrelated deploy restarts some pods; those pick up the new credential
- T+1dactionOld credential revoked as the final rotation step
- T+1d 1msignalEvery pod that has not restarted starts failing to authenticate
- T+1d 6mactionRolling restart triggered; pods pick up the new value as they come back
- T+1d 12mrecoveryErrors clear once the last old pod is replaced
The fix is not "remember to restart". It is to make the rotation procedure include the restart, and to verify from the database side that no session is authenticating with the old credential before revoking it (Rotation That Applications Survive).
How to do it properly
Most important first.
- Keep credentials in a purpose-built secret manager and give workloads short-lived, identity-based access to it rather than a long-lived string in the cluster (Workload Identity).
- If secrets do live as Secret objects, treat the object as a delivery mechanism and not as protection: enable encryption at rest for them, scope RBAC per namespace and per name, and audit reads (Audit Logs for Privileged Actions).
- Prefer file mounts over environment variables for anything sensitive. Files do not appear in crash dumps or process listings, and they can be updated without a restart.
- Never commit a Secret manifest with real values. Use an external secret reference, a sealed or encrypted form, or generate it in the pipeline from the secret manager (Secrets in CI).
- Validate configuration at startup and fail loudly on a missing or malformed key. Discovering a missing key at first use, hours later, is how a config change becomes a 3am page (Validate at Startup, Fail Clearly).
- Give configuration a version that appears in the running process and its logs, so "what config is this pod running?" is a lookup rather than an investigation (The Release Manifest).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A config or secret change applies to every pod that reads it, with no rollout gate; containment is startup validation that refuses to serve on a bad value, plus restarting one replica first.
What can go wrong
- Secret changed, pods not restarted, so half the fleet runs the old credential after a rotation — usually discovered when the old one is revoked (Rotation That Applications Survive).
- A key renamed in the ConfigMap and not in the code, so pods start and fail at the moment the value is first read rather than at startup.
- Environment-variable secrets in a stack trace sent to an error reporting service, which is now a second system holding your credentials.
- RBAC granting read on all secrets in a namespace to a workload that needed one, so a compromise of that workload is a compromise of everything in the namespace (Multi-Tenant Isolation).
- A config change made directly against the live object during an incident, never written back to the repository, and silently reverted at the next apply (Manual Production Changes).
- A ConfigMap consumed as a mounted file being updated mid-request, so a process that re-reads the file sees a partially consistent view of a multi-key change.
- "Secrets are encrypted." They are base64-encoded. Encryption at rest is a separate control plane setting, and it protects the storage layer, not readers with RBAC access.
- "Base64 in a manifest is safe to commit." It is a reversible encoding, not a protection. Anyone with the file has the value.
- "Using a Secret object satisfies our secret handling requirements." The requirement is about who can read it, how it is rotated and whether reads are audited. The object type is one small part of that (What Counts as a Secret, and Where It Must Not Be).
- "Changing a ConfigMap deploys the change." It changes the object. Whether a running process ever sees it depends on how it was consumed and whether it re-reads.
- "Config changes are lower risk than code changes." Config differs per environment, so it is the least-tested input in the system, and it applies everywhere at once (Configuration Drift).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- The running process reports which config version it loaded, and it matches the release record.
- A deliberately malformed value causes a fast, loud startup failure rather than a delayed runtime one.
- Rotating a credential shows every replica using the new one within a bounded time, verified from the dependency's side rather than assumed.
- An RBAC review shows no role granting read on all secrets to a workload identity.
- ConfigMaps and Secrets revert by reapplying the previous object, but the revert applies instantly and cluster-wide, with no rollout to stop and no canary (Blast Radius: If This Is Wrong, How Much Does It Affect?).
- A rollback only takes effect where the value is actually re-read. Environment variables need pods restarted, so a config rollback is a restart in disguise.
- A leaked secret does not roll back. It rotates, and everything holding the old value has to be updated — which is why the rotation path needs to exist before the leak (When Secrets Fail).
- Automate secret delivery from a secret manager, so cluster objects are generated rather than hand-maintained and rotation has one source of truth (Secret Managers and What They Actually Give You).
- Automate the restart or reload that a rotation requires, so "the credential changed" and "every replica is using it" are one operation.
- Automate scanning for committed secrets in the repository and in manifests, including base64 blobs that look inert (CI Security).
- Keep the decision about who can read which secret human and reviewed. RBAC generated by convenience drifts toward permissive.
- Cluster-native Secrets are convenient and put credential material in a system whose access control is coarse-grained by default. An external secret manager is a stronger boundary and another dependency in the startup path.
- File mounts allow rotation without restart and require the application to re-read the file, which many applications do not do.
- Immutable config objects remove a class of surprise and force a create-and-repoint workflow, which is more steps and much better provenance.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- KUBERNETES-SPECIFICConfigMap and Secret are Kubernetes objects with Kubernetes access control. Outside a cluster the same job is done by a cloud secret manager read at startup with an instance role, by encrypted files shipped with the artifact, or by environment variables set by a PaaS — where the analogous risk is the platform's own dashboard exposing the value and its own audit log recording, or not recording, reads.
- CLOUD-SPECIFICWhether Secrets are encrypted at rest, and with which key, depends on how the control plane was provisioned. Managed offerings differ in defaults and in whether you can bring your own key (Key Management and Encryption at Rest on the Cloud side).
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.