Secret Managers and What They Actually Give You
A secret manager is a store with access control, versioning, audit and encryption at rest — and the product details differ enough between providers that a working design is not portable without re-verification.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What does putting secrets in a dedicated manager actually buy, and what still has to be designed?
Once secrets are out of source control and out of images, they need somewhere to live that can answer who may read them, what happened to them, and what the previous value was.
Put them in the platform's built-in secret object. It keeps them out of the repository, the platform mounts them for us, and the job is done.
A platform secret object is often storage, not management. Depending on configuration, the value may be stored with only base64 encoding at rest, readable by anyone with sufficient access to the platform's data store or its API (ConfigMaps and Secrets).
- A platform secret object is often storage, not management. Depending on configuration, the value may be stored with only base64 encoding at rest, readable by anyone with sufficient access to the platform's data store or its API (ConfigMaps and Secrets).
- There is frequently no version history, so rollback of a bad secret change is impossible and "what was the previous value" is unanswerable.
- Read access is rarely audited at the value level, so you cannot answer "who read this credential, and when" after a suspected disclosure (Audit Logs for Privileged Actions).
- Access control is usually coarse — namespace or project level — so a service that needs one credential can commonly read every credential in its scope (Least Privilege).
- Values still have to reach the process. If that path is an environment variable injected at deploy, the secret is in the deployment specification and in whatever stores that specification.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A secret manager provides five things: encrypted storage under a key you can control, fine-grained access control per secret, audit logging of both writes and reads, versioning so a value can be rolled back and rotation can overlap, and programmatic issuance so no human handles the value (Rotation That Applications Survive).
- Storage is the least interesting of the five. Access control, audit and versioning are what make it a management system rather than a filing cabinet.
- Delivery to the process is a separate design decision from storage, and it is where most of the operational consequences live: an SDK call at startup, a file mounted by an agent, or an environment variable injected at deploy each have different rotation behaviour, different failure modes and different exposure (Build-Time and Runtime Configuration).
- The manager needs a root of trust. Something must authenticate the caller, and if that something is another static credential you have moved the problem rather than solved it — which is exactly what workload identity exists to fix (Workload Identity).
- Providers are not equivalents. They differ in whether secrets are versioned and how versions are addressed, in whether managed rotation exists for a given credential type, in caching behaviour, in per-secret cost and in request rate limits.
Storage is the least of the five
Teams adopt a secret manager for encrypted storage and then benefit mostly from the other four properties. Read the table as a checklist against whatever you are using now — the gaps are usually audit and versioning.
| Property | What it enables | What its absence costs |
|---|---|---|
| Encrypted at rest, with a key you control | Revoking the key revokes access to everything it protects | Anyone reading the underlying store reads the values (Encryption at Rest vs in Transit) |
| Per-secret access control | A compromised low-value workload cannot read high-value credentials | Blast radius of any compromise is the whole namespace |
| Audit of reads and writes | Answering "who read this, when" after a suspected disclosure | A disclosure investigation with no evidence either way |
| Versioning | Rollback of a bad write; overlap during rotation | Rotation must be a hard swap, and a bad write is unrecoverable |
| Programmatic issuance | No human ever holds the value | Copy-paste through chat, tickets and consoles |
| Managed rotation (where offered) | Rotation actually happens rather than being deferred | Credentials that are years old because rotation is manual |
How the value reaches the process
This choice is usually made by default and has more operational consequence than the choice of store. The deciding question is what happens when the value changes underneath a running process.
How should a running workload obtain a secret value?
when The default for most services. Gives rotation tolerance without a restart, and keeps the value off disk entirely.
cost A startup dependency on the store, application code that must handle refresh and failure, and request cost at scale.
when The application cannot easily call the store, or several processes share the value.
cost The value exists on a filesystem, so file permissions and volume type matter; the application must watch the file or it will never see a rotation (Volumes: Storage With a Lifecycle).
when Simple, widely supported, and acceptable where rotation via redeploy is fine.
cost Rotation requires a deployment; the value appears in deployment specifications, process environments, crash dumps and child processes.
when Available for the target system — a database or cloud API that supports dynamic credentials.
cost The strongest option and the most machinery: issuance latency on the connection path and a hard dependency on the issuer (Short-Lived Credentials).
The store is on the startup path of everything
The operational consequence teams underestimate is that a secret manager sits between every process and its ability to start. That is a new failure domain, and it is worst at the exact moment you are recovering — when the whole fleet restarts at once.
- 1Obtain workload identity
The platform attests what this workload is, without a stored credential.
fails by A static credential used to authenticate to the store, which recreates the original problem (Roles vs Static Keys).
evidence No credential exists anywhere in the artifact or its deployment specification.
- 2Authenticate to the store
Exchange the attested identity for store access.
fails by Clock skew or an expired trust configuration, producing an authentication failure that looks like a permission problem (Clock Synchronisation).
evidence A denied request is distinguishable in the logs from an unauthenticated one.
- 3Authorise
Policy decides which secrets this identity may read.
fails by A wildcard scope granted during setup and never narrowed (Anatomy of a Policy).
evidence Reading an out-of-scope secret is denied when attempted.
- 4Fetch
Retrieve the value, and note its version.
fails by Rate limits during a mass restart, throttling the whole fleet at the worst moment.
evidence A full-fleet restart completes without throttling, measured rather than assumed.
- 5Cache with a bounded lifetime
Hold the value in memory, refreshing periodically.
fails by Caching forever, which defeats rotation; or not caching, which makes the store a per-request dependency (Rotation That Applications Survive).
evidence A rotated value is picked up within the TTL, observed in a lower environment.
- 6Fail loudly if unavailable
Refuse to become ready rather than starting without credentials.
fails by Starting anyway and failing on the first request that needs the credential (Validate at Startup, Fail Clearly).
evidence An unreachable store produces a startup failure naming the secret, not a 500 an hour later.
Steps four and five are the ones that turn a recovery into an outage. A fleet that restarts together fetches together, and a store with a request rate limit will throttle exactly then (Headroom).
How to do it properly
Most important first.
- Choose the delivery mechanism deliberately, and choose it for the rotation story: an SDK fetch can re-read on demand, a mounted file can be updated in place, an injected environment variable cannot change without a redeploy.
- Scope access per secret and per workload, not per namespace or project. The default granularity is almost always too broad (Least Privilege in Production).
- Turn on read auditing and make sure it is retained long enough to answer questions raised weeks later (The Audit Trail).
- Use versioning: pin to a version where determinism matters, and use a moving alias where rotation must propagate without a deploy — knowing which you chose.
- Cache fetched values in memory with a bounded lifetime, so the store is not a per-request dependency and rotation still propagates (When Secrets Fail).
- Encrypt with a key you control where the provider allows it, so revoking the key revokes access to everything it protects (Key Management and Encryption at Rest).
- Keep the inventory generated from the store, including owner, consumers and last rotation.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Contained by policy scope and by credential lifetime, and by nothing else. If the store is on the startup path, its own outage is also everyone with nothing containing it — which is why local caching of the last known good value matters (When Secrets Fail).
What can go wrong
- The secret store as a hard startup dependency: its outage prevents every service from starting, including the ones needed to recover (When Secrets Fail).
- Request rate limits hit during a mass restart, so a recovery turns into a throttled crash loop exactly when the fleet is trying to come back (How Autoscaling Fails).
- Over-broad access policy, so a compromised low-value service can read high-value credentials.
- Secrets fetched at startup and cached forever, which silently defeats rotation (Rotation That Applications Survive).
- A value written by a human through a console with no record of what it replaced.
- Audit logs enabled but retained for less time than it takes to discover a disclosure.
- A secret manager per team, so there is no single inventory and no consistent policy.
- "We use the platform's secret object, so secrets are encrypted." Depending on the platform and its configuration, that object may be base64-encoded storage rather than encrypted storage, with encryption at rest as an opt-in feature (ConfigMaps and Secrets).
- "The secret manager solves secrets." It solves storage, access control, audit and versioning. Delivery, caching, rotation tolerance and the identity that authenticates to it are all still yours to design.
- "All the providers offer the same thing." They offer the same category. Versioning semantics, managed rotation coverage, audit granularity, rate limits and cost per secret differ enough to change a design.
- "Encrypted at rest means safe." It means safe from someone reading the underlying storage. Access control decides who can read the value through the API, and that is the path an attacker will actually use.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Read access to a specific secret is auditable: you can produce the list of identities that read it in the last thirty days.
- A workload attempting to read a secret outside its scope is denied, demonstrated by trying it rather than by reading the policy.
- Rotating a value propagates to running workloads within a known, measured time.
- The previous version of any secret is retrievable, so a bad write is recoverable.
- A bad secret write rolls back to the previous version where versioning exists — this is one of the concrete reasons versioning matters.
- Where the store has no versioning, a bad write is unrecoverable: the previous value is gone and the credential must be reissued at the source system, which is a longer and riskier path.
- Rolling back the *access policy* is straightforward and often insufficient — if a broader policy was live, assume anything in scope was readable and treat it accordingly (The Secret Lifecycle).
- Automate issuance, delivery and rotation. A human copying a value between systems is both the largest exposure and the least reliable step (Toil).
- Automate the inventory and the last-rotated report from the store itself.
- Automate alerting on anomalous read patterns — a workload reading secrets it has never read before is a signal worth having (Alert on Symptoms, Not on Causes).
- Do not automate broadening an access policy. Widening scope is the change that should always require a human and a reason.
- A managed secret store is a new runtime dependency with its own availability, latency and rate limits, and it sits on the startup path of everything.
- Fine-grained policies are more work to maintain, and the maintenance burden is what pushes teams back toward wildcard scopes.
- Per-secret pricing and per-request costs make a naive implementation that fetches on every request surprisingly expensive (Cost Drivers).
- Provider-native stores integrate best and bind you tightly to one provider's identity model, which is a real cost if multi-provider is a requirement.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- CLOUD-SPECIFICThe five properties are common to the category; the products are not equivalent. They differ in whether every write creates an addressable version or the store is last-write-wins, in whether managed rotation is offered for a given credential type or must be written yourself, in whether reads are audited by default or must be enabled, and in per-secret and per-request pricing. Verify each of those against the specific service rather than carrying an assumption across providers (Secrets in Infrastructure).
- PLATFORM-SPECIFICDelivery differs by runtime. Container platforms can mount a secret as a file that an agent updates in place, giving rotation without a restart if the application re-reads. Serverless platforms generally inject at deploy time, so rotation means a deployment unless the function fetches from the store itself. Virtual machines usually run an agent that writes to a local file, adding a local artifact that must be protected (Files, Paths and Names).
- SCALE-SPECIFICFor a handful of services, a platform secret object with encryption at rest enabled and tight access control is a reasonable stopping point. The case for a dedicated manager grows with the number of consumers, the need for read audit, and the need to rotate without redeploying — roughly, when nobody can name every consumer of a credential from memory.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — exercising the secret-store-unavailable path, which is on the startup route of every service and is almost never tested.