Rotation That Applications Survive

1. 1
   Old valid

   The steady state. Every consumer uses the current credential.

   fails by Not knowing who the consumers are, which is the usual starting condition.

   evidence The store's read audit and the target system's per-credential usage list the actual consumers (Audit Logs for Privileged Actions).

2. 2
   Create new, both valid

   Issue a second credential at the target system; write it to the store as a new version.

   fails by The target accepting only one credential, discovered at this step rather than during planning.

   evidence Both credentials authenticate successfully, tested explicitly.

3. 3
   Transition

   Consumers pick up the new value as their cache expires or on their next re-read.

   fails by Applications that cache for the process lifetime, so nothing transitions until a restart.

   evidence Usage of the new credential rises while usage of the old falls — two curves, not an assumption.

4. 4
   Verify adoption

   Confirm old-credential usage has reached zero and stayed there through a full cycle, including nightly jobs.

   fails by Waiting a fixed period instead of observing, so a weekly job is missed entirely.

   evidence Zero usage of the old credential across a period longer than your longest job interval.

5. 5
   Revoke old

   Delete or disable the old credential at the issuer.

   fails by Skipping it, which is extremely common and negates the rotation.

   evidence The old credential is rejected when deliberately tested.

6. 6
   Record

   Update the inventory with the rotation date and the new version.

   fails by No record, so the next rotation starts with the same enumeration problem.

   evidence The inventory's last-rotated date moved.

Phases one to four are reversible; phase five is not. Gate revocation on the evidence from phase four rather than on elapsed time, and expect the stragglers to be scheduled jobs rather than services (Cron Jobs in Production).

1// Wrong: correct for the life of the process, and only that.
2const dbPassword = await secrets.get('db/app')      // read once at boot
3const pool = createPool({ password: dbPassword })   // captured forever
4 
5// Right: the current value is always fetched through something
6// that knows how to get a newer one.
7const password = cached(() => secrets.get('db/app'), { ttl: '5m' })
8 
9const pool = createPool({
10  // Called per new connection, not once at construction.
11  credentials: async () => ({ password: await password.get() }),
12})
13 
14// And on rejection, invalidate rather than retrying the same value.
15pool.on('authError', async () => {
16  password.invalidate()        // next get() goes to the store
17  await pool.reconnect()       // bounded, with backoff
18})

Three properties, none of them about the secret store. The TTL bounds how long a rotation takes to propagate. The per-connection callback means pool connections created after a rotation use the new value without a restart. And invalidate-on-rejection turns the worst case — a rotation that propagated late — from an outage into a few seconds of retries. Without the third, a fleet whose cache has not yet expired fails for the full TTL (When Secrets Fail).

1. Mon 10:00changeNew credential created at the database. Both old and new valid. New version written to the secret store.
2. Mon 10:05signalServices with a 5-minute cache TTL begin using the new credential. Old-credential usage falls steadily.
3. Mon 10:30signalOld-credential usage from services reaches zero. The dashboard looks finished.
4. Tue 10:00changeAutomation revokes the old credential — 24 hours elapsed, as configured. No check on actual usage.
5. Tue 10:00signalNo effect. Every service is on the new credential. The rotation is recorded as successful.
6. Sun 02:00changeThe weekly reconciliation job starts. It reads its credential from a mounted file written at deploy time, six weeks ago.
7. Sun 02:00signalAuthentication refused. The job retries three times, exits non-zero, and alerts to a channel that is quiet at 2am on a Sunday.
8. Mon 09:15signalA finance query returns figures that are a week stale. Someone asks in a channel.
9. Mon 10:40actionThe failed job is found. The deploy timeline shows nothing for the previous six weeks, so the change is not obvious (Deploys on the Same Timeline as the Symptom).
10. Mon 11:20actionThe store's read audit shows the job never fetched the new version — it reads a file, not the store.
11. Mon 11:50recoveryJob redeployed to fetch from the store with a TTL. Backfill run for the missed week.

The overlap worked exactly as designed. What failed was gating revocation on elapsed time rather than on observed non-use — with a consumer whose interval was longer than the window. A usage check before revocation would have shown a non-zero count from an identity nobody had enumerated (Job Scheduler Reliability).