Break-Glass Access
Emergency elevated access that is explicit, audited, short-lived and automatically revoked — the answer to "but what if we need admin during an incident".
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
How do you make emergency access available in seconds without leaving it available all the time?
Least privilege will occasionally be wrong. At some point, during an outage, someone will genuinely need a capability their role does not have — and the two obvious responses are both bad: block them, or give everyone admin permanently.
Keep a shared admin credential in the password manager for emergencies. Everyone on call knows where it is.
A shared credential has no actor. The audit log records that the admin account did something, which is the least useful sentence a log can contain (Audit Logs for Privileged Actions).
- A shared credential has no actor. The audit log records that the admin account did something, which is the least useful sentence a log can contain (Audit Logs for Privileged Actions).
- It never expires and it is never rotated after use, so every person who has ever been on call still holds full production access.
- It gets used for non-emergencies — a quick fix, a Friday deploy — because it is the fastest path and nothing distinguishes an emergency from a hurry.
- It leaks like any other shared secret: pasted into a terminal that is screen-shared, copied into a script, left in shell history (When Secrets Fail).
- Because it is informal, nobody reviews its use. There is no signal that the everyday role is systematically insufficient, so the underlying gap is never fixed.
- It fails the moment the person who set it up leaves, or the password manager itself is part of the outage.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Break-glass is a procedure, not a credential. Its four properties are what make it work: it is explicit (deliberately invoked, never ambient), audited (individually attributed, with a reason), short-lived (expires on a clock, not on remembering) and revocable (removable immediately, and automatically at expiry).
- The design tension is stated exactly: it must be fast enough that nobody builds a faster unofficial path, and deliberate enough that using it is a decision. Those pull against each other, and the resolution is that it is fast to *invoke* and impossible to invoke *accidentally*.
- Approval is where organisations differ most. Pre-approval — you invoke it and someone reviews afterwards — optimises for incident speed. Two-person approval optimises for control and adds minutes. Which is right depends on what the elevated access can reach and what your regulator requires.
- Expiry is what makes the whole thing safe. Access granted for sixty minutes is gone in sixty minutes whether or not anyone remembers, which is the property no manual process has ever had.
- The frequency of use is a signal about the base role, and it is the most valuable output of the mechanism. Break-glass used weekly for the same capability means the everyday role is missing that capability, and the fix is to move it into the base role rather than to normalise elevation (Least Privilege in Production).
- It must be tested, because it is a recovery path and untested recovery paths do not work. The specific failure to guard against is dependence on something that is part of the outage — an identity provider, a network path, a service that is itself down (Restore Drills).
The lifecycle
Every step here exists because of a specific way informal emergency access fails. Read the failure column as the list of things a shared admin password does wrong.
- 1Trigger
An operator determines their role cannot perform the needed mitigation.
fails by Invoked out of habit because it is faster than the correct path.
evidence An incident in progress and a capability the base role demonstrably lacks.
- 2Invoke
Elevation requested against an individual identity, with a reason and an incident id.
fails by Shared credential with no actor and no reason.
evidence A log entry naming a person, a reason and a scope.
- 3Approve
Pre-approved, or approved by a second person, depending on what the elevation can reach.
fails by Approver unavailable and no fallback, so the procedure blocks the incident.
evidence A recorded decision, or a recorded pre-approval policy that covers this case.
- 4Grant
Scoped elevated access issued with a hard expiry.
fails by Global admin issued because scoping was never built.
evidence A session with a scope and an expiry timestamp.
- 5Announce
Invocation posted to a channel other people watch.
fails by Silent, so nobody knows production admin is live.
evidence A message with actor, reason, scope and expiry (Telling People What Is Happening).
- 6Act
The operator performs the mitigation. Every action is logged and tagged.
fails by Actions indistinguishable from routine ones in the audit trail.
evidence A queryable set of actions attributed to the elevated session.
- 7Expire
Access is removed automatically at the deadline.
fails by Expiry is manual and the incident ran long, so it persists for days.
evidence Session terminated by the clock, visible in the log.
- 8Review
The use is examined: was it necessary, and should the base role have covered it?
fails by Never reviewed, so the signal about the base role is lost.
evidence A short note per use, and a base-role change when a pattern appears (Access Review).
Two steps are load-bearing and routinely skipped. Announce is nearly free and is most of the practical control. Review is the only step that converts break-glass from a safety valve into a feedback loop about whether least privilege is sized correctly.
An invocation, minute by minute
A corrupted-write incident where the mitigation genuinely required a capability nobody holds day to day. Note what the elevation was scoped to: write access to one table, not administrator. Most real emergencies are that specific, and scoping them is the difference between a bounded risk and an unbounded one.
Note also the last two rows. The access expired on its own while the engineer was writing up the incident, and the manual repair was reconciled back into a migration the next day — because a hand-written UPDATE that only exists in an audit log is drift, however necessary it was at the time.
- 02:14signalAlert: order confirmation failures rising. On-call paged.
- 02:21actionDiagnosis: a bad release wrote null
provider_refto roughly 400 orders before it was rolled back. Rollback stopped new corruption; existing rows stay broken. - 02:24actionRepair requires writing to the
orderstable. The operator role has read access only. - 02:25actionBreak-glass invoked: identity
dana, reason "repair null provider_ref from v412, INC-2291", scopeorderstable write, expiry 60 minutes. - 02:25signalInvocation posted automatically to
#ops-alertswith actor, reason, scope and expiry. Incident commander acknowledges. - 02:31actionAffected rows identified by a read query and the exact count recorded in the incident channel before any write.
- 02:38actionRepair applied in batches. Every statement logged to the audit store under the elevated session.
- 02:44recoveryConfirmation failures return to baseline. Spot-check of ten repaired orders passes.
- 02:47recoveryOperator revokes the elevation manually rather than waiting out the window.
- 03:25recoverySession would have expired automatically here regardless. Expiry is the control; the manual revoke is courtesy.
- next daychangeRepair statements committed as a reviewed data migration, so the change exists in code rather than only in an audit log (Manual Production Changes).
- next weekchangeReview: base role stays read-only for
orders; a repair tool with a narrow interface is added instead, since this is the third such repair this year.
The review outcome is the interesting part. Three repairs in a year is a pattern, and the response is neither "grant standing write access" nor "keep doing break-glass" — it is to build the narrow capability, which is the strongest form of least privilege (Least Privilege in Production).
How break-glass fails
Emergency procedures fail in ways ordinary procedures do not, because they are exercised rarely and under the worst possible conditions. Every row here is a mechanism that was in place and did not work when it was needed.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Identity provider is part of the outage | Nobody can authenticate to request elevation | Emergency path depends on a system that can be down | An independent path with its own credentials, tested; the whole point of a break-glass is that it does not share fate (Failure Domains) |
| Approver asleep, no fallback | Incident extends while someone is phoned | Single approver with no rotation | Approver rotation, or pre-approval with mandatory post-review for scoped elevations |
| Expiry is manual | Elevated access active for days after the incident | Revocation depends on remembering, after a long night | Hard expiry enforced by the system, always |
| Used routinely | Weekly invocations, all for the same operation | The base role is missing a capability people genuinely need | Move that capability into the base role; do not normalise the emergency path |
| Global admin granted for a narrow need | One table repair carries the risk of full production control | Only one elevation profile exists | Several scoped profiles — data repair, infrastructure, access — each with its own approval |
| Never tested | First real use fails on a broken dependency | A documented procedure that nobody exercised | Scheduled tests, including a degraded-conditions test (Disaster Recovery as an Operation) |
| Actions not distinguished in the log | Postmortem cannot tell what was changed by hand | Elevated actions written to the same undifferentiated stream | Tag actions by session; make "what happened under elevation" one query (The Audit Trail) |
| Console-only path | Unreachable from a phone at 3am on hotel wifi | Designed for the desk, used away from it | Test from the device and network the on-call engineer will actually have (On-Call Is Production Ownership) |
How to do it properly
Most important first.
- Make it individually attributed. Every invocation names a person, never a shared account.
- Require a stated reason at invocation — a free-text sentence and an incident id. Nobody should have to search chat later to reconstruct why production admin was live at 3am.
- Set a short expiry — commonly thirty to ninety minutes — with explicit extension available. Extension is a fine thing to need and a useful thing to count.
- Grant a scoped elevation rather than global admin where you can. "Write to the orders database" is a much smaller emergency capability than "administrator", and covers most real emergencies.
- Alert on invocation, loudly, to a channel other people watch. Visibility is most of the control, and it is nearly free.
- Log every action taken under elevation, to a store the elevated identity cannot modify (Audit Logs for Privileged Actions).
- Review every use — briefly, routinely, and without treating use as a fault. The question is whether the base role should have covered it.
- Test it on a schedule, including the case where the identity provider or the primary region is unavailable (Region Failover).
- Make sure it works for the person actually on call, from the device and network they will actually have at 3am.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Contained by scope and by the clock: a scoped, expiring elevation bounds what an error or a compromise during that window can reach. An unscoped one bounds only the duration.
What can go wrong
- Break-glass that depends on a system that is down during the outage — the identity provider, the VPN, the approval tool, the cloud console in the failed region.
- Approval required from someone who is asleep, with no fallback, so the procedure exists and cannot be used.
- Elevation that does not expire because expiry was manual and the incident was long.
- Routine use, which converts an emergency control into the normal path and removes every property that made it safe.
- Use that is never reviewed, so the signal about the base role is collected and discarded.
- Actions under elevation not distinguished in the audit log from ordinary actions, losing the one record that most needed to be findable.
- A procedure documented and never tested, discovered to be broken during the first real emergency.
- Elevation that grants global admin when the emergency needed one table, because scoping the emergency path was more work than not.
- "Break-glass means everyone gets admin during incidents." It means one named person gets a scoped, expiring, logged elevation for a stated reason. The permanent version is the red flag this exists to replace.
- "If we need it often, we should make it permanent." If you need it often, the base role is wrong. Move the specific capability into the base role; do not move everyone into the emergency path.
- "The audit log is for blame." It is for reconstruction. Postmortems need to know what was changed by hand, and honest logging depends on nobody being punished for appearing in it (Postmortems).
- "We have a procedure, so we are covered." An untested emergency procedure is a document. The failure mode is a dependency on something that is down, and you only find it by trying.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Break-glass has been invoked in a test within the last quarter and worked end to end.
- Every invocation in the log has an individual actor, a reason and an incident reference.
- Elevated sessions in the log all show an expiry, and the expiries were reached automatically.
- Frequency is low and stable; where it clusters on one capability, that capability moved into the base role.
- The procedure has been exercised with the identity provider unavailable, or that dependency has been deliberately removed.
- Revocation is the rollback, and it must work in both forms: immediate manual revoke, and automatic expiry. Automatic expiry is the one that actually protects you, because manual revocation depends on someone remembering after the incident ends.
- If elevated access was used to make a manual change, the change itself still needs reconciling back into code — revoking the access does not undo the drift (Manual Production Changes).
- Rotate anything the elevated session could have exposed if the session was on a shared or untrusted device.
- Automate grant and expiry end to end. A human in the grant path is a delay during an incident; a human in the *review* path afterwards is the control.
- Automate the announcement — invocation posts to an operations channel with actor, reason and expiry.
- Automate action logging under elevation, tagged so that a later query for "what happened under break-glass last month" is one search.
- Automate the periodic test so the path is exercised without anyone scheduling it.
- Do not automate the decision to invoke. Deciding that this is an emergency is the human judgement the whole mechanism is built around.
- Do not automate away the review. A break-glass log nobody reads is a compliance artifact, not a control.
- Approval gates add minutes to an incident. Skipping them adds risk. There is no configuration that avoids the trade; there is only choosing which one you can afford for a given capability.
- Building this properly — identity, scoping, expiry, logging, alerting, testing — is a real project, and it competes with work that has visible output.
- Short expiry means renewal mid-incident, which is friction at the worst moment. Long expiry means standing access with extra steps. Thirty to ninety minutes is a common compromise and not a derived answer.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- ORG-SPECIFICApproval model, expiry duration and review cadence are conventions shaped by size and regulation. A regulated environment may require two-person approval and a formal record within a fixed window; a small team may use self-service elevation with loud announcement and weekly review. Both are legitimate; a shared admin password in a vault is not.
- CLOUD-SPECIFICProviders offer different primitives for temporary elevation — role assumption with a session duration, just-in-time role activation, time-bound policy bindings — and their maximum session lengths and revocation semantics differ. What is instant on one provider may be eventually-consistent on another, which matters for revocation (§178).
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — emergency procedures as recovery paths, which are only real once they have been exercised under the conditions they exist for.