Least Privilege in Production
The operational form of a security principle: permissions sized to the task, scoped to the resources, and bounded in time — with an honest account of what that costs during an incident.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
How do you apply least privilege to production operations without making incidents longer?
Least privilege is easy to state and hard to operate. Permissions are granted for a real need, the need passes, the permission stays, and the effective privilege of every long-serving engineer grows monotonically.
Grant what people ask for. They know what they need, and refusing slows everyone down for a theoretical benefit.
Requests are sized to the worst case someone can imagine, not to the task. "I might need to check the database" becomes standing database access.
- Requests are sized to the worst case someone can imagine, not to the task. "I might need to check the database" becomes standing database access.
- Grants have no expiry, so privilege only accumulates. The permission set of a five-year engineer is the union of every project they have touched.
- Nobody can compute the effective permissions of a person or a service, because they come from group membership, inherited policies and resource-level grants that interact (Anatomy of a Policy).
- Service accounts get the same treatment and are worse: a CI role granted deploy access to one service quietly becomes deploy access to everything, because that was easier than enumerating (CI Security).
- When an account is compromised, the blast radius is whatever accumulated. Nobody intended it and nobody can describe it.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Privilege has three independent dimensions and least privilege means reducing all three. What — which operations. Where — which resources. When — for how long. Most implementations reduce the first, sometimes the second, and forget the third entirely.
- The time dimension is the cheapest and most under-used. A permission that exists for two hours is unavailable to an attacker for the other 8,758 hours in the year, and that reduction requires no analysis of what the permission does (Short-Lived Credentials).
- The resource dimension is where the largest practical reduction usually is.
deploy any serviceanddeploy the three services my team ownsare the same operation with very different consequences when it goes wrong. - Privilege creep is structural, not a discipline problem: grants are made by people responding to an immediate need and revocations are made by nobody, because nothing generates the request to revoke. Expiry is the only mechanism that fixes an asymmetry like that (Access Review).
- The genuine cost is incident latency. A tightly-scoped role means that occasionally, at 3am, the operator does not have what they need, and the difference between a good implementation and a bad one is entirely how long it takes to get it (Break-Glass Access).
- The strongest form is not a smaller permission — it is no permission, because the capability is delivered by tooling that can only do the safe thing (Guardrails, Not Gates).
Three dimensions, and the one everyone forgets
A permission is a triple: an operation, a resource scope and a duration. Reducing any one reduces exposure, and they are independent — which means the cheapest available reduction is usually the one nobody has applied.
Work down the table. The first row is what "give them admin" produces. The last is the same engineer, able to do the same job, with an exposure surface that is smaller by a factor nobody would guess from how similar the two rows look on paper.
| Grant | What | Where | How long | Exposure if the credential leaks |
|---|---|---|---|---|
| Cluster admin, standing | Every operation | Every resource | Indefinite | Total control of production, permanently |
| Namespace admin, standing | Every operation | One namespace | Indefinite | Total control of one team's services, permanently |
| Operator role, standing | Restart, scale, roll back, read logs | Owned services | Indefinite | Disruption of owned services; no deletion, no data |
| Operator role, session-bound | Restart, scale, roll back, read logs | Owned services | Working session | The same, and only while a session is live |
| Operation via pipeline | Rollback only | Services the pipeline knows | Per invocation | One reversible operation, fully attributed |
Why privilege only ever goes up
Privilege creep is not caused by carelessness. It is caused by an asymmetry: granting has a requester, an urgency and a person who benefits, and revoking has none of those. Every grant is somebody's Tuesday problem; no revocation is anybody's.
Discipline does not fix asymmetries. Expiry does, because it moves the default: the permission goes away unless someone actively wants it, which is the reverse of the current arrangement and requires no one to remember anything.
- 1Request
Someone needs access to complete a specific task.
fails by Sized to the worst case they can imagine rather than to the task.
evidence A stated task and a scope that matches it.
- 2Grant
Permission is added, usually quickly and usually broadly.
fails by Granted by copying an existing role because enumerating was slower.
evidence The grant names operations and resources, not a role template.
- 3Use
The task is completed.
fails by Nothing fails here — this step always works, which is why the next one is skipped.
evidence Usage logged, so later removal can be evidence-based.
- 4Task ends
The reason for the permission stops existing.
fails by No event is generated. Nobody knows the need has passed, including the holder.
evidence An expiry date set at grant time, which is the only reliable version of this step.
- 5Revoke
Permission is removed.
fails by Requires someone to notice and act; nobody is assigned and nothing is broken by inaction.
evidence Automatic expiry, with review as the backstop rather than the mechanism (Access Review).
- 6Accumulate
What happens instead: the permission stays and the next one is added on top.
fails by This is the failure. Effective privilege becomes the union of every task the person has ever done.
evidence An effective permission set nobody can enumerate.
The only step with no natural forcing function is "task ends". Every implementation of least privilege that works substitutes an automatic mechanism for that step, and every one that fails relies on someone remembering.
What people actually do, and what it needs
Roles derived from titles are wrong in both directions at once. Roles derived from operations are the fix, and building one starts by listing what the job involves.
The pattern in this table is the point: nearly all routine work needs read and a small set of bounded, reversible operations. The dangerous capabilities correspond to rare tasks, which is exactly the profile that on-request access is designed for.
| Operational task | Minimum capability | How often | Grant shape |
|---|---|---|---|
| Diagnose elevated error rate | Read telemetry, read logs | Every incident | Standing |
| Roll back a bad release | Trigger rollback for owned services | Weekly to monthly | Standing, or via pipeline (Rollback: Only Useful If It Is Actually Safe) |
| Restart a stuck consumer | Restart owned workloads | Monthly | Standing, scoped to owned namespace |
| Scale for a traffic spike | Change replica count within a bound | Monthly | Standing, with a ceiling (Autoscaling) |
| Disable a broken feature | Toggle flags for owned services | Monthly | Standing (Feature Flags: Deploy Is Not Release) |
| Change a config value under pressure | Write config for owned services | Quarterly | Reviewed path; direct write on request (A Config Change Is a Production Change) |
| Investigate a data-shaped bug | Read specific tables, time-boxed | Quarterly | On request, audited, expiring (Database Privileges and Blast Radius) |
| Repair corrupted rows | Write specific rows | Rarely, always an emergency | Break-glass with a second approver (Break-Glass Access) |
| Recreate deleted infrastructure | Infrastructure write | Rarely | Through IaC review; break-glass if the pipeline is what is broken (Destructive Changes: What a Rename Really Does) |
How to do it properly
Most important first.
- Derive roles from observed operations. Log what people actually do for a month and build the role from that, rather than from what they say they need.
- Scope to resources by default: this namespace, these services, this bucket prefix, this database — not the account.
- Make grants expire. Default to a bounded window with easy renewal; renewal that is used is a signal that the base role is wrong.
- Separate humans from workloads. Machines get workload identity, scoped per service; humans never share a workload credential (Human vs Workload Identity).
- Grant the deploy pipeline what it needs, not what its most demanding service needs — one over-broad CI role is the shortest path from a repository to production (Securing the Pipeline Itself).
- Remove permissions that have not been used in a period. Unused permission is pure risk with no offsetting benefit.
- Measure the friction. If people are elevating routinely, the base role is too small, and that is a finding rather than a compliance failure.
- Invest in tooling that removes the need for the permission entirely, starting with the operations that currently require the most dangerous grants.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
The permission scope is the containment. That is the entire mechanism: a scoped role bounds a compromise or a mistake to the resources it names, and an unscoped one bounds nothing.
What can go wrong
- Roles derived from job titles rather than operations, which produces roles that are simultaneously too broad and missing something.
- Permission models complex enough that nobody can answer "who can do X", which is indistinguishable from having no model (Anatomy of a Policy).
- Expiry set so short that renewal is constant, training everyone to click through it.
- Scoping by resource names that change, so the scope silently stops matching and the grant either over-permits or breaks.
- Denying a permission that the deploy pipeline needs, discovered at the worst moment during a rollback.
- Least privilege applied to humans and ignored for service accounts, which is where most of the standing privilege actually lives.
- A privileged role removed without a working elevation path, producing an incident where the correct mitigation is not available to anyone awake.
- "Least privilege means nobody can do anything." It means the permission matches the task. Most operational tasks need modest permissions; the model should make those easy and the dangerous ones deliberate.
- "We did least privilege when we set up the roles." It is a continuous process. Roles set up two years ago describe a system and a team that no longer exist (Access Review).
- "Time-bounded access is too much friction for on-call." Standing access for the common operations plus fast elevation for the rare ones is the working arrangement. The friction argument is against the wrong design.
- "Service accounts do not need least privilege because they are not people." They are the accounts with standing credentials, no MFA and the broadest grants. They are the higher risk, not the lower one.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- You can produce the effective permission set for any human or service, and it is short enough to read.
- Unused permissions have actually been removed in the last quarter.
- The proportion of operations that require elevation is small and stable — high means the base role is wrong, zero means it is too broad.
- Service accounts are scoped per service; none of them is a shared deploy identity for everything.
- A compromised credential exercise can name what it would have reached, and the answer is bounded.
- Tighten in stages with the old permission still present and logged. When the log shows the broad grant unused for a period, remove it. This turns a risky cutover into an observation.
- Keep an elevation path available throughout, and rehearse it before the tightening, not after. The failure mode of a privilege reduction is an incident where nobody has what they need and the emergency path is untested.
- Automate grant, expiry and revocation. Manual revocation does not happen; the asymmetry between granting and revoking is the whole mechanism of privilege creep.
- Automate usage analysis: which permissions in each role were exercised in the last ninety days, so removal is evidence-based rather than a guess.
- Automate detection of over-broad service accounts — wildcards in policies, roles with permissions no workload has used (Policy as Code).
- Do not automate permission reduction directly from usage data. A permission used once a year during failover looks unused for 364 days (Region Failover).
- Do not automate emergency elevation approval. The human decision is the control.
- Tighter scoping means more roles to maintain, and a role per team per environment per service is a real maintenance burden that someone has to carry.
- Time-bounded access adds a step to every legitimate task. Where that step is slow, it will be routed around, and the routing-around is invisible.
- Deriving roles from observed usage encodes current practice, including the parts that were wrong. It needs review, not just data.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe three dimensions — what, where, how long — hold on any platform. Which are expressible varies: some systems make time-bounding trivial and resource-scoping awkward, and others the reverse.
- CLOUD-SPECIFICPolicy evaluation differs materially between providers — how deny interacts with allow, how inherited and resource-attached policies combine, what a wildcard matches. A policy translated between clouds without re-analysis routinely over-permits (§178).
- ORG-SPECIFICHow much friction is acceptable, and whether separation of duties is mandatory, is set by industry and regulator. Regulated environments may require an approver who cannot also be the requester; most organisations are choosing.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — System Design — designing operations so that the dangerous capability is never required by the routine path.