Human Approval for High-Risk Agent Actions
Money transfer, deletion, external communication and permission changes should pause at an explicit risk gate with a comprehensible diff.
Frame the problem
Security starts with a concrete asset, attacker capability and trust crossing.
Why the system fails
A generic confirmation, approval fatigue or a mutable action after approval makes the human gate ceremonial.
The important question is not “what is Human Approval for High-Risk Agent Actions?” but “which assumption let untrusted data or an over-scoped identity cross agent proposal → external side effect?” Trace the decision at the boundary, then constrain what can happen after the first control fails.
Design the control in layers
Start with the control closest to the interpretation or privilege boundary: Classify high-risk actions and require meaningful approval Then add a control that reduces blast radius and telemetry that proves the decision was enforced.
The resulting design is not labelled secure. Record the identified controls, the known failure paths, the remaining exposure, and the evidence you would need during an incident.
| Prevent | Detect | Recover |
|---|---|---|
| Classify high-risk actions and require meaningful approval · Show exact target, effect and principal · Bind approval cryptographically/logically to the immutable action | Approval rate, overrides and post-approval mutation | Contain the affected identity or component, scope impact from audit evidence, and preserve a regression test. |
Key points
- Asset: Irreversible or high-impact business actions.
- Boundary: Agent proposal → external side effect
- Primary control: Classify high-risk actions and require meaningful approval
- Detection signal: Approval rate, overrides and post-approval mutation
- Always ask what limits damage when the primary control fails.
Boundary control exercise
This lesson uses the shared boundary-control exercise.
Follow the attack
Safe conceptual simulation: capability → missing control → crossed boundary → asset impact.
- 1Attacker starts with: Malicious context or an erroneous model proposal.
- 2A generic confirmation, approval fatigue or a mutable action after approval makes the human gate ceremonial.
- 3The weak or missing boundary control is crossed: Agent proposal → external side effect
- 4Impact: Destructive or unauthorized action despite nominal human involvement.
- Destructive or unauthorized action despite nominal human involvement.
Defend, detect, recover
One prevention is a single point of security failure. Layer it and make failure observable.
- • Classify high-risk actions and require meaningful approval
- • Show exact target, effect and principal
- • Bind approval cryptographically/logically to the immutable action
- • Approval rate, overrides and post-approval mutation
- • Contain the affected identity or component.
- • Scope access from audit evidence.
- • Fix the boundary and add a regression test.
- • Misconfiguration and new access paths can bypass the intended control.
- • A privileged insider or compromised control plane may still reach the asset.