HITLhitlriskreversibilityblast-radiusdry-run

Approval Gates and Risk Classes

Classify actions by reversibility and blast radius, preview them with dry-runs, design approval UIs that show what will actually happen, and audit every decision.

Interview question
Progress

Two axes: reversibility and blast radius

Risk is not one number. Reversibility: can the action be undone, at what cost, within what window? Archiving a record is reversible; sending an email is not; a database delete is reversible only if you have a backup and a restore path you have tested. Blast radius: how many users, records, euros or systems does one wrong execution touch? Editing one CRM contact is small; a bulk update matching a wrong filter is large.

Combine them into a few named classes rather than a continuous score — humans and policies reason better in classes. Compute the class in code from the tool name and the arguments: update_records(filter="*") is a different class from update_records(id=42).

  • Class 0 – free: read-only, no PII exposure. Auto-run, log.
  • Class 1 – reversible, small: create draft, add label, archive one item. Auto-run with audit; nightly review.
  • Class 2 – reversible, large or irreversible, small: bulk tag, send one internal message, refund ≤ limit. Single approval.
  • Class 3 – irreversible, large: external communications at scale, payouts, deletes, production changes. Approval with preview; two approvers above a threshold.
  • Class 4 – forbidden: actions the agent should never take (IAM changes, key rotation). Not exposed as tools at all.

Dry-run and preview

The approver must see what will happen, not the agent's description of what it intends. For a bulk update, run the filter and show "matches 1,204 records; here are 5". For a deploy, show the diff. For an email, render the exact body and recipient list. For a payment, show amount, currency, beneficiary and remaining budget after execution.

Building preview usually means every side-effecting tool gets a dry_run: true mode that returns the plan without executing it. That same mode is what your Deterministic Evaluators use in tests.

Risk class derived from arguments, with a dry-run preview
1def classify(tool: str, args: dict) -> int:
2    if tool in {"search", "read_record"}:
3        return 0
4    if tool == "issue_refund":
5        return 2 if args["amount_eur"] <= 500 else 3
6    if tool == "update_records":
7        n = count_matches(args["filter"])          # dry run: how many would change?
8        return 1 if n == 1 else 2 if n <= 50 else 3
9    if tool in {"send_email", "deploy", "delete_records"}:
10        return 3
11    raise PermissionError(f"tool {tool} is not allowed")   # class 4
12 
13def execute(tool: str, args: dict, run_id: str):
14    cls = classify(tool, args)
15    preview = TOOLS[tool](**args, dry_run=True)
16    if cls >= 2:
17        decision = await_approval(run_id, tool, args, preview, approvers=2 if cls == 3 else 1)
18        audit(run_id, tool, args, cls, decision)
19        if decision.status != "approved":
20            return {"error": "rejected", "reason": decision.reason}
21        args = decision.edited_args or args      # approver may edit before approving
22    return TOOLS[tool](**args)

Approval UI and audit

The UI decides whether approvals are real or rubber stamps. Show the preview first and the agent's reasoning second; put the consequential detail (recipient, amount, record count) in large type; make "reject" and "edit" as easy as "approve"; and never batch-approve heterogeneous actions with one button. Track time-to-decision — approvals under 3 seconds on class 3 actions are a signal of fatigue, not efficiency.

Audit every gate decision with: run id, tool, arguments, risk class, preview hash, approver identity, decision, timestamp, and whether arguments were edited. This log is what an incident review reads, and what you use to recalibrate classes — actions reverted after approval were misclassified too low; actions always approved unread may be classified too high.

  • Approver edits are the highest-value signal: they show what the agent got almost right.
  • Expire approvals: an approval granted for a preview is void if the underlying data changed (Idempotency for the execution itself).

Key points

  • Classify by reversibility × blast radius into a handful of named classes.
  • Compute the class from tool name and arguments, in code.
  • Every side-effecting tool needs a dry-run mode; approvers see the preview, not the intent.
  • Design the UI so rejecting and editing are as easy as approving; watch time-to-decision.
  • Audit every decision; use reversal and rubber-stamp rates to recalibrate.

When to use — and when not to

Use it when
  • Designing the policy layer for any agent with side-effecting tools.
  • Turning a vague "ask before dangerous things" requirement into testable rules.
  • After an incident, to reclassify the action that caused it.
Avoid it when
  • Do not build five classes for an agent with two tools — two classes suffice.
  • Do not gate on tool name alone when arguments determine the blast radius.
  • Do not ask for approval without a preview; that trains rubber-stamping.

Failure modes

  • Blast radius hidden by a wildcard filter that the classifier did not evaluate.
  • Preview computed from stale data; the executed action differs from what was approved.
  • Batch approve button on mixed-class actions.
  • Audit log without arguments — useless in incident review.
  • Class 4 actions exposed as tools "just in case".
← Human-in-the-Loop OverviewIn-the-Loop vs On-the-Loop and Escalation →