AI Securityandagentsecurity

AI and Agent Security

Agents combine untrusted language, retrieved data, memory, models and tools; capability boundaries—not model obedience—control the outcome.

▶ Run the labFollow the failure

Frame the problem

Security starts with a concrete asset, attacker capability and trust crossing.

Asset
User data, tool capabilities, secrets and high-impact external actions.
Attacker & capability
A user, document, website, tool result or poisoned memory item able to influence model context.
Trust boundary
Untrusted context → model proposal → privileged tool
Asset→Threat→Attack Surface→Trust Boundary→Vulnerability→Exploit Path→Impact→Mitigation→Defense in Depth→Residual Risk

Why the system fails

The system treats generated output as authority and gives one agent broad tools, credentials and autonomy.

The important question is not “what is AI and Agent Security?” but “which assumption let untrusted data or an over-scoped identity cross untrusted context → model proposal → privileged tool?” Trace the decision at the boundary, then constrain what can happen after the first control fails.

Design the control in layers

Start with the control closest to the interpretation or privilege boundary: Keep authorization and policy outside the model Then add a control that reduces blast radius and telemetry that proves the decision was enforced.

The resulting design is not labelled secure. Record the identified controls, the known failure paths, the remaining exposure, and the evidence you would need during an incident.

PreventDetectRecover
Keep authorization and policy outside the model · Use narrow tools, scoped identity and approval for high-risk actions · Sandbox execution and isolate tenant contextTool calls, policy denials, context provenance and unusual action chainsContain the affected identity or component, scope impact from audit evidence, and preserve a regression test.

Key points

  • Asset: User data, tool capabilities, secrets and high-impact external actions.
  • Boundary: Untrusted context → model proposal → privileged tool
  • Primary control: Keep authorization and policy outside the model
  • Detection signal: Tool calls, policy denials, context provenance and unusual action chains
  • Always ask what limits damage when the primary control fails.

AI Security Failure Simulator

Change the system and observe which assumption moves.

Choose an attack to follow
Follow an Attack: Broken Authorization
Goal: Read another customer’s invoice using a valid account.
ATTACK SUCCEEDS
  1. 01
    Valid login
    The attacker starts as an ordinary customer.
  2. 02 · TRUST CROSSING
    Change resource ID
    A client-controlled identifier selects another record.
  3. 03
    Authentication passes
    The cookie is valid; identity is known.
  4. 04 · TRUST CROSSING
    Ownership check missing
    The API loads by id without tenant/owner scope.
  5. 05
    Sensitive data returned
    A valid 200 response makes the breach quiet.
Click a defense to add it, then watch the path rerun
The attack above re-evaluates immediately. Read the trade-off before you add it.
0 of 3 added
Blast radius
Cross-user or cross-tenant invoice disclosure. Authentication logs misleadingly show a valid user.
Detection
No detection control selected: this can fail silently.
Residual risk
The asset is reached; contain the identity and scope access from logs.

Follow the attack

Safe conceptual simulation: capability → missing control → crossed boundary → asset impact.

  1. 1
    Attacker starts with: A user, document, website, tool result or poisoned memory item able to influence model context.
  2. 2
    The system treats generated output as authority and gives one agent broad tools, credentials and autonomy.
  3. 3
    The weak or missing boundary control is crossed: Untrusted context → model proposal → privileged tool
  4. 4
    Impact: Data exfiltration, unauthorized actions, persistent poisoning or destructive tool use.
Blast radius
  • Data exfiltration, unauthorized actions, persistent poisoning or destructive tool use.

Defend, detect, recover

One prevention is a single point of security failure. Layer it and make failure observable.

Prevent
  • • Keep authorization and policy outside the model
  • • Use narrow tools, scoped identity and approval for high-risk actions
  • • Sandbox execution and isolate tenant context
Detect
  • • Tool calls, policy denials, context provenance and unusual action chains
Respond & recover
  • • Contain the affected identity or component.
  • • Scope access from audit evidence.
  • • Fix the boundary and add a regression test.
Residual risk
  • • Misconfiguration and new access paths can bypass the intended control.
  • • A privileged insider or compromised control plane may still reach the asset.

Misconceptions

Claim
“A single keep authorization and policy outside the model control makes this safe.”
Reality
One control changes risk; it does not erase it. Design prevention, detection, recovery, and blast-radius limits together.
Agent Trust Boundaries →