AI Securitydirectandindirectpromptinjection

Direct and Indirect Prompt Injection

A user or retrieved document supplies language that the model may confuse with authority; the robust defense is to constrain capabilities and decisions outside it.

▶ Run the labFollow the failure

Frame the problem

Security starts with a concrete asset, attacker capability and trust crossing.

Asset
Agent goals, private context and any reachable tool capability.
Attacker & capability
A direct user or author of external content retrieved by the agent.
Trust boundary
Untrusted language → model instruction-following behavior
Asset→Threat→Attack Surface→Trust Boundary→Vulnerability→Exploit Path→Impact→Mitigation→Defense in Depth→Residual Risk

Why the system fails

The system assumes a prompt can permanently order the model to ignore malicious instructions while still exposing sensitive context and powerful tools.

The important question is not “what is Direct and Indirect Prompt Injection?” but “which assumption let untrusted data or an over-scoped identity cross untrusted language → model instruction-following behavior?” Trace the decision at the boundary, then constrain what can happen after the first control fails.

Design the control in layers

Start with the control closest to the interpretation or privilege boundary: Treat all external content as untrusted data Then add a control that reduces blast radius and telemetry that proves the decision was enforced.

The resulting design is not labelled secure. Record the identified controls, the known failure paths, the remaining exposure, and the evidence you would need during an incident.

PreventDetectRecover
Treat all external content as untrusted data · Keep secrets out of unnecessary context · Authorize each tool action deterministically and require approval when high riskPolicy-denied tool requests and action/context provenanceContain the affected identity or component, scope impact from audit evidence, and preserve a regression test.

Key points

  • Asset: Agent goals, private context and any reachable tool capability.
  • Boundary: Untrusted language → model instruction-following behavior
  • Primary control: Treat all external content as untrusted data
  • Detection signal: Policy-denied tool requests and action/context provenance
  • Always ask what limits damage when the primary control fails.

Prompt Injection Attack Path

Change the system and observe which assumption moves.

Choose an attack to follow
Follow an Attack: Broken Authorization
Goal: Read another customer’s invoice using a valid account.
ATTACK SUCCEEDS
  1. 01
    Valid login
    The attacker starts as an ordinary customer.
  2. 02 · TRUST CROSSING
    Change resource ID
    A client-controlled identifier selects another record.
  3. 03
    Authentication passes
    The cookie is valid; identity is known.
  4. 04 · TRUST CROSSING
    Ownership check missing
    The API loads by id without tenant/owner scope.
  5. 05
    Sensitive data returned
    A valid 200 response makes the breach quiet.
Click a defense to add it, then watch the path rerun
The attack above re-evaluates immediately. Read the trade-off before you add it.
0 of 3 added
Blast radius
Cross-user or cross-tenant invoice disclosure. Authentication logs misleadingly show a valid user.
Detection
No detection control selected: this can fail silently.
Residual risk
The asset is reached; contain the identity and scope access from logs.

Follow the attack

Safe conceptual simulation: capability → missing control → crossed boundary → asset impact.

  1. 1
    Attacker starts with: A direct user or author of external content retrieved by the agent.
  2. 2
    The system assumes a prompt can permanently order the model to ignore malicious instructions while still exposing sensitive context and powerful tools.
  3. 3
    The weak or missing boundary control is crossed: Untrusted language → model instruction-following behavior
  4. 4
    Impact: Tool misuse, data disclosure or task hijacking.
Blast radius
  • Tool misuse, data disclosure or task hijacking.

Defend, detect, recover

One prevention is a single point of security failure. Layer it and make failure observable.

Prevent
  • • Treat all external content as untrusted data
  • • Keep secrets out of unnecessary context
  • • Authorize each tool action deterministically and require approval when high risk
Detect
  • • Policy-denied tool requests and action/context provenance
Respond & recover
  • • Contain the affected identity or component.
  • • Scope access from audit evidence.
  • • Fix the boundary and add a regression test.
Residual risk
  • • Misconfiguration and new access paths can bypass the intended control.
  • • A privileged insider or compromised control plane may still reach the asset.

Misconceptions

Claim
“A single treat all external content as untrusted data control makes this safe.”
Reality
One control changes risk; it does not erase it. Design prevention, detection, recovery, and blast-radius limits together.
← Agent Trust BoundariesAgent Tool Capability Security →