Security Overview: Guardrails and the Threat Model
An agent that reads untrusted text and can call tools is a confused deputy by construction; security is layered checks in code around the model, not a paragraph in the prompt.
Why agents change the threat model
A classic web app has a clear trust boundary: code is trusted, user input is data, and SQL parameters keep the two apart. An LLM agent erases that boundary inside the model. The system prompt, the user message, a retrieved document and a tool result all arrive as tokens in one context window, and the model has no hardware-enforced notion of which tokens are instructions and which are data.
Add tools and the consequence becomes concrete: whoever can influence the context can influence which tools run and with which arguments. That includes the user, any web page the agent browses, any file it opens, and any upstream system whose output lands in a tool result. This is the confused deputy problem: the agent has authority, and an attacker borrows it by supplying text.
The three assets you protect are the same as always: confidentiality (customer data, secrets, internal documents), integrity (no unauthorised writes, payments, deletions) and availability (no runaway cost or loops). What changes is that a single sentence of natural language can now be the exploit.
Defence in depth around the model
Because the model cannot be made reliably obedient, every control that matters lives outside it. The diagram shows the minimum set of checkpoints a tool-using agent should pass through. Each one is ordinary code: a classifier, a schema check, an authorisation lookup, a regex, a human approval queue.
The principle is that each layer assumes the previous one failed. The input guardrail will miss some injections, so the permission layer restricts what a hijacked agent can do. The permission layer will occasionally allow a dangerous call, so result validation and the output guardrail stop the damage from reaching a user or an external system.
- Input guardrail: classify and filter the incoming request (jailbreak patterns, off-topic, PII) before the model sees it.
- Tool permission layer: per-tool, per-user authorisation and argument validation; approval gate for irreversible actions.
- Result validation: schema-check and sanitise tool output; mark it as untrusted before it re-enters the context.
- Output guardrail: check the final answer for leaked secrets, PII, and policy violations before it leaves the system.
The threat catalogue
Most incidents fall into a handful of classes, each with its own lesson in this module. Knowing the class tells you which layer should have caught it.
- Direct prompt injection (Prompt Injection): the user tries to override the system prompt.
- Indirect prompt injection (Indirect Prompt Injection): a web page, document or tool result carries instructions.
- Tool misuse and exfiltration (Tool Misuse and Data Exfiltration): destructive calls, data sent out via URLs, emails or file writes.
- Over-privileged agents (Permissions, Authentication and Authorisation): the agent runs with credentials broader than the user’s.
- Secret leakage (Secrets and Untrusted Output): API keys in context, logs or traces.
- Guardrail bypass (Input and Output Guardrails): encoded, translated or multi-turn attacks that classifiers miss.
What good looks like
A secure agent is one where you can list, for each tool, who may call it, with what arguments, under what approval, and what happens to its output. If the answer to any of those is “the prompt says not to”, that control does not exist. Treat the model as a talented but suggestible intern: give it a scoped badge, not the master key.
Security is also measurable. Keep a red-team set of injection prompts and poisoned documents in your eval suite (Evaluating Agents: Testing Probabilistic Systems) and track the attack success rate per release, the same way you track task success.
Key points
- The model cannot distinguish instructions from data; every trust boundary must be enforced in code around it.
- Layer controls: input guardrail, tool permission layer, result validation, output guardrail. Each assumes the previous one failed.
- The agent’s authority is the attack surface: scope tools and credentials to the minimum the task needs.
- Irreversible or exfiltration-capable actions require a human approval gate, not a polite system prompt.
- Red-team prompts and poisoned documents belong in the eval suite with a tracked attack success rate.
Defence in depth
When to use — and when not to
- Any agent that calls tools with side effects (write, send, pay, delete).
- Any agent that reads content it did not author: web pages, emails, PDFs, tickets, database rows.
- Any agent that holds credentials broader than a single anonymous read.
- A pure text transformation with no tools and no untrusted input needs only an output check, not the full stack.
- Do not add guardrail latency to an internal batch job that has no external inputs and no side effects.
- Do not bolt on guardrails as a substitute for fixing an over-privileged tool design.
Failure modes
- Security lives entirely in the system prompt (“never reveal the key”) and is bypassed in one turn.
- A single global API key is shared by every user of the agent, so one injection exposes everyone’s data.
- Tool results are appended to the context verbatim and treated as trustworthy.
- Guardrails are added at the output only, after the destructive tool call already ran.
- No red-team set exists, so a regression in injection resistance ships unnoticed.
Tradeoffs
Guardrail layers add small fixed latency per step; the permission layer costs almost nothing and buys the most.