AgenticGENERALSIMPLIFIEDSCALE-SPECIFIC
What an Agent Adds to a Backend
The request path when a model sits in the middle: what is genuinely new, and what is the backend you already know.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
The question
What changes in a backend when a language model decides which work to do, and what stays exactly the same?
The requirement
Support wants an assistant that can answer "where is my order", check the shipment, and issue a refund when policy allows. It should feel like a chat box and behave like the rest of the product.
The obvious build
Add an endpoint that takes the user's message, sends it to the model with a list of tools, runs whatever tool the model asks for, and returns the model's final answer.
Why it breaks
The same request produces different tool sequences on different days, so a bug reported on Monday cannot be reproduced on Tuesday (Debugging a Backend in Production).
How it breaks in production
- The same request produces different tool sequences on different days, so a bug reported on Monday cannot be reproduced on Tuesday (Debugging a Backend in Production).
- Latency is no longer a property of your code: one request finishes in a second and another takes forty because the model chose six tool calls instead of one.
- Cost is now per request and unbounded from the caller's side — a loop that keeps calling tools keeps spending, and nothing in the HTTP layer notices.
- A plan fails halfway: the refund was issued, the ledger entry was not, and the model apologises in prose while the data is inconsistent (Where the Transaction Boundary Goes).
- A long-running request holds a connection, a worker slot and a pooled database connection for the entire agent loop (Connection Pool Exhaustion).
- The tool handler was written assuming your own frontend calls it, so it trusts its arguments — and now the caller is a model reading untrusted text (A Tool Call Is a Backend Call).
Requirement→API Contract→Application Logic→Data Access→External Deps→Concurrency→Failure→Security→Observability→Deployment→Scale
What is actually happening
- The request path gains a loop. Instead of request → handler → response, it is request → agent runtime → model → tool → model → tool → … → response, with the model deciding when to stop (The Agent Loop in the Agentic AI domain).
- The agent runtime is ordinary backend code: it builds a prompt, calls a model over HTTP, parses a structured tool request, dispatches it, and feeds the result back. Nothing about it escapes normal engineering.
- Each tool call is an inbound call to your own service from a client whose behaviour is statistical rather than specified. That is the whole security and correctness story of this module.
- Four properties are genuinely new: nondeterminism (the same input can produce different work), variable latency (bounded only by the loop), per-request cost (tokens, priced), and partial failure mid-plan (steps three through five ran, six failed, and there is no transaction spanning them).
- Everything else is the backend you already operate: authentication, authorization, validation, timeouts, idempotency, rate limits, observability, deployment (The Backend Security Checklist).
- Because the loop is long and the output streams, the natural shape is often not a synchronous request at all but a job with progress updates (Background Jobs).
The request path, with a loop in the middle
The most useful thing to draw on a whiteboard before building this is the path a request actually takes. It makes two things obvious that prose does not: the model sits outside your trust boundary in the same way a browser does, and the loop between model and tools has no natural stopping condition other than the one you impose.
Notice that the tools point back into the same services your public API points at. That is the whole reason the enforcement question is not optional.
UserLLMAgentToolDataDecisionHumanGuardrail
Four properties that are actually new
It is worth being precise about what changed, because the list is short. Almost everything else in an agent-enabled backend is a problem you have solved before under a different name — and treating it as novel is how teams end up with an authorization model that exists only inside a prompt.
The new failure surface
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| The model chooses a different plan for the same input | A bug that reproduces one time in five; support cannot repeat it | Nondeterminism: tool selection is sampled, not specified | Log the full trajectory and replay from it; assert on tool choice in evals rather than in unit tests (Agent Audit Logs) |
| A plan takes eleven steps instead of two | p99 latency an order of magnitude above p50 with no infrastructure cause | Latency is a function of step count, which the model decides | Bound steps and wall-clock time; report termination reason (Budgets, Deadlines and Step Limits) |
| A loop retries a failing tool indefinitely | One request accrues a large model bill; the budget alert fires after the fact | Cost is per request and has no natural ceiling | Enforce a token and cost ceiling inside the loop, not only as a monthly alert |
| Step 4 of 6 succeeds, step 5 fails | Refund issued, ledger not updated, user told "something went wrong" | No transaction spans a multi-step plan (The Dual Write Problem) | Make each step independently reversible or independently safe; gate irreversible steps on approval |
| Tool results accumulate in the context | Quality degrades late in a long conversation; occasional context-limit errors | Context is a bounded resource being consumed by tool output | Truncate and summarise tool results; return ids and let the model fetch detail (Context Selection & Compression in the Agentic AI domain) |
| The model provider degrades | The whole feature is unavailable; no partial degradation is possible | A single external dependency on the critical path (Calling Something You Do Not Control) | Timeouts, a breaker, and a defined non-agent fallback (Circuit Breakers) |
What is new, and what you already own
The clearest way to scope the work is a two-column split: which concerns are new engineering, and which are existing backend concerns wearing a new label. The second column is much longer than teams expect, and every item in it already has an answer in this domain.
| Concern | New, or backend you already have? | Where it is decided |
|---|---|---|
| Which tool runs | New — chosen by the model | Prompt and tool schemas; never a substitute for enforcement |
| Whether that tool may run for this user | Existing — authorization | Server-side, per object (Object-Level Authorization) |
| Whether the arguments are valid | Existing — validation | The tool handler, against a schema (The Three Validations) |
| Whether a repeat is safe | Existing — idempotency | Idempotency key per intent (Idempotency Keys) |
| How long it may take | Both — deadline per tool and per loop | Runtime budget plus per-call timeouts (Timeouts) |
| How much it may cost | New — tokens are priced per request | Explicit ceiling in the loop (Budgets, Deadlines and Step Limits) |
| What happened and why | Both — trajectory plus normal logs | Trace with a span per step (Agent Audit Logs) |
| What the user is allowed to see | Existing — tenant isolation | Query scoping in the tool, not prompt instructions (Tenant Isolation) |
| What to do when a dependency fails | Existing — timeouts, retries, breakers | Client configuration (Retries) |
| Whether the answer is correct | New — evaluation, not testing | Golden datasets and judges in the Agentic AI domain (Evaluating Agents: Testing Probabilistic Systems) |
How to build it
Most important first.
- Treat the agent runtime as a service, with the same health checks, timeouts, metrics and deploy process as any other (Health Checks: Startup, Readiness, Liveness).
- Put every tool behind the same enforcement the public API has — authenticate the caller, authorize the specific object, validate arguments, apply rate limits (A Tool Call Is a Backend Call).
- Bound the loop explicitly: wall-clock deadline, maximum steps, maximum tokens and a cost ceiling. Without these it is an unbounded
whileloop with a credit card attached (Budgets, Deadlines and Step Limits). - Make every state-changing tool idempotent, because retries and repeated tool selection are normal, not exceptional (Idempotency Keys).
- Decide the transaction story per plan. Multi-step plans are distributed transactions without a coordinator; either each step is independently safe and reversible, or a human approves before the irreversible one (The Dual Write Problem).
- Run long agent work as a job with a request that returns immediately and a stream or poll for progress (Job Queues).
- Log the whole trajectory — prompt, tool calls, arguments, results, decision to stop. Without it, no agent incident is diagnosable (Agent Audit Logs).
What can go wrong
Failure modes
- The loop never terminates because the model keeps calling a tool that returns an error it does not understand.
- Tool results grow the context until the model degrades or the request exceeds the context limit mid-plan.
- A tool that is slow makes the whole agent slow, and there is no per-tool timeout (Timeouts).
- Concurrency amplification: one user request produces a dozen concurrent tool calls, so N users produce 12N load against internal services (Unbounded Concurrency).
- A model provider outage takes down a feature with no fallback path, because the agent is the only route to the functionality.
- Streaming responses that have already emitted text when a later tool fails, so the user sees a confident partial answer followed by an error.
- Caching model responses to save cost and serving one user's answer to another because the cache key omitted the identity (Cache-Aside).
What can race
- Parallel tool calls in one plan can write to the same row simultaneously, exactly like concurrent requests would (Backend Races).
- A user sends a second message while the first plan is mid-flight, producing two agents acting on the same state (Optimistic Concurrency).
- A cancelled or timed-out request may leave tools still executing, so side effects continue after the response is gone.
Security
- Every tool argument originates, directly or indirectly, from text the model read — which may include a document, a web page or a database row an attacker controls. Arguments are untrusted input (Every Input Surface).
- The agent must act with the user's permissions, not the service's. A tool executing under a service credential is a confused deputy by construction (Agent Authorization).
- Tool outputs re-enter the prompt. A tool that returns user-generated content is an injection channel into the next model turn (Indirect Prompt Injection in the Agentic AI domain).
- A tool that fetches a URL the model supplied is server-side request forgery with extra steps (SSRF — When the Backend Fetches a URL).
- Prompts, tool results and traces routinely contain personal data and secrets; the observability pipeline for an agent is a data-protection surface (Secrets in Logs).
Misreads
- "The model decides what to do, so validation is the model's job." The model is a client. Clients do not enforce anything (The Trust Boundary).
- "Agents are a new kind of system." The runtime is a normal service calling a slow external API in a loop. The new parts are nondeterminism, latency variance, cost and partial plans — the rest is your existing backend.
- "Nondeterminism means we cannot test it." You can test tools deterministically, assert on tool selection with fixed inputs, and evaluate trajectories statistically (Deterministic Evaluators in the Agentic AI domain).
- "We need a framework." A loop, a schema and a dispatch table are enough for many services; a framework is a choice with its own cost (Option 0: No Framework in the Agentic AI domain).
- "It is just an LLM feature, the backend is unchanged." The backend gained an unbounded loop, a per-request cost, a new untrusted client and a new class of partial failure.
Operating it
How you see it in production
- A trace per request with a span per model call and per tool call, so the trajectory is visible as a waterfall (Tracing From the Backend's Side).
- Steps per request, as a distribution. A rising tail is a loop that is failing to converge.
- Tokens in and out, and cost, per request and per tenant — the metric with no analogue in a conventional backend.
- Tool call counts, error rates and latency per tool; the slow tool decides the request's latency.
- Termination reason: answered, step limit, deadline, budget exhausted, error. This single label explains most of what happens in production.
- Model provider latency and error rate as an external dependency like any other (Calling Something You Do Not Control).
What changes at 10x and 100x
- Cost scales with usage in a way infrastructure does not: doubling traffic doubles the model bill immediately and visibly, which changes what "scaling" conversations are about.
- Provider rate limits become a hard ceiling you do not control; queueing and shedding move upstream of your own capacity (Rate Limiting).
- Fan-out to internal services multiplies: an agent that calls five tools per request puts five times the load on the services behind them (Cascading Failure).
- At small scale, a synchronous request with a deadline is fine. Past that, the loop belongs in a job system because holding HTTP connections for a minute does not scale (Background Jobs).
What this costs
- You trade specified behaviour for flexibility. A workflow you can enumerate is cheaper, faster, testable and more predictable — many agent features would be better as a form (When Planning Helps in the Agentic AI domain).
- Bounding the loop makes it terminate and makes some legitimate long tasks fail. Budgets are a product decision, not only a technical one.
- Full trajectory logging is what makes agents debuggable and is expensive and privacy-sensitive at volume (Agent Audit Logs).
- Running tools with user permissions is correct and means the agent can only do what the user could — which sometimes disappoints the person who wanted it to be more capable than they are.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe shape — runtime, model, tools, loop — holds across providers and frameworks.
- SIMPLIFIEDDrawn as one agent with a flat tool list. Real systems add retrieval, memory, planning and sometimes multiple agents; those belong to the Agentic AI domain, and none of them change the backend obligations described here.
- SCALE-SPECIFICBelow modest volume, a synchronous endpoint with a hard deadline is a reasonable implementation. Above it, the same design exhausts connections and worker slots, and the loop has to move into a job system.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
Securityprompt-injection
Agenticthe-agent-loopsingle-agentwhat-agentic-engineers-buildtool-calling-basicsarchitecture-tradeoffsfailure-scenarios
Domains that do not exist yet
- — Distributed Systems — a multi-step plan with side effects is a distributed transaction without a coordinator, and the compensation story is the same one sagas address.