Architecture Tradeoffs
A side-by-side of single agent, agent + RAG, router, workflow, supervisor, and multi-agent on complexity, latency, cost, reliability, and debuggability — and the handful of questions that decide between them.
The ladder, restated
Every architecture in this module is a response to a specific failure of the one below it. The ordering from Choosing the Right Abstraction — plain code → single LLM call → structured output → tool calling → RAG → workflow → agent → multi-agent — is not a maturity curve to climb. It is a cost curve: each rung adds latency, cost, and debugging surface, and you take a rung only when a measurement on the current one says you must.
The six shapes compared here are the ones you will actually choose between in a design review. Two of them (workflow and router) are mostly deterministic and belong lower on the ladder than "agent"; two (supervisor and general multi-agent) are at the top and should be rare.
Comparison
Ratings are 1–5. For complexity, latency and cost, 1 is low. For reliability and debuggability, 5 is best. They assume each architecture is well-built and measured; an unmeasured system of any shape is reliability 2.
- Single agent — complexity 1, latency 2, cost 2, reliability 3, debuggability 5. One loop, one context, ≤ 15 tools; linear trace; degrades with step count and tool count. See Single Agent.
- Agent + RAG — complexity 3, latency 3, cost 3, reliability 4, debuggability 4. Adds an index and ingestion pipeline; grounded answers with citations; new failure class of relevant-but-wrong retrieval. See Agent + RAG.
- Router — complexity 2, latency 2, cost 2, reliability 4, debuggability 5. Cheap classifier plus heterogeneous handlers; most traffic never touches a large model; fails on multi-intent and overlapping labels. See Router Architecture.
- Workflow (state graph) — complexity 3, latency 2, cost 2, reliability 5, debuggability 5. Typed state, fixed graph, capped retries, checkpoints; predictable p95 and cost; rigid when the task shape changes. See Workflow State Graph.
- Supervisor — complexity 4, latency 4, cost 4, reliability 3, debuggability 2. Isolated worker contexts and tool partitioning; coordination overhead of 2+ calls per delegation; serial bottleneck; needs nested traces. See Supervisor Architecture.
- Multi-agent (pipeline / hierarchical / swarm) — complexity 5, latency 4, cost 5, reliability 2, debuggability 1. Emergent coordination, peer hand-offs, shared state; the hardest to test and the easiest to over-build. See Multi-Agent Systems Overview and When Not to Use Multi-Agent.
Reading the table
Two columns matter more than the others. Debuggability determines whether you will be able to improve the system after launch; anything rated 2 or below needs a tracing investment before it ships, not after. Cost at scale is the column that kills products: a supervisor at 5× the per-request cost of a router is fine at 1,000 requests a day and a budget crisis at 1,000,000.
Note that reliability does not increase with sophistication. The workflow is the most reliable shape and one of the simplest; multi-agent is the least reliable and the most complex. Adding agents adds places to fail. The only time a higher rung improves reliability is when the lower rung is failing for a structural reason — context saturation, tool overload, privilege mixing — that the higher rung specifically removes.
Latency ratings are per request at p95. A supervisor with parallel workers can beat a single agent on wall-clock for a heavy task, but on the median request it is slower, and products are judged on the median.
The discriminating questions
Most architecture decisions collapse to a short list of questions. Ask them in order and stop at the first architecture that fits.
- Is the sequence of operations known before the request arrives? Yes → workflow; use LLM calls only for the fuzzy steps.
- Do requests fall into a few intents with different costs or risk levels? Yes → router, with each handler at its own rung.
- Do answers depend on private or changing documents? Yes → add RAG, always-on if every request needs it, as a tool if requests are heterogeneous.
- Does the task need open-ended tool use with ≤ 15 tools and ≤ 15 steps? Yes → single agent with hard caps.
- Has a single agent measurably failed on tool selection, context saturation, or privilege isolation? Yes → supervisor with compressed worker outputs.
- Do sub-tasks require peer coordination, long-running shared state, or specialists that negotiate? Only then → multi-agent, and only with hierarchical tracing already in place.
- Which architecture can you debug with the traces you have today? If the answer is "none of the ones above the current rung", stay on the current rung.
Worked example: a support bot
A support product receives order-status checks (60% of traffic), policy questions (25%), refund requests (10%), and open-ended complaints (5%). The naive design is one agent with a search tool, a database tool, and a refund tool. It works in the demo. In production it calls the refund tool on a policy question once a week, spends 20k tokens on "where is my order", and nobody can explain a given answer.
The measured design is a router. order_status is a database call and a template — zero LLM tokens, 100 ms. policy is always-on RAG with citations — one small model call. refund is a workflow with a validation step and an approval gate. complaint is a bounded single agent with a search tool, 10 steps, and no write tools. A supervisor appears nowhere: nothing in the traffic requires open-ended decomposition.
The result is cheaper by roughly an order of magnitude, faster at the median, and each path has its own eval set. That is the pattern to reach for in a design review before drawing any boxes labelled "agent": the interview question the "choose an architecture for a support bot" interview question is testing precisely this reasoning.
Tradeoffs of the synthesis itself
Composite systems inherit the worst debuggability of their parts, so a router with a supervisor behind one intent is only as debuggable as the supervisor. Keep the expensive, opaque rungs behind the rarest intents and instrument them first.
Complexity compounds across rungs even when each is simple; a router with five handlers of different shapes is five systems to evaluate. Reliability, however, tends to improve with composition because failures are confined to one path. Latency and cost become traffic-weighted averages, which is exactly why routing the common case to code pays off.
- Complexity: composite — sum of the parts; budget evaluation effort per handler.
- Latency and cost: traffic-weighted; optimise the top two intents first.
- Reliability: improves with confinement; a failing agent path does not take down the status check.
- Debuggability: bounded by the worst component on the hot path.
Key points
- Each architecture fixes a specific, measurable failure of the one below it; climb only on evidence.
- Reliability and debuggability peak at the workflow and fall toward multi-agent — sophistication is not robustness.
- Cost at scale and debuggability are the columns that decide whether a product survives.
- Ask the discriminating questions in order and stop at the first fit; most products stop at router or workflow.
- Composite designs are common and good: a router dispatching to code, RAG, a workflow, and one bounded agent.
- Never adopt a rung you cannot trace; instrument before you escalate.
When to use — and when not to
- At the start of any design review, before boxes are drawn.
- When an existing agent is expensive or unreliable and you need to decide whether to add or remove parts.
- When comparing vendor or framework claims that a more agentic design is inherently better.
- As a checklist to justify a pre-chosen architecture — the questions only help if you accept the first fit.
- When there is no traffic data yet; build the single agent or workflow, measure, then return here.
- When the real problem is retrieval quality or prompt quality — no architecture change fixes bad data.
Failure modes
- Choosing by novelty: multi-agent because it is interesting, with no failure of the simpler rung to point at.
- Rating on demo behaviour rather than on traces from representative traffic.
- Ignoring the traffic distribution and optimising the rare, interesting case instead of the common, boring one.
- Escalating to fix a symptom (context overflow) whose cause was a missing truncation step.
- Mixing rungs without confining failures, so one flaky agent path degrades every request.
Tradeoffs
Ratings for the typical composite (router + code + RAG + workflow + one bounded agent); individual rungs are rated in the comparison section.