Failure Scenarios: Detection and Runbooks
Each of the seven production failure scenarios has a characteristic signal in traces and metrics and a concrete runbook; knowing both turns a 3 a.m. page into a ten-minute fix.
Tool failure and wrong tool selection
Scenario 1: a tool fails. Signal: tool span error rate rises for one tool; retries cluster; task success drops while step count rises. Runbook: classify the error (timeout, 429, 5xx, 4xx, malformed). Transient errors get exponential backoff with jitter, capped at three attempts and a total timeout. Persistent errors get a fallback (a secondary API, a cached value, a degraded answer) and the error text is returned to the model as the tool result so it can change plan. Never retry a non-idempotent call without an idempotency key (Idempotency).
Scenario 2: the model chooses the wrong tool or arguments. Signal: wrong-tool rate on the golden set; in production, a spike in argument validation failures or in a tool being called with suspiciously uniform arguments. Runbook: inspect twenty traces where the wrong tool was called. Usually the cause is two tools with overlapping descriptions, a missing example in the description, or a parameter whose semantics the model guessed. Fix the schema and description, add the failing cases to the eval set, and re-run before shipping (Tool Schemas, Golden Datasets). If ambiguity is inherent, add an explicit router step (Router Architecture).
Bad retrieval and provider outage
Scenario 3: retrieval returns bad context. Signal: answer faithfulness drops while retrieval latency is normal; users report confident wrong answers; retrieved chunks have low reranker scores. Runbook: check the query first (is the rewritten query sensible?), then the index (was the corpus re-embedded after the model change? are new documents missing?), then chunking (are answers split across chunk boundaries?), then filtering (is a metadata filter excluding everything?). Fix the earliest broken stage; measure precision@k and faithfulness before and after (RAG Evaluation).
Scenario 4: the provider is unavailable. Signal: model span error rate or latency jumps across all requests simultaneously; 429 and 503 codes; the provider status page. Runbook: the fallback chain should already be engaging automatically (Fallbacks, Caching and Model Routing). Verify it did, verify the fallback model passes your smoke evals, and watch cost, since fallbacks are often more expensive. If no fallback exists, switch to a degraded mode (cached answers, a static message) rather than erroring for every user.
Loops, cost and latency
Scenario 5: the agent loops. Signal: steps-per-task p99 hits the limit; the same (tool, args) pair appears repeatedly in a trace; the model alternates between two tools. Runbook: the step limit should have terminated the run with a clear message. Read the trace: loops are usually a tool returning an error the model does not understand, a tool that returns nothing so the model retries, or a goal the tools cannot achieve. Fix the tool result to be actionable, add loop detection (Budgets, Limits and Termination), and add the case to the eval set.
Scenario 6: execution is expensive. Signal: cost per task drifts up over days; context tokens per step grow linearly with step number; a small number of requests dominate spend. Runbook: find the top 1% of requests by cost and read their traces. Common causes: full tool results appended verbatim, conversation history never compressed, retries multiplying a large prompt, an expensive model used for a trivial sub-task. Apply token budgets and compression (Context Selection & Compression), route sub-tasks to smaller models, and set a hard cost ceiling per request.
Scenario 7: latency is too high. Signal: p95 latency well above p50; waterfall traces show serial spans that do not depend on each other. Runbook: parallelise independent tool calls (Parallel vs Sequential Tool Calls), enable prompt caching for the stable prefix, stream tokens so the user sees progress, and route easy requests to a faster model. Set per-call timeouts so one slow dependency cannot hold the whole request.
A scenario table for the on-call engineer
Keep this in the runbook so the first responder can classify in under a minute.
- Tool errors up, one tool → scenario 1 → check dependency status, confirm retries and fallback engaged.
- Validation failures up, or eval wrong-tool rate up → scenario 2 → diff the tool schemas and prompt since last deploy.
- Faithfulness down, latency normal → scenario 3 → check index freshness and query rewriting.
- All model calls failing or slow → scenario 4 → confirm fallback chain engaged; check provider status.
- Steps p99 at limit → scenario 5 → read a looping trace; find the unhelpful tool result.
- Cost per task up → scenario 6 → top 1% traces by cost; check context growth.
- Latency p95 up, error rate flat → scenario 7 → look for serial spans and cache misses.
Key points
- Every failure scenario has a distinct signal in traces and metrics; classify before you fix.
- Tool failures need backoff, caps, fallbacks and actionable error text returned to the model.
- Wrong-tool and bad-retrieval problems are eval problems: reproduce on the golden set, fix the pipeline, re-run.
- Loops are almost always caused by an unhelpful tool result; the step limit is the safety net, not the fix.
- Cost and latency problems hide in the top 1% of requests; read those traces first.
When to use — and when not to
- Writing the on-call runbook for an agent before launch.
- Triaging a live incident or a regression in the dashboards.
- Post-mortems: map the incident to a scenario and check whether the mitigation existed.
- Do not apply every runbook at once; classify first, one scenario usually explains the incident.
- Do not fix a wrong-tool issue by adding prompt text “please use tool X”; fix the schema.
- Do not raise the step limit to “fix” a loop; find the unhelpful tool result.
Failure modes
- Tool errors are swallowed and the model sees an empty string, so it retries the same call.
- Retrieval regression after re-indexing goes unnoticed because faithfulness is not measured.
- Fallback provider engages but its prompt format differs and quality silently drops.
- Cost alert fires on a daily total, hours after a single runaway request.
- On-call has metrics but no per-scenario mapping and spends the incident guessing.