Reliability Engineering
Failure scenarios and their mitigations: retries, fallbacks, limits, budgets, caching.
Production agents fail in a small number of recurring ways; each has a known mitigation, and a reliable system is one that has a planned response for all seven before launch.
Each of the seven production failure scenarios has a characteristic signal in traces and metrics and a concrete runbook; knowing both turns a 3 a.m. page into a ten-minute fix.
An agent loop must be bounded by hard limits on steps, tokens, cost and wall-clock time, with loop detection and a graceful degradation path, all enforced in code outside the model.
Provider fallback chains keep the product up, prompt caching and result caching cut cost and latency, and routing by difficulty sends each request to the cheapest model that can handle it.