Reliabilityreliabilityfallbackcachingroutingtimeouts

Fallbacks, Caching and Model Routing

Provider fallback chains keep the product up, prompt caching and result caching cut cost and latency, and routing by difficulty sends each request to the cheapest model that can handle it.

Interview question
Progress

Provider and model fallback chains

Model providers have outages, rate limits and latency spikes like any dependency, and a product that depends on a single one inherits its availability. A fallback chain is an ordered list of (provider, model) pairs; on a retryable failure (429, 5xx, timeout) the call moves to the next entry. The chain is configured, not hard-coded, so an incident response is a config change.

Fallbacks are not free. A different model has different tool-calling behaviour, different output formats and different prices. Keep prompts model-agnostic where possible, run your smoke evals against every model in the chain (Regression Gates and Online Evaluation), and record which model served each request in the trace so quality regressions during a fallback are visible.

  • Retryable: 429, 500–504, connection reset, timeout. Retry the same provider once with backoff, then fall back.
  • Not retryable: 400 (bad request), 401/403, content policy refusals. Fail fast; a fallback will fail the same way.
  • Circuit breaker: after N failures in a window, skip the provider for a cool-down period instead of paying the timeout every request.
  • Degraded mode: if the whole chain fails, return a cached or static answer and record the outage.

Caching: prompt prefixes and results

Prompt (prefix) caching is provider-side: if the first N tokens of a request match a recent request, the provider reuses the computed key-value state and charges less for those tokens with lower time-to-first-token. To exploit it, put stable content first (system prompt, tool definitions, reference documents) and variable content last (conversation, the current query). Reordering a tool definition or injecting a timestamp at the top of the prompt breaks the prefix and silently doubles your cost (Context Ordering & Lost in the Middle).

Result caching is on your side: an exact-match cache keyed on the full normalised request returns a stored response without a model call. It works well for deterministic tool calls and for repeated identical queries (health checks, FAQ traffic). An LRU eviction policy is the usual choice; the DSA transfer is direct.

Semantic caching and its pitfalls

Semantic caching keys on an embedding of the query and returns a cached answer when a new query is within a cosine-similarity threshold. It sounds like free savings and is one of the easiest ways to ship wrong answers. “Cancel my order 4471” and “cancel my order 4417” embed almost identically. “What is the refund policy?” and “what is not covered by the refund policy?” are close in embedding space and opposite in meaning.

If you use it at all, restrict it to read-only, user-independent, low-stakes queries; include the user or tenant id in the key; set a tight threshold (0.95+) and measure the false-hit rate with a labelled set; and never cache anything that depends on live data or per-user state.

Routing by difficulty, timeouts and hedging

Model routing sends each request to the cheapest model that meets the quality bar. A small classifier or a set of heuristics (query length, presence of code, tool-use required, conversation depth) picks between a fast small model and a capable large one. A common pattern is cascade routing: try the small model, and escalate to the large one only if a confidence signal or a validator rejects the answer (Confidence Thresholds). Measure the routing decision quality with evals; a router that sends 30% of hard queries to the small model is a quality regression dressed as a cost saving.

Timeouts are set per call and per request: a model call gets a timeout tied to expected output length; a tool call gets one tied to its dependency; the request gets a deadline that all of them inherit. Hedged requests issue a second call to a fallback after the p95 latency has elapsed and take whichever returns first; they cut tail latency at the cost of extra spend, so reserve them for latency-sensitive paths and cancel the loser.

A fallback chain with a per-provider circuit breaker and a hedged second call after the p95.
1type Route = { provider: string; model: string }
2const chain: Route[] = [
3  { provider: 'primary', model: 'large-v2' },
4  { provider: 'secondary', model: 'large-equivalent' },
5  { provider: 'primary', model: 'small-v2' },   // last resort: cheaper, degraded quality
6]
7const breaker = new CircuitBreaker({ failures: 5, windowMs: 60_000, cooldownMs: 30_000 })
8 
9export async function complete(req: Request, deadlineMs: number) {
10  for (const route of chain) {
11    if (breaker.isOpen(route.provider)) continue
12    try {
13      const p95 = latency.p95(route)
14      const primary = callModel(route, req, { timeoutMs: deadlineMs })
15      const hedged = sleep(p95).then(() => callModel(route, req, { timeoutMs: deadlineMs - p95 }))
16      const res = await Promise.any([primary, hedged])
17      trace.set('model.route', route)
18      return res
19    } catch (e) {
20      if (!isRetryable(e)) throw e
21      breaker.recordFailure(route.provider)
22    }
23  }
24  return degradedResponse(req)                      // cached or static; recorded as an outage
25}

Key points

  • A configured fallback chain with a circuit breaker keeps the product up through provider outages; record which model served each request.
  • Prompt caching depends on a stable prefix: stable content first, variable content last, no timestamps at the top.
  • Exact-match result caching with LRU eviction is safe; semantic caching returns wrong answers for near-identical queries unless tightly constrained.
  • Route by difficulty with a cascade and measure routing quality; a cheap router can be an expensive quality regression.
  • Per-call timeouts inherit a request deadline; hedged requests cut tail latency at extra cost.

When to use — and when not to

Use it when
  • Any customer-facing product with an availability target.
  • High-volume systems where the stable prefix is a large share of tokens.
  • Mixed workloads where most requests are easy and a few need the large model.
Avoid it when
  • Do not add a fallback model you have never evaluated; an untested fallback is an outage with extra steps.
  • Do not semantic-cache anything with per-user state, live data, or side effects.
  • Do not hedge every request; it doubles cost on the paths where latency does not matter.

Failure modes

  • Fallback engages during an outage and quality drops for hours because nobody is looking at the model label in traces.
  • A timestamp in the system prompt breaks prefix caching and cost doubles silently.
  • Semantic cache returns the refund answer for a “not covered by refund” question.
  • Router sends hard queries to the small model; task success drops 15% and finance celebrates.
  • Hedged requests are never cancelled and double the provider bill.

Tradeoffs

Complexity
low → high
Latency
low → high
Cost
low → high
Reliability
poor → strong
Debuggability
hard → easy

Fallbacks and caching complicate debugging: always record the route and cache hit status on the span.

← Budgets, Limits and Termination