How can retries make an outage worse?
“A payment provider slows down. Ten minutes later it is completely down and so is your checkout. Your services all retry on failure. Explain what happened and how to retry safely.”
What this tests
- Retry amplification across layers as multiplication, not addition
- Timeouts, retry budgets, backoff with jitter, and where retries belong
- Circuit breakers as the mechanism that stops the storm
- Distinguishing a slow dependency from a failed one
Answers by level
Read the beginner answer first and notice what is missing.
Retries multiply through layers. Client retries 3×, gateway retries 3×, order service retries 3× against the payment service: one user click becomes up to 27 calls to a provider that was already slow. The slowdown becomes an overload, the provider fails harder, more calls time out, more retries fire. That is a retry storm, and it turns a degradation into an outage — with your own traffic.
Safe retrying: retry in one layer, not every layer; retry only idempotent operations and only on errors that a retry can fix; use exponential backoff with jitter; cap with a retry budget (for example, retries may be at most 10% of the request rate) so a broad failure cannot amplify. And put a circuit breaker in front of the dependency so that once the failure rate crosses a threshold, calls fail fast for a cooling period instead of piling up — see Circuit Breaker and Reliability Patterns.
Green flags · Red flags
- Computes the amplification factor (3 × 3 × 3 = 27×)
- Retries in one layer only, with backoff + jitter and a retry budget
- Timeouts shorter than the caller's; mentions thread-pool exhaustion
- Circuit breaker that counts slow calls, not only errors
- A concrete degraded mode for checkout
- "Just increase the timeout so the retries have time to succeed."
- Retries at every layer "for safety"
- Retries non-idempotent calls without a key
- Fixed 1 s retry delay with no jitter