The Dependency That Was Only Slow Until Everyone Retried
Read what each party saw, commit to a cause, and only then find out which of them was right. The root cause, why the obvious reading was wrong, and the fixes are all held back until you have answered.
The symptom
What was reported, before anyone knew what was happening.
What each party saw, and what each concluded
The evidence, in the form it actually arrived in — several parties, several partial views, several confident conclusions.
| Who | What they could actually see | What they concluded | Verdict |
|---|---|---|---|
| Recommendations service | Inbound request rate rising from 4k/s to 31k/s while its own upstream calls were unchanged. | A traffic spike or a bot. Shed nothing; scale up. | ✕ wrong |
| Product API (its caller) | p99 to recommendations crossing its 500ms deadline. Retry policy: 3 attempts, 50ms fixed delay. | Transient failures; retrying is the correct response. | ✕ wrong |
| Edge gateway (the caller’s caller) | Product API timing out. Retry policy: 2 attempts. | Same — retry. | ✕ wrong |
| On-call engineer | Traffic to recommendations 7.75× normal at the moment of failure. | We were overwhelmed by a traffic surge. | ✕ wrong |
4 of 4 parties reasoned correctly from what they could see and still reached the wrong conclusion. Nobody in this table is careless. Each one acted on complete-looking local information, and the information was local. That gap — between what a node can observe and what is true — is the whole domain, and one of these readings will usually be yours.
Commit before you read on
Real user traffic did not change during the incident. Before reading on: work out where a 7.75× multiplier comes from with three tiers, and say which single tier’s configuration you would change first.
Write it down even if you are unsure. An unwritten guess quietly becomes “that is what I thought” the moment you read the answer.