Find the Bottleneck
Twelve sets of production signals with no labels attached. Read them, name the constraint, then check your reasoning — arriving at the right answer for the wrong reason does not transfer to the next incident.
Challenges
Signals → candidate diagnoses → reasoning → what the evidence rules out.
“Support says the product feels sluggish. It started sometime this week — nobody can say exactly when. The infrastructure channel has been pointing at a CPU graph that is "basically flat at 100%" for two days.”
“The inventory service started timing out during the afternoon peak. It clears up by evening. The on-call engineer says "all the graphs are green" and has been restarting instances, which helps for about ten minutes.”
“The session service gets OOM-killed roughly every day and a half. It has been happening for about three weeks. Someone doubled the memory limit two weeks ago, which changed the interval from 18 hours to 36 and was declared a fix.”
“The reporting API is slow, and the application team and database team disagree about whose problem it is. The app team points at database spans of nearly a second. The DBA points at a dashboard showing the database at 30% CPU and says there is nothing to fix.”
“At 09:12 the product API started returning errors. It recovered on its own around 09:40. Nobody deployed anything. The team lead wants to know whether this will happen again, and the incident channel has settled on "the database fell over".”
“Order confirmation emails are going out hours late. The queue depth chart has been climbing since about 06:00 and is now at 2.1 million. The team is in a thread arguing about how many workers to add, and someone has already scaled the worker pool from 40 to 120 with no effect.”
“European customers say the dashboard is slow. American customers say it is fine. The team has checked the EU deployment thoroughly and found nothing wrong — CPU, memory and database metrics are all healthy in both regions, and the EU instances are the same size as the US ones.”
“The payments dependency started returning errors at 11:20. Our team enabled aggressive retries at 11:35 to improve reliability. By 11:50 the dependency was almost entirely unavailable and our own error rate was worse than before the retries were turned on.”
“A weekly performance review flagged that p99 on the search API has gone from 380 ms to 810 ms over the past two weeks. p50 has not moved. Nobody reported an incident and error rates are unchanged. The team is not sure whether this is worth investigating.”
“The team lead noticed that the order history page takes about two seconds to load and asked someone to "look at the database". The database team reports that every query on that endpoint runs in single-digit milliseconds and there are no slow queries logged at all.”
“A trading dashboard service has "latency spikes". Users describe it as the page freezing briefly and then catching up. It has been happening since a release ten days ago that added a new market-data enrichment step. Average latency looks fine and nobody can reproduce it on demand.”
“Sales say the demo "feels sluggish" on customer laptops, though it is fine in the office. The backend team has checked and the API p95 is 78 ms. Someone suggested adding a cache to the API.”