Diagnosis challenges
Twelve sets of counters and code with the hardware cause unlabelled. Read the evidence, name what the machine is doing, then check your reasoning — the right answer for the wrong reason does not transfer to the next machine.
Challenges
Counters → candidate causes → reasoning → what the evidence rules out.
“We swapped a vector for a linked list so insertions in the middle would be cheap. Insertions did get cheaper. But the nightly aggregation pass, which just walks the whole thing and sums a field, went from about 40 ms to nearly 400 ms on the same data. Nobody changed the loop.”
“We parallelised our event counter across worker threads. Each worker has its own counter, no locks, no shared state. One thread does about 40 million events a second. Four threads do about 9 million. Total. We have checked three times that the workers never touch each other's counters.”
“Our image convolution has a column-summing pass. It runs fine on most images. On one specific size it takes about six times longer, and the sizes either side of it are fine. Someone suggested it is a coincidence in our test data, but it reproduces every time on exactly that width.”
“A colleague claims that sorting our data before the filter loop makes the loop three times faster, even counting the sort. That makes no sense — sorting is strictly more work and the loop does exactly the same comparisons either way. One of us is measuring wrong.”
“Our aggregation job pins every core at 100% CPU. We moved it to instances with twice the cores and it got about 8% faster, not twice. Finance is asking why we doubled the bill for 8%. Someone wants to try the highest-clocked instance type next.”
“A lookup-heavy service degraded when we grew the in-memory index from about 200 MB to about 6 GB. That was expected to cost something, but the index nodes are small and hot, and our cache hit rate is still good. The slowdown is much worse than the cache numbers suggest it should be.”
“We added one boolean flag to a particle struct and memory use for the particle array jumped by about 30%. One boolean. The array is 20 million elements and we are now over our memory budget. Someone thinks the language runtime is boxing the field.”
“Our graph processing job scales beautifully to about 16 threads on a developer workstation. On the big two-socket production server it barely improves past 16 and gets slightly worse past 40. The production machine has four times the cores. We think our parallel decomposition must be wrong.”
“We rewrote a numeric kernel expecting a four-times speedup from vectorization. We enabled optimisation, the arrays are contiguous floats, the loop body is one multiply and one add. We got about 4% and cannot work out why the CPU is not using its vector units.”
“We moved our image filter to the GPU. The kernel benchmarks at 0.8 ms against 11 ms on the CPU — a 14× win. But the endpoint that calls it got slower, not faster. The GPU is clearly doing its job, so we assume the overhead is somewhere in our web framework.”
“Our new vectorized encoder benchmarks 35% faster than the old one. In production it is about 4% faster, and on the busiest nodes it is slower. The benchmark runs the same input on the same instance type. We have run it dozens of times and it is consistent.”
“Our physics step reads one field from each particle to build a bounding box. It is 20 million particles and it takes far longer than it should for what is essentially a min/max scan. Profiling says memory-bound, but we are only reading 4 bytes per particle — that is 80 MB, which should stream in no time.”