advanced
Adding Threads Made It Slower
Read the counters before the options. Nothing here is labelled with the answer.
The report
We parallelised our event counter across worker threads. Each worker has its own counter, no locks, no shared state. One thread does about 40 million events a second. Four threads do about 9 million. Total. We have checked three times that the workers never touch each other's counters.
Per-worker counters — no locks, no shared writes
struct Counters {
uint64 count[4] // one slot per worker, 8 bytes each
}
shared Counters counters
// worker w, hot loop
for event in stream:
counters.count[w] += 1CountersSIMULATED
| instructions (total, 4 threads) | ≈ 4.0× the single-thread run | Four threads retire about four times the instructions, as expected for four times the work attempted. |
| cycles (total, 4 threads) | ≈ 17.8× the single-thread run | Cycles grow far faster than instructions as threads are added. |
| IPC | 0.23 with 4 threads (1.94 with 1) | Per-cycle progress collapses as threads are added. |
| L1-dcache-load-misses | 18.9% of loads (1 thread: 0.02%) | A counter that was always resident in the single-threaded run now misses frequently. |
| LLC-load-misses | 0.3% of loads | Almost none of those misses reach main memory. |
| page-faults | unchanged | No additional memory is being mapped. |
Addresses of the four counter slots, printed at startup
counters.count[0] 0x7f3c9a001900 counters.count[1] 0x7f3c9a001908 counters.count[2] 0x7f3c9a001910 counters.count[3] 0x7f3c9a001918
What is the hardware doing?