advanced
Scales to Sixteen Threads, Falls Over at Sixty-Four
Read the counters before the options. Nothing here is labelled with the answer.
The report
Our graph processing job scales beautifully to about 16 threads on a developer workstation. On the big two-socket production server it barely improves past 16 and gets slightly worse past 40. The production machine has four times the cores. We think our parallel decomposition must be wrong.
Setup and compute phases
// setup — single-threaded
graph = allocate(48 GB)
load_graph(graph) // main thread writes every page
// compute — parallel
parallel_for worker in 0 .. threads-1:
for node in partition(worker):
for neighbour in graph.edges[node]:
accumulate(neighbour)CountersSIMULATED
| scaling, 1 → 16 threads | 14.2× speedup | Near-linear improvement over the first sixteen threads. |
| scaling, 16 → 64 threads | 1.08× further speedup | Almost no additional improvement from the remaining forty-eight threads. |
| IPC at 64 threads | 0.24 (0.98 at 16 threads) | Per-cycle progress falls substantially as thread count rises. |
| LLC-load-misses | 44% of last-level accesses | Many accesses that reach the last level do not find their data. |
| local vs remote DRAM accesses | 6% local, 94% remote | Almost all memory traffic is served by a node other than the one requesting it. |
| interconnect utilisation | ≈ 91% of capacity | The link between sockets is close to fully occupied. |
Machine topology as reported at startup
node 0 cpus 0-15,32-47 memory 128 GB free 79 GB
node 1 cpus 16-31,48-63 memory 128 GB free 127 GB
node distances: 0->0 = 10 0->1 = 21
1->0 = 21 1->1 = 10What is the hardware doing?