expert
The Data Fits in Cache and It Is Still Slow
Read the counters before the options. Nothing here is labelled with the answer.
The report
A lookup-heavy service degraded when we grew the in-memory index from about 200 MB to about 6 GB. That was expected to cost something, but the index nodes are small and hot, and our cache hit rate is still good. The slowdown is much worse than the cache numbers suggest it should be.
The lookup path — random probes into a large index
// index: ~6 GB, hash-distributed across the region
for query in batch:
slot = hash(query) % index_slots
node = index[slot] // random location in a 6 GB region
result = probe(node, query)CountersSIMULATED
| IPC | 0.31 (was 1.44 at 200 MB) | Per-cycle progress dropped by roughly a factor of four. |
| L1-dcache-load-misses | 6.8% of loads | Most loads are still served by the first-level cache. |
| LLC-load-misses | 11.2% of last-level accesses | The large majority of accesses reaching the last level find their data there. |
| dTLB-load-misses | 38.4% of loads (was 0.3%) | More than a third of loads fail to find a translation cached. |
| dtlb_load_misses.walk_active | 41% of cycles | Address translation hardware is busy for a large share of all cycles. |
| page-faults | negligible after warm-up | Pages are resident; nothing is being brought in from storage. |
What is the hardware doing?