Cache Warmth and the Real Cost of Migration
The expensive part of a context switch is not saving registers. It is that the thread resumes on a core whose caches and TLB hold someone else's data, so it must take a burst of cold misses to rebuild a working set that existed perfectly well a moment ago somewhere else.
Software view, hardware view
The gap between what you wrote and what the machine does is where this whole domain lives.
Two costs, very different sizes
The direct cost is saving one register set and loading another, plus scheduler bookkeeping. It is small, roughly fixed, and it is what microbenchmarks measure — which is exactly why microbenchmarks report context switches as cheap. A benchmark that switches between two threads doing nothing has no working set to lose, so it measures the direct cost and nothing else.
The indirect cost is the cold misses the resumed thread takes rebuilding its working set: data in L1 and L2, translations in the TLB, and on a migration possibly a different NUMA node entirely. This cost is not fixed at all. It is proportional to how much state the thread had and how much of it was destroyed, so it ranges from negligible for a thread that touches almost nothing to very large for one with a substantial working set.
The gap between these two is one of the more common measurement traps in systems work, and it is a specific instance of the general problem in Every Way a CPU Microbenchmark Lies: the benchmark removed the thing that was expensive.
Why migration is worse than a plain switch
A switch that resumes a thread on the *same* core may find some of its state surviving — if the intervening thread did not evict everything, part of the working set is still there and the thread warms up quickly. This is why short slices with a small number of threads per core can be relatively cheap.
A migration to a different core has no such luck. The new core's private caches contain nothing relevant, its TLB has none of the thread's translations, and the data is warm somewhere the thread can no longer reach cheaply — worse, some of it may now require coherence transfers from the old core. On a multi-socket machine the memory itself may now be remote, adding the NUMA: Not All Memory Is Equally Far cost on top for the lifetime of the thread's stay.
This is the mechanism that makes Thread Affinity: Pinning and Its Price a tail-latency tool. Migrations are unpredictable, and each one injects a burst of stall cycles into whatever request the thread happened to be serving. The mean barely moves; the tail does.
| Scenario | L1/L2 state | TLB entries | Typical impact |
|---|---|---|---|
| Same core, brief interruption | Mostly survives | Mostly survive | Small — quick to warm back up |
| Same core, long slice by another thread | Largely evicted | Largely evicted | Moderate — a rebuild burst |
| Migration to a sibling core | Lost — different private caches | Lost | Large; some data reachable via coherence |
| Migration across NUMA nodes | Lost | Lost | Largest — plus remote memory for the duration |
Where it shows up, and what to do
Three familiar symptoms trace back to this. Oversubscribed thread pools: more runnable threads than hardware contexts means more switching and more mutual eviction, so throughput falls even though the work is unchanged. Tail latency spikes with no corresponding code path: a request that happened to span a migration pays a stall burst that a request on a quiet core does not. Noisy-neighbour effects in shared environments: another tenant's thread on a sibling logical CPU evicts your working set from the shared L1 and L2, and nothing in your process is responsible.
The controls follow directly and are mostly about reducing switch frequency rather than switch cost: bound runnable threads near the hardware thread count, use affinity for the threads whose tail matters, and prefer batching so that a thread does more work per scheduling opportunity. The last one is underrated — a thread that processes fifty items per wake amortises one warm-up over fifty items instead of paying it per item.
- Bound runnable threads near the hardware thread count; excess work belongs in a queue, not in more threads.
- Batch per wake. Amortising one cache warm-up over many items is often a larger win than any micro-optimisation.
- Pin the latency-critical few to stop unpredictable migrations from landing inside a request.
- Expect noise in shared environments. A sibling tenant can evict your working set and you cannot prevent it.
Key points
- A context switch has a small fixed direct cost and a large variable indirect cost from lost cache and TLB state.
- Microbenchmarks measure the direct cost only, because a thread doing nothing has no working set to lose.
- Migration to a different core is far worse than a same-core switch: nothing survives, and memory may become remote.
- This is the mechanism behind oversubscription penalties, migration-induced tail latency, and noisy neighbours.
- Reducing switch *frequency* — bounded pools, batching, selective affinity — beats trying to make switches cheaper.
Follow the mechanism
The path through the machine, hop by hop — and the conclusions it invites that are wrong.
- 1Timer or preemption → scheduler: the running thread's registers are saved and another thread is selected.
- 2Intervening thread → caches: it fills L1, L2 and TLB with its own data, evicting the previous occupant's.
- 3Resumed thread → cold caches: the original thread restarts and misses on data that was resident moments earlier.
- 4Miss burst → memory hierarchy: those misses walk L2, LLC and possibly DRAM, stalling the thread repeatedly.
- 5On migration → different core: private caches and TLB hold nothing relevant, and coherence or NUMA costs may apply.
- • "A context switch costs about a microsecond" — that is the direct cost; the cache rebuild is usually larger.
- • "The benchmark says switching is cheap, so oversubscription is fine" — the benchmark had no working set to lose.
- • "Latency spikes must be GC or the network" — an unlucky migration produces the same shape and leaves no trace.
- • "My container has a CPU limit, so it just runs proportionally slower" — throttling deschedules, and each restart is cold.
Consequences, controls and cost
- • Context-switch cost that varies by orders of magnitude between workloads and between benchmarks and production.
- • Oversubscribed pools performing worse than smaller ones on identical work.
- • Tail latency spikes on requests that happened to span a migration.
- • Performance variance in shared or containerised environments that no application change explains.
- • Bound runnable threads near the hardware thread count, queueing excess work rather than creating more threads.
- • Batch work per wake so one warm-up is amortised over many items.
- • Use affinity for the small set of latency-critical threads to remove unpredictable migrations.
- • In containers, check whether CPU limits are causing throttled descheduling, which forces the same cold restart.
- • Track voluntary and involuntary context switches separately; involuntary ones indicate preemption from oversubscription.
- • Track thread migrations per second, not just switches — migrations carry the larger cost.
- • Correlate latency outliers with migration events to confirm the mechanism rather than assuming it.
- • Compare cache miss rate immediately after a switch against steady state; the burst is the indirect cost made visible.
- • Bounded pools reduce switching but can underuse hardware if threads block unexpectedly.
- • Batching amortises warm-up at the cost of increased latency for the first item in each batch.
- • Affinity removes migrations but reduces the scheduler's ability to balance load.
Scope
§224 — what these claims are specific to.
- PLATFORM-SPECIFICMigration policy, slice length and rebalancing aggressiveness are OS and configuration properties; container CPU quotas add forced descheduling at period boundaries with the same effect.
- SIMPLIFIEDThe model treats caches as fully lost on migration. In practice some data is reachable via coherence from the previous core, and shared LLC contents may survive — which reduces but does not remove the cost.
Misconceptions
Apply it
Where the rest of this lives
The concurrency argument is about queueing and resource limits. The hardware argument is here: past the hardware thread count, extra runnable threads mostly evict each other's cache state, so they add interference rather than throughput.