SchedulingAdvanced

The real cost of a context switch

“What does a context switch actually cost, and why is the answer "more than saving the registers"?”

What this tests

  • What is saved and restored, and where
  • Direct cost vs the indirect cost of cold caches and TLB
  • Thread switch vs process switch vs mode switch
  • How to measure switches and recognise when they are the problem

Answers by level

Read the beginner answer first and notice what is missing.

The direct work is: enter the kernel (a timer interrupt, a blocking syscall, or a wake-up of a higher-priority task), save the running task’s registers and stack pointer into its kernel-side state, run the scheduler to pick the next task, switch kernel stacks, and — if the next task belongs to a different process — load a new page-table root (CR3 on x86). Then restore and return. That is roughly 1–5 µs on a modern server, more if the scheduler has a long run queue to consider (Context Switching).

The indirect cost is usually larger. The task that resumes finds the L1/L2 caches full of someone else’s lines and, if the address space changed, a TLB that has been flushed or whose entries belong to another process (PCID/ASID tagging lets the hardware keep them but they still compete for the same slots). The next few thousand memory accesses miss; the branch predictors are cold. A switch that "costs 2 µs" can degrade the next 50–100 µs of execution (The TLB).

That is why the *kind* of switch matters. A switch between two threads of the same process does not change CR3, so no TLB flush. A system call is a mode switch, not a context switch: the same task continues in kernel mode, on the order of 100 ns–1 µs, though Spectre/Meltdown mitigations made it more expensive (System Calls). Green threads and goroutines switch in user space — a few hundred nanoseconds — because they skip the kernel entirely.

Switches are also either voluntary (the task blocked) or involuntary (preempted). A process doing 50,000 involuntary switches a second on one core is being time-sliced against too many competitors; 50,000 voluntary switches a second usually means a lock or a tiny I/O pattern is bouncing threads between sleep and wake.

Green flags · Red flags

Strong green flag · Says the indirect cost dominates and explains why goroutines/green threads are cheap to switch.
Green flags
  • Separates direct save/restore cost from cache and TLB pollution
  • Knows a same-process thread switch avoids the address-space change
  • Distinguishes a syscall (mode switch) from a context switch
  • Names a way to measure switches and interprets voluntary vs involuntary
  • Gives plausible numbers (~µs direct, tens of µs indirect)
Red flags
  • Thinks a system call is a context switch
  • Cannot say why switching threads is cheaper than switching processes
  • Gives "milliseconds" for the direct cost

Follow-up questions

F1
Why does PCID (or ASID on ARM) help?
F2
Your service shows 200k context switches/s on 8 cores. First two things to check?
F3
Why is a goroutine switch cheaper than a thread switch?

Scenario

A queue consumer was reconfigured from 8 to 64 threads on an 8-core box to "process faster". Throughput went down 15% and vmstat shows cs went from 4k to 180k. Explain and recommend.

Learn this topic