Hardware Threads Are Not OS Threads
A §224 distinction the whole concurrency stack rests on. A hardware thread is a fixed execution context built into silicon. An OS thread is an allocated software object. The OS multiplexes many of the second onto few of the first, and every scheduling cost you can measure lives in that mapping.
Software view, hardware view
The gap between what you wrote and what the machine does is where this whole domain lives.
Two different kinds of object
The clearest way to hold the distinction: hardware threads are capacity, software threads are demand. Capacity is fixed when the chip is manufactured. Demand is whatever your program creates. When demand exceeds capacity, the scheduler time-slices, and time-slicing is not free.
A software thread that is not currently mapped onto a hardware thread is not slow — it is *not executing at all*. It occupies memory for its stack and a slot in a run queue. When the scheduler selects it, its saved registers are loaded into a hardware context and it resumes. That load-and-resume is a context switch, and its direct cost is small; its indirect cost, discussed in Cache Warmth and the Real Cost of Migration, is often much larger.
| Property | Hardware thread | OS thread |
|---|---|---|
| Nature | Physical register set inside a core | Memory allocation plus kernel metadata |
| How many exist | Fixed by the silicon | Thousands; bounded by memory and policy |
| Created by | Manufacturing | A runtime call, at any time |
| When it executes | Whenever the core issues from it | Only while mapped onto a hardware thread |
| Switching mechanism | Hardware interleaving, near-free | Kernel context switch, save and restore |
| Represents | Execution capacity | Demand for execution capacity |
| Adding more gives you | Nothing — you cannot add any | More contention, not more throughput |
What oversubscription actually costs
Once runnable software threads outnumber hardware threads, every additional one adds cost without adding capacity. Each switch saves and restores register state, and — far more expensively — the incoming thread arrives on a core whose caches hold the *outgoing* thread's data. It then takes cold misses to rebuild a working set that already existed somewhere.
This is why a thread pool sized far above the hardware thread count can perform worse than a small one on identical work. The threads are not idle; they are actively evicting each other from cache. The mechanism is hardware, the symptom is a scheduling curve, and the fix is a smaller pool.
It is also why "just add more threads" fails as a response to a CPU-bound bottleneck, while sometimes working for an I/O-bound one — in the I/O case the threads are mostly blocked and never contend for a hardware thread at all.
Reading the distinction in real decisions
Three common decisions become straightforward once the two are kept apart. Pool sizing: bound CPU-bound pools near the hardware thread count, because past it you are buying switches. Blocking calls: a blocked software thread holds memory but no hardware thread, which is why thousands of blocked threads are affordable and thousands of runnable ones are not. Affinity: pinning constrains which hardware thread a software thread maps onto, which is meaningless unless you understand that a mapping exists.
The OS side of all this — run queues, scheduling classes, preemption policy — is the Operating Systems domain's subject; see Context Switching and Scheduling Simulator: FCFS, Round Robin, Priority. What belongs here is the hardware fact that makes it necessary: there are very few register sets, and they are the only place execution happens.
- Blocked threads are cheap. They hold a stack, not a hardware context. Thousands are fine.
- Runnable threads are expensive past the hardware thread count. Each one adds switching and cache disruption.
- A context switch's real cost is mostly cache, not register saving — which is why it varies so much between workloads.
- Affinity only makes sense as a constraint on the mapping, which requires knowing the mapping exists.
Key points
- Hardware threads are fixed execution capacity; OS threads are allocated demand for it.
- A software thread executes only while its state is loaded into a hardware context — otherwise it is not slow, it is stopped.
- Oversubscription costs switching plus, more importantly, the cache working set the incoming thread has to rebuild.
- Blocked threads cost memory but no hardware context, which is why thousands of them are affordable.
- Context-switch cost is dominated by cache effects, not by register save and restore.
Follow the mechanism
The path through the machine, hop by hop — and the conclusions it invites that are wrong.
- 1Program → kernel: thread creation allocates a stack and a scheduling entity; no hardware context is committed.
- 2Run queue → scheduler: the thread becomes runnable and waits for a hardware thread to become available.
- 3Scheduler → hardware context: saved registers are loaded into a core's register set and execution resumes.
- 4Core → caches: the thread begins missing on data the previous occupant evicted, rebuilding its working set.
- 5Timer interrupt → scheduler: the slice expires, state is saved back to memory, and another thread takes the context.
- • "Threads are cheap, so more is better" — blocked threads are cheap; runnable ones past capacity are not.
- • "A context switch is a microsecond" — the direct cost is small, the cache rebuild usually is not.
- • "The thread is running slowly" — an unscheduled thread is not running at all, which is a different problem with a different fix.
- • "Thread count should match the workload's concurrency" — it should match the hardware's capacity for *runnable* work.
Consequences, controls and cost
- • Large thread pools on CPU-bound work perform worse than small ones, despite doing identical work.
- • Context-switch cost measured in microbenchmarks understates production cost, because the benchmark has no working set to lose.
- • Thousands of blocked threads are affordable while a few hundred runnable ones are not.
- • Latency variance rises with oversubscription as threads wait for a hardware context.
- • Bound runnable CPU-bound threads near the hardware thread count; use queues rather than threads for excess work.
- • Prefer blocking or asynchronous designs for I/O concurrency, where threads do not hold hardware contexts.
- • Where latency matters most, reduce migration with affinity so threads keep their cache state.
- • Measure context switches per second alongside throughput — a rising switch rate with flat throughput is oversubscription.
- • Track context switches per second against throughput; divergence indicates threads fighting over contexts.
- • Track involuntary context switches specifically — those are preemptions caused by oversubscription.
- • Compare cache miss rate at low and high thread counts on the same total work; a rise is threads evicting each other.
- • Sweep pool size and plot throughput and p99; the peak is usually well below the maximum you can create.
- • Small pools reduce switching but can leave hardware idle when threads block unexpectedly.
- • Async designs avoid oversubscription but move complexity into the program and complicate debugging.
- • Affinity preserves cache warmth at the cost of scheduler flexibility under imbalance.
Scope
§224 — what these claims are specific to.
- GENERALThe capacity-versus-demand distinction holds on every platform with preemptive scheduling. What varies is switch cost and scheduler policy, both OS-specific.
- PLATFORM-SPECIFICContext-switch cost, preemption granularity and scheduling classes are properties of the OS and its configuration, not of the CPU.
Misconceptions
Where the rest of this lives
Whether to use threads, an event loop or coroutines is a concurrency design decision. This lesson supplies the hardware constraint it has to respect: execution contexts are few and fixed, everything else is bookkeeping.