The question this answers
I have eight cores and sixty-four runnable threads — what exactly does the extra fifty-six cost me?
An image-processing service that spawns one thread per uploaded image, each doing 200 ms of pure CPU work, receiving bursts of 64 uploads on an 8-core machine.
No application state is shared — every thread works on its own image. What *is* shared is the hardware: eight cores, the last-level cache, the memory bus and the scheduler run queue. Oversubscription is contention for resources your code never mentions.
Every submitted image is processed exactly once and the total CPU work is constant regardless of thread count. That holds — which is the point. Oversubscription changes nothing about the work and everything about how long it takes, because the invariant that fails is "useful CPU time ≈ elapsed CPU time": a growing share of the machine goes to switching and cache refills rather than to images.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
Eight cores, sixty-four threads, and where the time goes
The first-order effect is simple division and is not a problem in itself: 64 threads on 8 cores means each thread receives roughly one eighth of a core, so each 200 ms job takes about 1.6 seconds of wall clock. Total throughput is unchanged; only per-job latency has changed. If you needed all 64 done, you finish at the same time either way.
The second-order effects are the cost, and there are three. Each preemption costs a context switch — see The Cost of a Context Switch — which is small but is now happening thousands of times a second. Each resumed thread finds its working set evicted by the seven threads that ran in between, so it restarts cold and runs slower for a while. And the scheduler itself does more work: a longer run queue, more balancing decisions across cores, more timer interrupts.
Together these mean total throughput actually *falls* as thread count rises past the core count, rather than merely staying flat. The image work does not get cheaper; the machine simply spends less of itself doing it. The timeline below shows the shape: at 8 threads the cores are essentially always executing image code, and at 64 the same cores carry visible bands of switch and cache-refill overhead.
The four costs, and which one dominates
It is worth separating these because they respond to different fixes and because their magnitudes differ by orders of magnitude. The direct switch cost is small and well known. The *indirect* cost — cache and TLB state destroyed by the intervening threads — is usually several times larger and is the one people omit from their mental model.
The third cost, memory, is what turns oversubscription from slow into fatal. Sixty-four threads is fine; sixty-four thousand is an out-of-memory kill, because each thread reserves stack address space and a kernel structure whether or not it is running. This is the mechanism behind "one thread per connection stops working at a few thousand connections" — see Thread per Connection and c10k.
The fourth is the one that surprises: oversubscription makes *lock contention worse than linearly*, because a thread can be preempted while holding a lock. The remaining threads then wait not for the critical section but for the holder to be rescheduled — which is Priority Inversion without priorities, and it is why a system that is fine at 8 threads can collapse at 64 even though the lock was never the bottleneck.
| Cost | Mechanism | Rough magnitude | What reduces it |
|---|---|---|---|
| Direct switch cost | Save/restore registers, kernel entry, scheduler decision. | Roughly a microsecond per switch — small, and the one everyone quotes. | Fewer runnable threads; longer time slices; cooperative scheduling. |
| Cache and TLB pollution | The resumed thread's working set was evicted by the threads that ran in between, so it restarts cold. | Often several times the direct cost, and it scales with working-set size. Dominant for data-heavy work. | Fewer threads per core; affinity so a thread returns to a warm core. See Thread Affinity: Pinning, and What It Costs You and Parallelism Can Destroy Locality. |
| Memory footprint | Per-thread stack reservation plus kernel task structures, held whether running or not. | Megabytes of address space per thread. This is what makes very high thread counts fail rather than merely slow. | Tasks or coroutines instead of threads — kilobytes rather than megabytes. See A Task Is Not a Thread. |
| Scheduler and run-queue overhead | Longer queues, more load-balancing across cores, more timer work. | Small per event, visible in aggregate at high thread counts. | Bounded pools; pinning; avoiding thread-per-request. See Thread Pools. |
| Preempted lock holders | A thread is descheduled while holding a lock; every waiter now waits for it to be rescheduled, not for the critical section. | Can be catastrophic — turns a 500 ns critical section into a multi-millisecond one. | Bounding concurrency; never spinning under oversubscription; shorter critical sections. See Busy Waiting. |
What it looks like, and what the fix is not
The metric signature is distinctive once you know it: CPU near 100% (unlike lock contention, which reads as idle), context switches an order of magnitude above baseline, run-queue length far above core count, and throughput *lower* than at a smaller thread count. That last comparison is the one that proves it, and it requires running the experiment.
The fix is to bound the number of threads that are runnable at once, not to make threads cheaper. For CPU-bound work the bound is related to core count, but this domain refuses to give you a formula and means it: the right number depends on how much of the work is truly CPU-bound, whether hyperthreading helps this workload, what else runs on the box, and the container CPU quota — which is the one people miss most often, because the runtime typically sees the host's core count and not the cgroup limit. See Sizing a Thread Pool and concurrency-limits.
For I/O-bound work the answer is different in kind, not just in number. Sixty-four threads waiting on network responses are not oversubscribed at all — they are blocked, consuming no CPU, and the machine is fine. The failure in this lesson is specifically about *runnable* threads exceeding cores. Classifying the work first is what tells you which regime you are in; see Classifying the Work: Computing or Waiting? and cpu-bound-vs-io-bound.
threads wall_clock throughput ctx_sw/s cpu% p99_latency rss
4 3.24 s 19.8 im/s 1 200 51% 0.81 s 0.3 GB <- underused
8 1.63 s 39.3 im/s 2 400 99% 0.82 s 0.4 GB <- best throughput
16 1.69 s 37.9 im/s 31 000 100% 1.55 s 0.6 GB
64 1.94 s 33.0 im/s 248 000 100% 1.91 s 1.7 GB <- oversubscribed
512 3.10 s 20.6 im/s 1 090 000 100% 3.05 s 11.2 GB <- overhead dominant
read it as:
throughput PEAKS at ~core count for CPU-bound work and then DECLINES.
p99 latency rises monotonically with thread count from 8 upward: the
extra threads buy nothing and cost queueing. CPU stays at 100% the whole
way, so CPU utilisation cannot distinguish "working" from "thrashing" --
only throughput-vs-thread-count can.
NOTE: this is a purely CPU-bound workload. The same table for I/O-bound
work peaks far above core count, which is why there is no single formula.Key points
- Oversubscription is contention for hardware your code never mentions: cores, cache, memory bus, run queue.
- The first-order effect (each thread gets a fraction of a core) is harmless; the second-order effects — switches, cold caches, memory, preempted lock holders — are the cost.
- Cache and TLB pollution usually exceeds the direct switch cost by several times, and it is the term people leave out.
- CPU sits at 100% either way, so utilisation cannot detect it — only throughput measured against thread count can.
- It applies to *runnable* threads. Sixty-four threads blocked on I/O are not oversubscribed, which is why classifying the work comes first.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • More runnable threads exist than cores, so the scheduler time-slices them and each receives a fraction of a core.
- • Every slice boundary costs a context switch: register save/restore, kernel entry, and a scheduling decision.
- • The incoming thread finds its cache lines and TLB entries evicted by the threads that ran in between, so it executes slower until its working set is refetched.
- • Each thread holds memory — stack reservation and kernel structures — whether it is running or not, so footprint scales with thread count rather than with parallelism.
- • If any thread is preempted while holding a lock, every waiter's wait is extended by a full scheduling round rather than by the critical section.
- • 8 threads on 8 cores: each runs to completion on a warm cache; 64 images finish in ~1.63 s at ~39 images/s.
- • 64 threads on 8 cores: each core rotates through 8 threads; every job finishes at roughly the same late moment, so p99 latency is 1.9 s and throughput is ~33 images/s — worse on both axes.
- • 64 threads where one holds a shared lock and is preempted: 63 threads wait not for a 500 ns section but for that thread's next slice, milliseconds later.
- • 64 threads blocked on network I/O instead of computing: only a handful are runnable at any instant, the cores are idle, and none of this lesson applies. Same thread count, entirely different regime.
- • 512 threads: switch overhead and 11 GB of stack reservations dominate, and wall-clock time is worse than the 4-thread case that left half the machine idle.
- • The OS guarantees every runnable thread eventually gets CPU time. It does not guarantee it gets it soon, or that the aggregate work rate is preserved.
- • Creating N threads guarantees N execution contexts. It guarantees nothing about parallelism, which is bounded by cores — see Which One Does This Workload Need?.
- • A thread pool guarantees a bound on concurrent execution. It does not guarantee the bound is right; sizing is measurement, and this domain deliberately gives no formula. See Sizing a Thread Pool.
- • Blocked threads are guaranteed not to consume CPU, which is why I/O-bound thread counts far above core count are legitimate. They still consume memory.
- • A container CPU limit guarantees a quota, and does not guarantee your runtime knows about it — most default their pool sizes to the host core count, which silently oversubscribes every container on the box.
- • Contention for cores shows as run-queue length; the standard rule of thumb is that a run queue persistently above core count means threads are waiting for CPU.
- • Contention for last-level cache is invisible in any thread-level metric and shows up as a rising cache-miss rate and falling instructions-per-cycle.
- • Memory-bus contention caps throughput independently of core count for bandwidth-heavy work — adding threads there does nothing even below the core count. See Memory Bandwidth: More Cores, Same Bus.
- • Lock contention is amplified: with 8× oversubscription, the expected wait for a lock whose holder can be preempted rises by roughly the oversubscription factor.
- • Throughput decline past the peak, with latency rising monotonically — the core signature.
- • Out-of-memory kill at very high thread counts, caused by stack reservations rather than by heap growth.
- • Latency collapse under burst, where a spike in arrivals spawns a spike in threads that makes every in-flight request slower — a positive feedback loop. See Unbounded Concurrency.
- • Preempted lock holders converting a fast critical section into a scheduling-latency-bound one, and from there into a Lock Convoys.
- • Container throttling: the cgroup quota is consumed early in each period and every thread stalls for the remainder, producing latency spikes with no corresponding CPU signal inside the container.
- • Modest oversubscription is genuinely useful when threads occasionally block: extra threads keep cores busy during page faults, short I/O and lock waits, so the peak is often somewhat above core count rather than exactly at it.
- • It helps for latency fairness across many small jobs, where time-slicing prevents one long job from monopolising a core.
- • It is harmless for blocked threads: an I/O-bound service with hundreds of waiting threads is not oversubscribed, and forcing it down to core count would reduce throughput.
- • For pure CPU-bound work, where every thread beyond core count adds overhead and subtracts throughput.
- • For cache-sensitive work, where the pollution term dominates and even a small amount of oversubscription is expensive.
- • In containers, where the runtime's default pool sizing sees host cores and the cgroup allows a fraction of one — the most common accidental oversubscription in modern deployments.
- • Wherever threads hold locks: the amplification of lock waits by scheduling delay is the mechanism that turns a slow system into a stalled one.
- • Throughput as a function of thread count, on the same input. The peak is the answer, and its location is workload-specific — this is the measurement that replaces the formula nobody should give you.
- • Context switches per second (
vmstat,pidstat -w,perf stat -e context-switches,cs). Order-of-magnitude jumps track oversubscription directly. - • Run-queue length (
vmstatrcolumn, load average interpreted against core count) persistently above core count. - • Instructions per cycle and last-level cache miss rate (
perf stat). Falling IPC with unchanged code is the cache-pollution term made visible. - • RSS and virtual size against thread count, to catch the memory ceiling before the OOM killer does.
- • In containers: cgroup
cpu.statthrottled time. Non-zero throttling with high internal CPU is the container-specific version of this failure.
- • Bounding concurrency requires a pool, a queue, and a decision about what happens when the queue is full — reject, block, or shed. That decision is real design work. See Bounding Concurrency.
- • Correct sizing requires a benchmark harness and a repeatable workload, because the right number is measured rather than derived.
- • Separate pools for CPU-bound and I/O-bound work mean classifying every task and routing it, plus the risk of one pool starving while the other saturates.
- • Container-aware sizing means reading cgroup limits rather than core count, which most runtimes do not do by default and which must be configured explicitly.
- • A bounded thread pool sized by measurement, with a queue in front of it — the standard answer. See Thread Pools.
- • Tasks or coroutines over a small thread pool, so the number of *concurrent* items is large and the number of *runnable threads* stays near core count. See A Task Is Not a Thread and Coroutines: Functions That Can Pause.
- • An event loop for I/O-heavy work, where thousands of in-flight operations need no threads at all. See Event Loops as a Concurrency Model and
io-models. - • Separate pools per workload class, so CPU-bound batch work cannot oversubscribe the cores serving latency-critical requests.
- • Admission control at the edge, so a burst of 64 uploads becomes a queue rather than 64 threads. See Backpressure.
More workers than cores
Thread pool: utilization and queue
capacity = workers / service = 8 / 50 ms = 160.0 req/s ρ = arrivals / capacity = 120 / 160.0 = 0.750 Little L = λ × W → 0.120/ms × 59.8 ms = 7.2 in flight engine status = healthy
Scheduler timeline
What people believe, and what is true
More threads means more work gets done.
For CPU-bound work, throughput peaks near core count and declines after. The extra threads share the same cores and add switching, cache-refill and memory costs on top.
CPU at 100% means the machine is being used well.
At 100% CPU the machine may be executing image code or executing scheduler code and cache refills. Utilisation cannot tell the difference; only throughput against thread count can.
Threads waiting on I/O are oversubscribed too.
Blocked threads are off the run queue and consume no CPU. Oversubscription is about *runnable* threads exceeding cores, which is why I/O-bound pools are legitimately much larger than core count.
Go deeper
Overview
Sixty-four CPU-hungry threads on eight cores means each runs at an eighth speed, plus the cost of switching between them and refilling caches. Total output goes down, not up.
Practical
Bound the pool and measure throughput at several sizes to find the peak. Check your container CPU quota against what your runtime thinks the core count is — that mismatch is the most common accidental case. Watch context switches per second and run-queue length.
Advanced
The cost that dominates is not the switch but the cache state the switch destroys, which is why the penalty scales with working-set size and why affinity helps. It is also why oversubscription and lock contention multiply rather than add: a preempted lock holder converts a nanosecond critical section into a scheduling-latency-bound one, and every waiter pays.
Internals
This is the argument for tasks over threads. A coroutine or green task costs kilobytes and switches in user space without a kernel transition or a full cache flush, so a runtime can keep a hundred thousand of them in flight over a thread pool the size of the core count. The concurrency is unbounded and the parallelism stays matched to the hardware — which is exactly the separation this lesson says you need. See A Task Is Not a Thread and Hybrid Runtimes: It Was Never Threads Versus Async.