The question this answers
If threads make things faster, what is the right number — and why does that question have no general answer?
A batch of one thousand CPU-bound checksum jobs, each 50 ms of pure computation, on a four-core machine.
Nothing in the application. The shared resources are four cores, one memory bus, one last-level cache and one scheduler — none of which appear in the code, and all of which are the reason the thread count does not translate into speed.
The total CPU work required is 1 000 × 50 ms = 50 seconds of core-time, regardless of how it is scheduled. Four cores can supply four seconds of core-time per second, so the floor on wall-clock time is 12.5 seconds and no thread count reduces it. Everything above that floor is overhead — and thread count only ever adds to it.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
The curve, and where the peak is
This is the measurement that settles the argument, and the shape is consistent enough to be worth memorising: speedup rises roughly linearly up to about the core count, flattens, and then declines. It never becomes a plateau that you can safely sit on top of — past the peak it goes down, and it keeps going down.
The floor set by total work is the important half of the picture. Fifty seconds of core-time on four cores cannot finish faster than 12.5 seconds. At four threads you get close to it. At sixteen you are slightly worse. At a thousand you are 60% worse and using 8 GB of memory that four threads did not need. Nothing was gained anywhere on that curve after the fourth thread.
The reason to internalise the shape rather than a number is that the peak location is workload-dependent and not derivable. For pure computation it is near core count. For work that blocks half the time it may be two or three times core count. For work bound by memory bandwidth it can be *below* core count, because the bus saturates before the cores do — see Memory Bandwidth: More Cores, Same Bus. There is no formula that spans those cases, which is why this domain does not offer one.
Where the extra threads actually go
The timeline makes the arithmetic physical. Four cores are four cores. A thousand threads do not create a fifth; they subdivide the four that exist into a thousand slices, and every slice boundary costs a switch and a cache refill (The Cost of a Context Switch). The useful-work band per core shrinks while the overhead band grows.
Look at what changed and what did not. The total amount of checksum computation is identical in both rows — the same instructions run either way. What differs is how much of each core reaches those instructions. At four threads it is nearly all of it; at a thousand, a visible fraction is spent switching and re-warming, and the remaining compute runs at lower IPC because the caches are cold.
The latency consequence is the one that catches teams out in a service context. With four threads, jobs complete progressively: the first finishes at 50 ms and the queue drains steadily. With a thousand threads all in flight, nothing finishes early — every job progresses at a thousandth speed and they all complete near the end together. Throughput is worse and *every single job's* latency is worse. There is no axis on which the thousand-thread version wins.
What threads are actually for
The honest framing is that a thread is a unit of *concurrency*, not a unit of *speed*. Adding one buys the ability to have another thing in flight. Whether that translates into throughput depends entirely on whether the resource the work needs has spare capacity — and for CPU-bound work on a saturated machine, it does not.
The table below is the version of this reasoning worth carrying, because the answer genuinely differs by workload class and the differences are not small. The most useful entry is the second row: for I/O-bound work, thread count above core count is correct, because the threads are blocked rather than runnable and consume no CPU while waiting. Applying the CPU-bound rule there would leave the machine idle. Classifying first is the whole of the technique — see Classifying the Work: Computing or Waiting? and cpu-bound-vs-io-bound.
The last row is the one that ends most of these discussions. When a lock serialises the work, thread count is irrelevant in both directions: the ceiling is one over the critical-section length (What Contention Actually Costs, False Parallelism) and no thread count moves it. A team adding threads to a lock-bound service is optimising a variable that does not appear in the equation.
| Workload class | What limits it | Does another thread help? | Roughly where the peak is |
|---|---|---|---|
| CPU-bound, independent | Core count. | Up to about core count, yes, nearly linearly. After that it costs throughput and memory. | At or slightly above core count. Verify with a curve, not a formula. |
| I/O-bound (network, disk) | The remote service or device, plus your connection limits. | Yes, well past core count — waiting threads are blocked and use no CPU. This is why "threads = cores" is wrong as a general rule. | Wherever the downstream resource saturates. Often 10–100× core count, and bounded by that resource, not by yours. See Bounding Concurrency. |
| Memory-bandwidth-bound | The memory bus, shared by all cores. | Often no, even below core count — the bus saturates before the cores do. | Can be 2 threads on an 8-core machine. See Memory Bandwidth: More Cores, Same Bus. |
| Lock-bound / serialised | The critical section: ceiling = 1 / hold time. | No, in either direction. More threads lengthen the queue and change nothing else. | Irrelevant. Fix the lock; the thread count is not the variable. See What Contention Actually Costs. |
| Mixed (typical service) | Whichever of the above binds first, and it moves under load. | Sometimes, and only until the binding constraint changes. | Measured, per deployment, and revisited when the workload changes. Separate pools per class beat one shared number. |
Key points
- Total work divided by core count is a floor on wall-clock time that no thread count can go below.
- Speedup for CPU-bound work peaks near core count and then declines — it does not plateau safely.
- Threads are a unit of concurrency, not of speed: another thread helps only if the binding resource has spare capacity.
- The correct thread count differs by an order of magnitude between CPU-bound, I/O-bound, bandwidth-bound and lock-bound work, which is why no universal formula exists.
- At extreme thread counts the failure is memory (stack reservations), not slowness — and it arrives suddenly.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • Each thread is an independent execution context that the scheduler may place on any available core.
- • When runnable threads exceed cores, the scheduler time-slices them; total core-seconds available per second is fixed at the core count.
- • Each slice boundary adds a context switch and a cache refill, so the fraction of each core reaching application instructions falls as thread count rises.
- • Each thread reserves stack address space and kernel structures whether running or not, so memory grows linearly with thread count.
- • For blocked threads none of this applies — they are off the run queue — which is why I/O-bound thread counts far above core count are correct.
- • 4 threads on 4 cores: each job runs uninterrupted to completion; the first finishes at 50 ms and the batch drains steadily to 13.1 s.
- • 1 000 threads on 4 cores: every job advances a thousandth at a time; nothing finishes early and the batch completes at 21.0 s, with every individual latency worse.
- • 1 000 threads where the jobs are network calls: only a handful are runnable at any instant, the cores are mostly idle, and 1 000 concurrent calls is a perfectly reasonable design — bounded by the downstream service rather than by your cores.
- • Mixed batch, one shared pool: a burst of CPU-bound work saturates the pool, and the I/O-bound requests that would have been fine queue behind it. This is why separate pools per class beat one tuned number.
- • Lock-bound: 4 threads and 1 000 threads produce identical throughput, because the serial critical section is the ceiling in both cases. The 1 000-thread version just has a longer queue and 8 GB of stacks.
- • Creating N threads guarantees N execution contexts, and guarantees the memory for them is reserved.
- • It does not guarantee N-way parallelism, which is bounded by cores, nor N-way throughput, which is bounded by whichever resource binds first.
- • The OS guarantees each runnable thread eventually runs. It does not guarantee aggregate throughput is preserved as thread count grows — measurably, it is not.
- • A pool size guarantees a bound on concurrent execution. It does not guarantee that bound is correct for the current workload, which is why it needs re-measurement when the workload changes.
- • No formula — not "cores + 1", not "cores × 2", not any Little's-Law rearrangement — guarantees a good thread count, because the binding resource differs by workload and moves under load.
- • Beyond the peak, every additional thread contends for cores, cache, memory bandwidth and run-queue attention, and contributes no throughput.
- • Contention for the memory bus can bind below core count, which is the case that makes "threads = cores" wrong in the other direction.
- • If any lock exists, oversubscription amplifies its cost, because a preempted holder blocks every waiter for a scheduling round. See Oversubscription.
- • Latency contention is the least-tracked form: with a thousand threads in flight nothing completes early, so p50 latency approaches p99 and both are worse.
- • Throughput regression after "adding parallelism", which is the canonical form of this failure.
- • Latency collapse under burst when thread-per-request meets a traffic spike — more arrivals spawn more threads, which slows every in-flight request. See Unbounded Concurrency.
- • Out-of-memory from stack reservations at very high thread counts, which is a hard failure rather than a gradual one.
- • Container throttling, where the pool was sized to host cores and the cgroup grants a fraction of one.
- • Optimising the wrong variable entirely: adding threads to a lock-bound or bandwidth-bound system, where the count does not appear in the ceiling.
- • Up to about core count for independent CPU-bound work, additional threads deliver close to linear speedup — this part is real and worth having.
- • Far past core count for I/O-bound work, where threads are blocked and the limit is the downstream resource, not yours.
- • For latency structure rather than throughput: a separate thread keeps a UI or a control loop responsive while long work proceeds, which is a correctness-adjacent benefit rather than a speed one. See UI Concurrency: One Thread Owns the Screen.
- • Past the peak for CPU-bound work, always, on both throughput and latency simultaneously.
- • When the binding constraint is a lock, the memory bus or a downstream service — in each case thread count is not the variable in the equation.
- • When it hides the real fix. "Add threads" is fast to try and it postpones finding the serial section that actually caps the system.
- • Throughput against thread count on a repeatable workload, at 1, 2, 4, 8, 16 and beyond. The peak is your answer and the shape is the explanation.
- • p50 and p99 latency at each of those points, because throughput alone hides the fact that latency degrades monotonically past the peak.
- • RSS and virtual size against thread count, to find the memory ceiling before it finds you.
- • Context switches per second and IPC at each point, which explain *why* the curve turned over — see The Cost of a Context Switch.
- • The binding resource: CPU saturation, lock held-percentage, memory bandwidth, or downstream latency. Which one is at its limit determines whether the thread-count question is even the right question.
- • Doing this properly means owning a benchmark harness and a repeatable workload — the measurement is the deliverable, and most teams do not have one.
- • Separate pools per workload class means classifying tasks, routing them, and sizing each pool independently, plus the risk of one pool idling while another saturates.
- • Any measured number is valid for one workload on one machine shape and needs revisiting after workload changes, instance-type changes and library upgrades.
- • Bounding concurrency forces a decision about what happens to work that does not fit: queue it, reject it, or shed it. That is design work the unbounded version avoided by failing instead.
- • Bound the pool and queue the excess, so concurrency is explicit and the queue depth is an observable signal. See Thread Pools and Backpressure.
- • Use tasks or async for I/O-bound concurrency, so thousands of operations are in flight over a thread count near core count. See A Task Is Not a Thread and Await Is a Yield Point.
- • Make the work cheaper. A 2× algorithmic improvement beats any thread-count tuning and does not add a scheduling problem. See Algorithmic Cost in a Request Handler in Observability & Performance.
- • Scale horizontally when the machine is genuinely saturated — but only after confirming that it is CPU-saturated rather than lock- or bandwidth-bound, because horizontal scaling does not fix a serial section per instance.
- • Separate pools per workload class rather than one global number, which is usually a bigger win than tuning any single pool.
What people believe, and what is true
More threads means more throughput.
Only while the binding resource has spare capacity. For CPU-bound work on a saturated machine, throughput peaks near core count and then falls — the extra threads consume the machine rather than using it.
There is a correct formula, like cores × 2 + 1.
The right count differs by more than an order of magnitude between CPU-bound, I/O-bound and bandwidth-bound work, and moves as the workload changes. Any formula that ignores which resource binds is guessing.
Extra threads are harmless if they are mostly idle.
Blocked threads are indeed nearly free in CPU terms — that is why I/O pools are large. But every thread costs stack reservation and kernel structures, and *runnable* extra threads cost throughput directly.
Go deeper
Overview
Four cores can do four cores' worth of work per second. A thousand threads do not add a fifth core; they slice the four you have into a thousand pieces and charge you for every slice boundary.
Practical
Plot throughput and p99 against thread count on your own workload. Take the peak. Then classify the work — CPU, I/O, bandwidth or lock — because that determines whether the peak is near core count, far above it, or irrelevant.
Advanced
The generalisation is that concurrency is only useful when it lets you use a resource that would otherwise be idle. Extra threads help I/O-bound work because a blocked thread leaves the CPU free; they do not help CPU-bound work because there is no idle CPU to claim. Every thread-count decision reduces to identifying the idle resource, and if there is none, the answer is a better algorithm rather than more threads.
Internals
Little's Law is the honest tool here and it is often misused: concurrency = arrival rate × latency describes the number of items *in flight*, which for I/O-bound work is a good guide to pool size. It is not a formula for CPU-bound threads, because there the latency term itself depends on the thread count you are trying to compute. Using it in that direction produces a fixed point, not an answer — see littles-law and Sizing a Thread Pool.