The question this answers
How many workers should this pool have, and why can nobody hand me the number?
One pool serving two task families: JPEG re-encoding (pure CPU, ~80 ms each) and outbound webhook delivery (~5 ms of CPU, 300–2000 ms waiting on a remote server).
The cores themselves, the memory bandwidth, and — critically — the downstream resources every worker reaches for: a 20-connection database pool, a partner API with a 50 req/s quota, a disk with a queue depth.
At any instant, active tasks never exceed the pool size — so whatever number you choose *is* the concurrency contract every downstream resource is being held to, and every one of them must survive it.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
Why the formula you were given is wrong
The number that circulates is cores × (1 + wait/service). It is a useful *intuition* — more waiting means more workers can be in flight before the CPU is the constraint — and a bad *formula*, because it assumes wait and service times are constants, that the only shared resource is the CPU, and that nothing downstream has an opinion. In the webhook pool above, wait/service is somewhere between 60 and 400 depending entirely on which partner is slow this hour, so the formula returns a range spanning two orders of magnitude and calls it an answer.
The honest framing: pool size is the point where one of several curves bends, and which curve bends first is a property of your system, not of concurrency. Throughput rises with workers until the first saturated resource stops it, then flattens, then *falls* as coordination overhead exceeds the marginal work — More Threads Is Not More Speed is the lesson for that fall, and Oversubscription for its mechanism.
So the deliverable of this lesson is not a number. It is: know which resource saturates first, know the failure signature at each end, and get the number from a load test on hardware that resembles production. Anyone who gives you a pool size without asking what the tasks do is guessing.
| Variable | What it is | Push size up when… | Push size down when… | How you observe it |
|---|---|---|---|---|
| Service time | CPU actually consumed per task | It is a small fraction of total task duration | It dominates the task | CPU profile per task (Off-CPU Time: The Thing a CPU Profiler Cannot See) |
| Wait fraction | Share of task duration blocked on I/O or a lock | Tasks are mostly waiting on remote calls | Tasks are compute-dense | Thread state sampling in a thread dump |
| Available cores | What the scheduler can actually run at once | You have more of them | The container CPU quota is below the visible core count | Cgroup quota, not nproc |
| Downstream limit | Connections, quotas, disk queue depth | Downstream comfortably absorbs more | Downstream is the binding constraint | Downstream saturation and error rate |
| Memory per task | Stack plus working set held while running | Tasks are small and stateless | Each task pins tens of MB | RSS versus active worker count |
| Latency target | What the caller will tolerate | Throughput matters more than per-request latency | A queue would blow the p99 budget | Queue age at p99 |
| Variance of task duration | Spread between fastest and slowest task | Tasks are uniform | A few tasks are 1000× the median | Duration histogram, not the mean |
The two failure signatures, and how they look different
Undersized and oversized pools both show up as "it is slow", and they need opposite fixes, so distinguishing them is the practical skill. The timeline below contrasts four workers on four cores against sixteen workers on the same four cores for identical CPU-bound work.
Undersized, CPU-bound: cores are pegged, the queue grows, queue age rises, and CPU utilisation is near 100% with low involuntary context switching. The work is genuinely arriving faster than the machine can do it, and more workers will not help — you need more machines, less work per task, or a rejection policy.
Oversized, CPU-bound: cores are also pegged, but a large share of that time is scheduler overhead. Involuntary context switches climb, per-task latency inflates roughly proportionally to the oversubscription ratio because every task is time-slicing against fifteen others, and total throughput is flat or slightly worse than it was at four workers. Cache locality degrades as tasks bounce between cores (Parallelism Can Destroy Locality).
Oversized, I/O-bound looks different again: local CPU is *low*, the pool looks idle, and the damage is entirely downstream — the database is at connection-pool saturation, or the partner API is returning 429s. The pool is fine; it is doing exactly what you told it to, to somebody else.
How to actually get the number
Treat it as an experiment with one independent variable. Fix the workload, sweep the pool size, and record throughput, latency percentiles and the saturation of every candidate bottleneck at each point. The right size is at or just below the knee — the last size where throughput is still rising and the p99 latency is still inside budget. Past the knee you are buying latency with no throughput.
Two rules that matter more than the sweep. First, sweep against a realistic mix: a pool sized on the median task is destroyed by a workload where 1% of tasks take 100× longer. Second, watch the *downstream* saturation curve, not just yours — the most common sizing incident is a pool tuned to local CPU that quietly sits at four times the database's connection limit, so every worker spends its life waiting for a connection and you have built an expensive queue (Pool Saturation).
And separate the pools. One pool for JPEG re-encoding sized against cores, one for webhook delivery sized against the partner quota. A single shared pool has to be sized for the worse case of both, which means it is wrong for each of them and one starves the other.
- Sweep, do not solve. The output is a curve with a knee; the knee moves when the workload moves.
- Instrument the downstream resource during the sweep — it is usually the thing that bends first.
- Re-measure after any change to task duration, dependency latency, container CPU quota or machine class.
- In a container, read the cgroup CPU quota, not the host core count. A 64-core host with a 2-core quota will happily let you create 64 workers that share two cores.
1for size in [1, 2, 4, 8, 16, 32, 64, 128]:2 pool = Pool(size)3 replay(production_task_mix, duration = 10.minutes) # realistic mix, not medians4 record(5 size,6 throughput = tasks_completed / elapsed,7 p50, p99 = latency_percentiles(),8 queue_age_p99 = time_in_queue_percentile(99),9 cpu_util = host_cpu(),10 ctx_switches = involuntary_context_switches(),11 db_pool_waits = downstream_pool_wait_count(), # the one people forget12 downstream_429 = partner_rate_limit_rejections(),13 )14 15# Choose the largest size where BOTH hold:16# throughput(size) is still meaningfully above throughput(size / 2)17# p99 latency and downstream saturation are inside budget18# Then re-run it when the workload changes, because it will.Key points
- No universal pool-size formula exists;
cores × (1 + wait/service)is an intuition whose inputs are neither constant nor the only constraint. - CPU-bound work is bounded by cores actually available (the cgroup quota, not
nproc); waiting-bound work can support many more in-flight tasks than cores. - The binding constraint is frequently downstream — a connection pool or a partner quota — and no local measurement reveals it.
- Undersized and oversized look identical in "it is slow" and opposite in context switches, queue age and downstream saturation.
- Separate pools for separate task classes; one pool sized for a bimodal workload is wrong for both halves.
- The answer is a sweep against a realistic task mix, repeated when the workload changes.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • Fix the workload: a replay of a realistic production task mix, including the slow tail, not a synthetic uniform load.
- • Sweep pool size across a doubling ladder, holding everything else constant.
- • At each point record throughput, latency percentiles, queue age, host CPU, involuntary context switches, and saturation of every downstream dependency.
- • Plot throughput against size: it rises, flattens at the first saturated resource, and eventually declines as coordination cost exceeds marginal work.
- • Choose at or just below the knee, subject to the p99 latency budget and downstream limits.
- • Encode the choice as configuration with the reasoning in a comment, and re-run the sweep when any input changes.
- • With 16 workers on 4 cores: W1 runs 1 quantum, is preempted, W2 runs 1 quantum, is preempted — after 12 quanta of wall clock, no task has completed, whereas 4 workers completed 4.
- • With a pool of 50 in front of a 20-connection database: 20 workers hold connections, 30 block acquiring one, and the pool is a queue for a queue. Pool utilisation reads 100% while CPU reads 4%.
- • A pool sized 8 against a median task of 80 ms meets a batch where every task takes 40 s: all 8 workers are held, the queue grows without bound, and short tasks behind them time out — head-of-line blocking from duration variance.
- • Two task families share one pool: a burst of webhook deliveries occupies every worker waiting on a slow partner, and CPU-bound re-encode tasks queue behind them while the CPU sits idle.
- • Guaranteed: the pool size caps simultaneous *task occupancy*, which is what downstream resources actually experience.
- • NOT guaranteed: that N workers give N-way parallelism. On 4 cores, 16 CPU-bound workers give 4-way parallelism plus scheduling overhead (False Parallelism).
- • NOT guaranteed: that a size measured today is right tomorrow. It is a function of task duration and dependency latency, both of which drift.
- • NOT guaranteed: that a bigger pool raises throughput. Past the first saturated resource it raises latency and nothing else.
- • NOT guaranteed: that the pool sees the machine. In a container the runtime often reports host cores while the scheduler enforces a quota a fraction of that size.
- • CPU-bound oversubscription contends for cores: every extra worker adds context switches and cache refills that no task benefits from.
- • I/O-bound oversubscription contends downstream: connection pool waits, rate-limit rejections, and disk queue depth are where the cost lands.
- • Memory bandwidth saturates before cores do for streaming workloads — more workers then reduce throughput even with cores free (Memory Bandwidth: More Cores, Same Bus).
- • A shared pool across task classes creates contention between the classes themselves: slow tasks hold workers that fast tasks queue behind.
- • Oversubscription: more runnable workers than cores, so latency inflates roughly with the ratio while throughput stays flat.
- • Downstream saturation: a correctly sized local pool that is four times the database's connection limit — every worker waits, and the CPU graph looks healthy.
- • Head-of-line blocking from duration variance: a handful of very slow tasks occupy every worker.
- • Starvation between task classes sharing one pool.
- • Container quota blindness: sizing against the visible core count when the cgroup allows a fraction of it.
- • When you have a stable, measurable workload and a real bottleneck to size against — the sweep converges quickly and the number holds.
- • When separating pools per task class, which usually improves both latency and throughput more than any single-pool tuning.
- • When the downstream limit is known and hard (a partner quota); sizing the pool to it turns a rate-limit incident into a queue.
- • When the workload is bimodal or heavy-tailed: any single number is wrong for one of the modes, and the fix is separate pools, not a better number.
- • When latency matters more than throughput: the knee of the throughput curve is usually past the latency budget, and sizing to the knee blows the SLO.
- • When the number is tuned once and enshrined — a size measured against a dependency that has since gotten slower is now a cap you do not understand.
- • Throughput versus pool size across a sweep, on a realistic mix — the curve, not a single point.
- • p99 queue age alongside p99 task duration: rising queue age with flat duration means undersized; rising duration with flat queue age means oversubscribed.
- • Involuntary context switches per second, normalised by completed tasks — the direct signature of CPU oversubscription.
- • Downstream pool waits and rate-limit rejections during the sweep — the constraint you did not think to graph.
- • Task duration histogram, not the mean. A bimodal histogram invalidates any single-number sizing.
- • Effective cores: the cgroup CPU quota, checked at runtime, not the core count the runtime reports.
- • The number is now a tuned parameter with an expiry date, and it needs an owner, a recorded rationale and a re-measurement trigger.
- • Separate pools per task class multiply the configuration surface and add the question of how to split total capacity between them.
- • Sizing correctly requires observability you may not have yet: queue age, downstream saturation and per-task CPU attribution.
- • The sizing decision couples your service to a dependency's capacity, so a change on their side becomes a required change on yours.
- • An adaptive limiter (additive-increase / multiplicative-decrease on observed latency), when the downstream capacity moves — it finds the knee continuously instead of once.
- • A queue plus autoscaled worker processes, when the work is elastic and horizontal capacity is cheap: scale worker count on queue age rather than tuning threads.
- • An async runtime with an explicit in-flight limit, when tasks are I/O-bound — decouples "how many can be waiting" from "how many threads exist" (Await Is a Yield Point).
- • Do not size it: run inline and let the caller's own concurrency be the limit, when the caller is already bounded.
More workers than cores
Thread pool: utilization and queue
capacity = workers / service = 8 / 50 ms = 160.0 req/s ρ = arrivals / capacity = 120 / 160.0 = 0.750 Little L = λ × W → 0.120/ms × 59.8 ms = 7.2 in flight engine status = healthy
Why is 8 cores only 4.5×?
| workers | ideal | Amdahl only | realistic | limited by |
|---|---|---|---|---|
| 1 | 1.0× | 1.00× | 1.00× | none |
| 2 | 2.0× | 1.90× | 1.85× | serial |
| 4 | 4.0× | 3.48× | 3.19× | serial |
| 6 | 6.0× | 4.80× | 4.17× | serial |
| 8 | 8.0× | 5.93× | 4.17× | bandwidth |
| 10 | 10.0× | 6.90× | 4.17× | bandwidth |
| 12 | 12.0× | 7.74× | 4.17× | bandwidth |
| 14 | 14.0× | 8.48× | 4.17× | bandwidth |
| 16 | 16.0× | 9.14× | 4.17× | bandwidth |
What people believe, and what is true
Pool size should be cores × (1 + wait/service).
That is an intuition with unstable inputs and only one modelled resource. It ignores every downstream limit, memory, and duration variance. Use it to reason, never to configure.
For I/O-bound work, more threads are basically free.
They are free locally and expensive downstream. Every in-flight worker is a connection, a quota slot or a socket somewhere else.
The runtime reports 64 cores, so I can run 64 CPU-bound workers.
Inside a container the cgroup quota is the real limit and is often a small fraction of the reported count. Sizing against the reported count manufactures oversubscription.
Go deeper
Overview
More workers help until something saturates, then they hurt. Which thing saturates depends on your workload, so the number has to be measured.
Practical
Sweep the size against a realistic task mix; pick at or below the knee subject to the p99 budget; instrument the downstream dependency while you do it; split pools per task class.
Advanced
Duration variance breaks single-number sizing entirely. Bimodal workloads need separate pools or a scheduler that is aware of task class; a heavy tail needs a timeout, not a bigger pool.
Internals
The oversubscription cost is a direct switch (register save, kernel entry) plus an indirect one (cache and TLB refill) that usually dominates it, which is why the throughput curve declines rather than flattening.