Thread & Worker Pools
Pools as bounded resource use rather than unlimited spawning, the reasoning behind sizing (and the refusal to give a universal formula), work stealing, saturation, and bounding concurrency as a first-class design decision.
A pool is not a performance trick, it is a *bound*. Tasks go into a queue, a fixed set of workers takes them out, and the number of things running at once stops being a function of how many requests arrived. Everything interesting about pools is about what queues behind them.
Q · Why hand work to a fixed set of workers instead of starting a thread for every task?
There is no formula. CPU-bound work relates to core count; waiting-bound work can support many more workers than cores; and both numbers are bounded by something downstream that has its own limit. This lesson gives you the variables, the failure signatures at each end, and the instruction to measure.
Q · How many workers should this pool have, and why can nobody hand me the number?
The pool shape does not care what a worker is. Threads, processes, containers and remote machines all give you the same queueing structure — and radically different startup costs, failure modes and answers to the question "what happens to the task the dead worker was holding?"
Q · The queueing shape is identical whether a worker is a thread or a machine — so what actually changes?
Give every worker its own deque, let it push and pop its own end without synchronizing, and when it runs dry let it steal from the far end of somebody else's. Near-zero coordination in the common case, automatic load balancing in the bad case — paid for with cache locality and one genuinely hard race at the last element.
Q · Why does a runtime give each worker a private queue instead of sharing one, and what does an idle worker do about it?
Every worker busy, the queue growing, wait time climbing. Saturation is not a CPU problem and often not even a pool problem — it is arrival rate exceeding completion rate, and Little's Law tells you exactly what the wait will be before you measure it.
Q · All eight workers are busy and the queue is growing — is the pool too small, or is something else the real constraint?
Ten thousand tasks, a semaphore with fifty permits, at most fifty running. The pattern is four lines; the design decisions are where the limit lives, what number it holds, what happens to task fifty-one, and the release you forgot to put in a finally block.
Q · I have ten thousand independent tasks and firing them all at once destroys something downstream — where exactly do I put the limit, and how do I pick it?