The question this answers
Which concurrency model should this server use, and what decides it?
One HTTP request: parse, authorize, one database query, one call to a third-party API, render a response. Roughly 8ms of computation and 180ms of waiting, at 3,000 requests per second across 40,000 open connections.
A connection pool, an in-process cache, and per-connection buffers. What differs by model is not what is shared but *how many concurrent accessors exist* and therefore what synchronization is needed at all.
Every accepted connection is eventually served or explicitly rejected, and the number of requests executing concurrently never exceeds what the process has resources for.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
Five models, and the two numbers that pick between them
The models are not exotic. Thread-per-request gives each request its own OS thread and blocks freely. Thread pool bounds that: a fixed set of threads pulls from a queue — The Thread Pool Server. Event loop runs one thread that multiplexes over ready file descriptors and never blocks — The Event Loop, Blocking, Non-blocking, Multiplexed, Asynchronous. Async tasks keep the event loop but let each request be written as straight-line code that suspends at await points. Worker processes run N copies of any of the above, each with its own memory.
Two numbers select between them more than anything else. The first is the *waiting fraction*: 180ms of waiting against 8ms of computing means a thread spends 96% of its life parked, which is exactly the case where holding an OS thread per request is expensive and buying nothing. The second is *concurrent connections*: 40,000 simultaneous connections means 40,000 threads under thread-per-request, and the memory for their stacks alone is the answer to the question. This is the C10K: Ten Thousand Connections, Then a Million problem, and it is why event-driven servers exist.
The third input is not technical: what the team can operate. An event-loop server has a failure mode — one CPU-bound handler stalls every connection — that a thread-per-request server simply does not have, and that failure mode requires discipline nobody has to think about in a blocking server. Complexity is a real input to this decision, not an excuse.
| Model | Concurrency unit | Good when | Falls over when | The thing that bites |
|---|---|---|---|---|
| Thread per request | One OS thread | Conns in the hundreds; blocking libraries; simple mental model wanted | Conns in the tens of thousands | Stack memory and context-switch cost scale with connections, not with work |
| Thread pool | A bounded set of OS threads | Mixed workloads; you want a hard concurrency ceiling | Every worker blocks on one slow dependency | The queue behind the pool is invisible unless you measure it |
| Event loop | A callback / ready event | Huge connection counts, tiny per-request compute | Any handler computes for more than a few ms | One slow callback stalls every connection, including health checks |
| Async tasks | A suspendable task | High conns, mostly waiting, and you want readable code | A blocking call sneaks into an async path | Task count is unbounded by default, so an outage becomes an OOM |
| Worker processes | A process | Isolation wanted; runtime cannot use cores in one process | State needs sharing across workers | Nothing is shared, so caches, sessions and locks are per-worker |
The decision, as a sequence of questions
The order matters, because the first question can eliminate models outright. Start with connection count: if simultaneous connections are in the tens of thousands, thread-per-request is gone before any other consideration, purely on memory. Then ask about the waiting fraction — Classifying the Work: Computing or Waiting? is the lesson on getting that number honestly, and Computing or Waiting? on reading it from signals. Mostly-waiting work is what async and event loops are for; mostly-computing work needs actual cores and therefore threads or processes.
Then ask what the ecosystem gives you. A blocking database driver in an event-loop server is not a small problem; it converts the loop's single thread into a serialized queue and every advantage evaporates. Conversely, a mature async ecosystem makes the async choice nearly free. This is a genuine input and is usually decided by the language, not by you — see JavaScript: One Event Loop per Agent, Not One Thread per Runtime, Python: Threads, Processes and the GIL and C++: Threads, Atomics and a Memory Model With Teeth.
Finally, ask what happens under overload, because every model needs an answer and only some make it obvious. A thread pool has a queue you can bound. An event loop accepts connections until the descriptor limit and then fails in a way that is hard to attribute. Async tasks spawn until memory runs out unless something bounds them — Bounding Concurrency. Pick the model whose overload behaviour you can live with, and then actually configure the bound.
The same request under three models
The timeline below runs the identical 8ms-compute, 180ms-wait request under thread-per-request, an event loop and a thread pool. The work is the same in all three; what differs is what is occupied while the request waits, and that is the entire decision.
Under thread-per-request an OS thread with its stack is held for the full 188ms while doing nothing for 180 of them. Under an event loop the thread is released back to serve other connections at each suspension, and the request occupies only a small task structure while waiting. Under a thread pool the thread is still held — a pool does not change the blocking, it only bounds how many can block at once, which is why a pool of 32 workers with a 180ms wait tops out around 178 requests per second regardless of how fast the code is.
That last number is the calculation people skip. Pool throughput is bounded by workers divided by mean request duration, and no amount of code optimization changes it while the duration is dominated by waiting. Either the waiting has to stop occupying a worker — which is what async does — or there have to be more workers, which is bounded by memory and scheduling. Little's Law as Working Intuition and Sizing a Thread Pool are the arithmetic.
Key points
- Five models: thread-per-request, thread pool, event loop, async tasks, worker processes. This is a decision, and it has inputs.
- Two numbers dominate: the fraction of a request spent waiting, and the number of simultaneous connections.
- Tens of thousands of connections eliminates thread-per-request on stack memory alone, before any other argument.
- A thread pool does not stop blocking, it bounds it: throughput is workers divided by mean duration, and code speed cannot move that while duration is waiting.
- The ecosystem is an input you usually do not control — one blocking driver turns an event-loop server into a serialized one.
- Every model needs an overload answer. Pick the one whose overload behaviour you can live with, then actually configure the bound.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • Thread-per-request: accept, spawn a thread, run the handler with blocking calls, let the OS scheduler multiplex. Simplicity comes from the kernel doing the multiplexing.
- • Thread pool: accept, enqueue, a fixed set of threads dequeues and runs handlers. The bound is explicit and the queue becomes a first-class object to measure.
- • Event loop: register all sockets with a readiness mechanism, wait for events, dispatch callbacks. Nothing may block, because there is one thread for everyone.
- • Async tasks: the same loop, but each request is a state machine that suspends at await points and is resumed when its awaited resource is ready.
- • Worker processes: N independent processes, usually one per core, each running one of the above, with a load balancer or a shared listening socket in front.
- • Thread pool of 32, 180ms mean duration, 3,000 req/s arriving: 170/s served, 2,830/s queued. Within a second the queue holds thousands of requests whose clients have already timed out — work is being done for nobody. See The Backlog Arithmetic: Four Levers and a Drain Time.
- • Event loop, one handler performs a 400ms synchronous compression: every other connection — including the health check — waits 400ms. The load balancer marks the instance unhealthy and removes it, and the remaining instances get more load. See Blocking the Event Loop.
- • Thread-per-request with 40,000 connections: 40,000 threads, tens of gigabytes of stack, and the scheduler spending a large fraction of its time switching between threads that are all blocked anyway.
- • Worker processes with an in-process cache: eight workers, eight independent caches, an 8x increase in cache misses and eight copies of every warmed entry. Nothing is wrong, and nothing is shared.
- • A thread pool guarantees a bound on concurrent execution. It does not guarantee a bound on *queued* work unless the queue is bounded too, and an unbounded queue is a memory leak with a schedule.
- • An event loop guarantees no data races between handlers on the loop, because only one runs at a time. It guarantees nothing about logical race conditions across await points — a task can be suspended mid-invariant.
- • Thread-per-request guarantees isolation of blocking: one slow request cannot stall another. It guarantees nothing about resource exhaustion.
- • Worker processes guarantee memory isolation, which also means they guarantee that nothing in memory is shared — including the lock you were relying on.
- • No model guarantees throughput. Every one of them has a ceiling set by a resource, and the model determines which resource.
- • Thread-per-request and pools contend on any shared structure — the cache, the pool, the metrics registry — because many threads execute simultaneously.
- • An event loop contends on nothing within the loop, which removes an entire class of bug, and instead contends on *time*: every handler competes for the same single thread.
- • Worker processes contend on shared external resources instead — the database connection limit is now consumed by N processes, and N times the pool size is often past the server's limit.
- • Pool exhaustion: all workers blocked on one slow dependency, queue growing, and the service failing for everything including endpoints that never touch that dependency.
- • Loop stall: one CPU-bound handler makes every connection slow simultaneously, and the symptom looks like a total outage rather than a slow endpoint.
- • Thread explosion: unbounded thread-per-request under a traffic spike, ending in memory exhaustion or scheduler collapse.
- • Unbounded task growth: async spawning with no limit converts a downstream outage into an out-of-memory kill.
- • Silent state divergence under worker processes, where a per-process cache or rate-limit counter is assumed to be global and is not — see A Mutex on Server A Does Nothing About Server B.
- • Making the decision explicitly, once, with the two numbers written down — most servers inherit their model from a framework and never revisit it against their actual traffic shape.
- • Diagnosing a saturated service, because each model has a distinct saturation signature and knowing the model tells you which signal to look at first.
- • Planning a migration, where the honest framing is that you are trading one failure mode for another, not eliminating failure.
- • Switching models to fix a problem the model did not cause. A slow database query is slow under every model, and an async rewrite is an expensive way not to fix it.
- • Choosing an event loop for compute-heavy work, which converts a scaling problem into an availability problem.
- • Optimizing the model when the real ceiling is downstream — a connection pool of 20 caps you at 20 concurrent queries regardless of how many tasks you can hold.
- • Waiting fraction per request: on-CPU time divided by wall-clock duration. Under about 20% on-CPU, the case for async is strong.
- • Peak simultaneous connections, not requests per second — the two are different and only the first sizes thread-per-request.
- • Worker utilization and queue age together, which is the pool saturation signature — Twenty Workers, All Busy, Five Hundred Waiting.
- • Event-loop lag p99, which is the only signal that catches a stalled loop before users do — Event-Loop Lag: One Callback, Everybody Waits.
- • Memory per in-flight request, which is the number that decides whether 40,000 concurrent anything is feasible.
- • Async and event-driven models impose a discipline — never block the loop — that has to be enforced in review forever, including in every dependency.
- • Thread-based models impose synchronization on every shared structure, and each lock is a potential contention point and deadlock participant.
- • Worker processes impose an explicit answer to "where does shared state live", usually meaning an external cache or database.
- • Hybrid models, which most real servers become, require understanding two schedulers at once — see Hybrid Runtimes: It Was Never Threads Versus Async.
- • Do not increase concurrency at all: make the request faster. Removing a 150ms downstream call beats every model change on this list.
- • Horizontal scaling with a simple model, when machines are cheaper than the complexity of an async rewrite — Horizontal vs Vertical Scaling.
- • Move the slow work out of the request entirely with a job queue, which changes the shape of the problem rather than the model — Background Jobs and Workers.
- • Bound concurrency explicitly with a semaphore in front of the slow dependency, which fixes pool exhaustion without changing the server model — Bounding Concurrency.
Server model lab
Each model is the same simulator given a different worker shape: a thread per in-flight request, a fixed pool, or one task per core where waiting does not occupy a worker. Memory is a per-thread stack estimate. Real servers differ by orders of magnitude in all of these, and every runtime has its own hybrids. Concurrency in flight is the knob; offered load is derived from it as concurrency ÷ service time.
What people believe, and what is true
Async servers are faster.
Async servers hold more concurrent connections per unit of memory. For a single request they are usually slightly slower, and for compute-bound work they are worse.
A thread pool fixes blocking.
It bounds it. The worker is still held for the full blocking duration, and throughput is workers over duration no matter what the pool is called.
Pick the model, then design the system.
The traffic shape picks the model. Connection count and waiting fraction are inputs you measure, not preferences you hold.