The question this answers
Why is the simplest server model also the one that stops working first, and at what point exactly?
One HTTP request served start-to-finish on a dedicated OS thread, blocking on the database and on an upstream API, at connection counts from 100 to 40,000.
Nothing per-request — each thread has its own stack and locals, which is the model's central advantage. Shared state is only what you deliberately share: a cache, a pool, a counter.
Each request is served by exactly one thread from accept to response, and the number of live threads equals the number of in-flight requests.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
What the model buys, stated properly
The advantages are not merely aesthetic and it is worth being precise about them. First, *the stack is the request*: a stack trace at any moment shows the entire causal chain that led here, which is the single most useful debugging artefact a server can have. Second, request-local state is just local variables, so there is no shared mutable state to protect and therefore no race to reason about — most of this domain does not apply inside a handler. Third, blocking libraries work, which in practice means every library works.
Fourth, and least appreciated: the OS scheduler does the multiplexing, and it is very good at it. Preemption means a runaway handler cannot starve the others — the kernel will take the core away. That is a genuine robustness property that cooperative models do not have, and it is why Blocking the Event Loop has no counterpart here.
The mechanism is Thread per Connection and The Blocking Server in Operating Systems; what this lesson adds is the reasoning about where the model's cost curve turns, because the answer is not "it does not scale" but a specific set of three limits with specific numbers behind them.
Three axes of degradation, with the arithmetic
The first axis is memory. Each thread gets a stack, and the reserved size is typically measured in hundreds of kilobytes to a megabyte or more. At 1,000 threads that is under a gigabyte and unremarkable; at 40,000 it is tens of gigabytes of virtual reservation, and the resident portion grows with how deep the handlers actually go. Deep frameworks with long call chains make this much worse than the reserved number suggests.
The second axis is scheduling. A context switch costs on the order of a microsecond in direct cost, and considerably more in indirect cost as the new thread finds cold caches and a cold TLB — The Cost of a Context Switch and Context Switching. With threads far exceeding cores, the run queue lengthens and a runnable thread waits its turn; the useful mental model is that past the core count you are not adding parallelism, you are adding queueing. Crucially, a *blocked* thread costs almost nothing to the scheduler — the cost appears when many threads are simultaneously runnable, which happens exactly when a downstream recovers and thousands of blocked threads wake at once.
The third axis is connections and descriptors. One thread per request usually means one connection held per request: one database connection, one upstream socket, one client socket. The database connection limit, not your thread count, then becomes the ceiling — Connection Pool Saturation: Waiting in Front of an Idle Database — and this is the limit teams hit first in practice, long before memory. It is also the one that produces the most confusing incident, because the server has idle threads and the database has idle capacity while everything waits in the pool.
What to do before abandoning it
The most common mistake is treating "we hit a limit" as "we need async". The scaling curve above bends at the connection pool, which is an external resource that an async rewrite does not enlarge. Moving to async would let you hold 40,000 pending tasks instead of 40,000 threads — and all 40,000 would then queue on the same 100-connection pool. The bottleneck is not the model.
The three fixes that actually extend the model: bound the concurrency to a pool so thread count stops tracking connection count — that is Thread Pools and turns thread-per-request into thread-per-*active*-request; shrink stack sizes if the runtime permits and the handlers are shallow; and add machines, because a model that works fine at 500 connections per process works fine at 20,000 across forty processes and costs less engineering than a rewrite.
The matrix below is the honest comparison at three connection scales. Note that thread-per-request is not merely acceptable at the low end — it is *better*, because the debugging properties are real value and the alternatives' failure modes are real cost. The model does not deserve its reputation as a legacy choice; it deserves a connection-count bound.
| Concurrent connections | Thread-per-request | Thread pool | Async / event loop |
|---|---|---|---|
| ~200 | Best choice. Simple, debuggable, no discipline required. | Fine, adds a bound you may not need yet. | Overkill; you pay the discipline cost for nothing. |
| ~5,000 | Workable: ~5GB of stack reservation, scheduler still fine, but the connection pool is almost certainly the real limit. | Best choice. Bounds concurrency where the external limit already is. | Reasonable, if the ecosystem is genuinely async. |
| ~40,000 | Not viable: tens of GB of stacks, long run queues on wakeup storms. | Viable — but the queue behind the pool is now most of the latency. | Best choice, provided nothing blocks the loop. |
| Debuggability | Stack trace = the whole request. Nothing beats it. | Same, plus a queue to reason about. | Suspended tasks often have no meaningful stack — Task Dumps: When the Threads Look Idle and Nothing Is Moving. |
| Failure under overload | Thread explosion, then memory exhaustion. | Queue growth; bounded if you bounded it. | Unbounded pending tasks, then OOM. |
Key points
- The stack is the request: a single stack trace shows the whole causal chain, which no other model matches.
- Request-local state is just local variables, so most of this domain's hazards do not exist inside a handler.
- The OS preempts, so one runaway handler cannot starve the others — a robustness property cooperative models lack.
- It degrades on three axes: stack memory per thread, scheduler cost when many threads are simultaneously runnable, and one held connection per request.
- A blocked thread is cheap. The scheduler cost appears on wakeup storms, when thousands of blocked threads become runnable at once.
- The limit teams actually hit first is the connection pool, and an async rewrite does not make that pool bigger.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • Accept a connection, spawn or check out a thread, and run the entire handler on it with ordinary blocking calls.
- • Blocking calls park the thread in the kernel; the scheduler removes it from the run queue and it consumes no CPU while parked.
- • Each thread reserves a stack at creation; the reservation is virtual, but the pages actually touched become resident and stay so.
- • On I/O completion the kernel marks the thread runnable; if many complete at once, many threads become runnable simultaneously and queue for cores.
- • The thread returns to the pool or exits when the response is written, releasing its connection and its stack.
- • Eight threads, eight cores, all blocked on the database: CPU is near zero, memory is trivial, and the model is behaving perfectly. Blocking is not the cost.
- • A downstream recovers after a 30-second outage: 8,000 blocked threads become runnable within milliseconds, the run queue is 1,000 deep per core, and latency spikes for everything — including requests that never touched the downstream. This is the wakeup storm, and it is thread-per-request's characteristic failure.
- • One handler compresses a 40MB response for 160ms. The kernel preempts it repeatedly; every other request proceeds normally. Under an event loop this same handler would have stalled every connection in the process.
- • 512 threads all holding a database connection against a pool of 100: 412 threads are blocked inside pool.acquire, the database is at 8% CPU, and the server has 412 threads doing nothing. The ceiling is external and no thread-count change moves it.
- • It guarantees isolation of blocking: one slow request cannot delay another as long as threads and cores remain available.
- • It guarantees preemptive fairness at the OS level — a CPU-bound handler will be interrupted, so it cannot monopolize the process.
- • It guarantees a coherent stack trace per request, which is a debugging guarantee no other model on the list provides.
- • It does not guarantee any bound on resource use. Unbounded thread-per-request under load is a memory-exhaustion mechanism, and only an explicit pool fixes that.
- • It does not guarantee that request-local means race-free: any shared cache, counter or pool the handler touches is shared by every thread simultaneously.
- • Threads contend for cores only when simultaneously runnable, which is the wakeup-storm case rather than the steady state.
- • Any shared structure the handler touches is contended by the full thread count at once — a global cache lock with 512 threads is a very different object from one with 8.
- • The connection pool is the usual contention point, and it is contention on an external limit rather than an internal one.
- • Memory allocation is shared: hundreds of threads allocating simultaneously can contend on the allocator itself, which is invisible in application-level profiling.
- • Thread explosion under a spike: unbounded spawning until memory is exhausted and the process is killed.
- • Wakeup storm after a downstream recovery, converting a resolved outage into a second latency incident.
- • Pool exhaustion: every thread blocked in
pool.acquire, the database idle, and the service returning timeouts — the most confusing shape in the list. - • Stack overflow in deeply-recursive handlers, which under thread-per-request is per-request rather than global — see Stack Overflow.
- • Silent oversubscription: thread count grown to "fix" latency, which lengthens the run queue and makes latency worse — More Threads Is Not More Speed.
- • Connection counts in the hundreds to low thousands, which describes an enormous share of real internal services.
- • Codebases dominated by blocking libraries, where the async alternative would mean rewriting or wrapping every dependency.
- • Teams where debuggability matters more than peak connection density — most teams, most of the time.
- • Compute-heavy handlers, where the OS preemption property is not merely convenient but load-bearing.
- • Tens of thousands of long-lived connections — websockets, SSE, long polling — where nearly every thread is idle and the memory is pure waste. See WebSockets and Polling vs Long Polling vs SSE vs WebSockets.
- • Very high request rates with tiny handlers, where thread creation and switching costs approach the cost of the work.
- • Memory-constrained environments, where stack reservations crowd out the heap the application actually needs.
- • When the model is used unbounded, at which point it is not a model but an absence of one.
- • Live thread count against core count and against connection count — the two ratios answer different questions.
- • Runnable-but-not-running time, which is the wakeup-storm and oversubscription signal — Off-CPU Time: The Thing a CPU Profiler Cannot See.
- • Resident memory attributable to stacks, not the virtual reservation, which overstates it substantially.
- • Time blocked in
pool.acquireas a share of request duration, which is usually the real ceiling. - • Context switches per second, and specifically involuntary ones, which indicate more runnable threads than cores.
- • Almost none inside a handler, which is the entire point and should be weighed as a genuine benefit.
- • The complexity moves to configuration: thread limits, stack sizes, pool sizes, and the queue in front of the pool.
- • Shared structures touched by every handler need synchronization designed for the full thread count, not for a handful.
- • Capacity planning is per-process and per-machine rather than per-task, which is simpler to reason about but harder to scale smoothly.
- • A bounded thread pool, which keeps every property of this model while capping thread count — usually the correct next step rather than a rewrite. See Thread Pools.
- • Async tasks, when connection counts are genuinely high and the ecosystem supports it — Event-Driven Servers: Many Connections, One Loop.
- • More processes on more machines, which preserves the simple model and is frequently cheaper than an architectural change.
- • Moving the slow dependency out of the request path with a job queue, which reduces request duration and therefore the thread count needed — Background Jobs and Workers.
What people believe, and what is true
Blocking threads waste CPU.
A blocked thread is off the run queue and costs the scheduler essentially nothing. It wastes memory, not CPU — and the CPU cost appears only when many wake at once.
Thread-per-request does not scale.
It scales further than expected and then hits a wall, usually an external one. The honest statement is that it scales with memory and stops at the connection pool.
We hit our limit, so we need async.
If the limit is the database connection pool, async lets you queue more work in front of the same pool. Check which resource ran out before changing models.