Backend Runtime Models
Four ways a server serves many requests at once, and what each one makes cheap, expensive and dangerous.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
While one request is waiting on the database, what is my server doing with the other two hundred?
The service must handle concurrent callers. Someone has to decide what "concurrent" is implemented as, and the decision is usually inherited rather than made.
The framework handles concurrency. Write the handler, the server runs many of them at once, and how it does that is an implementation detail.
One slow endpoint makes every endpoint slow, and nothing in the slow endpoint's code explains why the others are affected (Blocking the Event Loop).
- One slow endpoint makes every endpoint slow, and nothing in the slow endpoint's code explains why the others are affected (Blocking the Event Loop).
- Doubling CPU cores changes nothing, because the runtime executes application code on one thread per process and you did not add processes (Worker Processes).
- Adding threads or workers exhausts the database instead of helping, because every worker holds its own pool and the pools multiply (Connection Pools).
- A load test at 100 concurrent users looks perfect and 200 falls off a cliff — the runtime's unit of concurrency ran out, and the queue behind it is invisible in application metrics.
- Advice copied from another stack makes things worse: "use async" applied to a CPU-bound workload adds machinery and removes nothing (Choosing a Runtime).
What is actually happening
- Every model answers one question: what happens to everything else while this request waits on I/O? Requests spend most of their life waiting on a database, a cache or an external API, so the answer decides your throughput.
- Process per request — fork or hand a request to a dedicated process. Complete isolation, and a process is the most expensive unit there is. Largely historical for HTTP, still the shape of CGI-style and some job runners.
- Thread per request — a thread blocks on I/O while the OS schedules others. Familiar and debuggable; each thread costs a stack (typically measured in hundreds of kilobytes to megabytes) plus context-switch overhead, so the count is bounded in the low thousands, and often in the hundreds.
- Event loop — one thread, non-blocking I/O, and a queue of callbacks. Waiting costs almost nothing, so tens of thousands of idle connections are cheap. In exchange, any code that does not yield stops everything on that thread.
- Lightweight tasks over a thread pool — goroutines, virtual threads, coroutines. Code is written in blocking style; the runtime parks a task when it blocks on I/O and schedules another onto the same OS thread. The ergonomics of threads with a much cheaper unit.
- The models differ mainly in the cost of a waiting request and in what one badly behaved request can damage. Everything else — syntax, ecosystem, framework — sits on top of that.
- Almost every real runtime is a hybrid. Node has one JavaScript thread plus a small pool for file and DNS work; Python asyncio has a loop plus an executor; a JVM server has threads plus non-blocking network I/O underneath.
What one waiting request costs
Compare the models on the one number that matters for a typical backend: a request that is waiting on a database. Nearly all of your requests are in that state nearly all of the time, so its cost is your capacity.
Read the last column as the real distinguishing feature. It is not performance — it is blast radius. Each model has a different answer to "one request behaves badly; who else suffers?".
| Model | A waiting request holds | Concurrency limited by | One request misbehaving hurts |
|---|---|---|---|
| Process per request | A whole process | Memory; process creation cost | Only itself — the strongest isolation available |
| Thread per request | A thread and its stack | Thread count; context switching | Itself, plus contention for shared locks |
| Event loop | A callback registration | Memory per connection; one thread of CPU | Everything on that loop, if it does not yield |
| Lightweight tasks | A parked task (small stack, growable) | Memory; scheduler and pool capacity | A fraction of the pool, unless it blocks an OS thread |
| Worker pool of processes | One worker of N | N, and the resources each worker holds | One worker; N-1 keep serving (Worker Processes) |
Blocking, in three runtimes
The same mistake — a synchronous, CPU-bound operation inside a handler — produces three completely different production incidents. This is the most useful thing to internalise about runtime models, because the debugging path differs entirely.
Notice that none of the three is "the code is slow". In each case the code takes the same time; what differs is who else pays for it.
1// --- Node: one JS thread per process ---2app.get('/report', (req, res) => {3 const csv = buildHugeCsvSynchronously(rows) // 500ms of pure CPU4 res.type('text/csv').send(csv)5})6// Effect: for 500ms this process runs NO other JavaScript.7// Every in-flight request, every timer, every pending accept and8// the health-check endpoint are all delayed by up to 500ms.9// Symptom: correlated latency across unrelated endpoints.10 11// --- Python, sync Gunicorn worker (N processes) ---12// @app.get("/report")13// def report():14// return build_huge_csv(rows) # 500ms of pure CPU15// Effect: exactly ONE worker is occupied. With 8 workers you have16// lost 1/8 of capacity. Requests queue only once all 8 are busy.17// Symptom: throughput drops, other requests unaffected until saturation.18 19// --- Go: goroutines across all cores ---20// func report(w http.ResponseWriter, r *http.Request) {21// csv := buildHugeCSV(rows) // 500ms of pure CPU22// w.Write(csv)23// }24// Effect: one goroutine occupies one OS thread on one core. On an25// 8-core machine you have lost ~1/8 of CPU capacity; the scheduler26// keeps running everything else.27// Symptom: CPU utilisation rises; latency degrades gradually.Same code, same duration, three different incidents: total correlated latency, a capacity fraction, and a gradual CPU cost. The fix — get CPU work off the request path — is the same in all three; the urgency is not.
Choosing by workload, not by preference
The decision below is the honest version. There is no winner, and the criteria are workload shape, isolation requirement and what your team can operate at 3am.
The strongest signal is the ratio of waiting to computing. A service that spends most of a request waiting on other systems is a different problem from one that spends it in your own code (Choosing a Runtime).
What does a typical request spend its time doing?
when Mostly waiting on I/O; many concurrent connections; long-lived connections such as websockets or SSE.
cost Every CPU-bound mistake is a service-wide incident; you need a discipline for offloading work (Blocking the Event Loop).
when Moderate concurrency, mixed workload, a team that values straightforward stack traces and debuggers.
cost Memory per thread caps concurrency; shared mutable state needs real synchronisation.
when High concurrency and blocking-style code, with a runtime that parks tasks on I/O.
cost A scheduler you must understand when it misbehaves; a blocking syscall can still consume an OS thread.
when Isolation matters, third-party code is untrusted, or the language has a global lock making threads unhelpful for CPU work (Python Runtime Models).
cost Memory per worker and a multiplied connection footprint; no shared in-process cache.
when The service is infrastructure — a proxy, gateway or storage layer — and predictable tail latency is the product (C++ Backend Services).
cost Development speed, memory-safety risk, and a much smaller pool of engineers who can maintain it.
How to build it
Most important first.
- Classify the workload before choosing anything: is a request mostly waiting (I/O-bound) or mostly computing (CPU-bound)? The models diverge almost entirely on that axis (Computing or Waiting? in Performance is the same distinction).
- Find your unit of concurrency and its count: loop threads, worker processes, pool threads, tasks. That number is your concurrency ceiling, and everything queues behind it (Resource Limits).
- Bound every downstream resource per unit — pool size per worker, in-flight external calls per process — and multiply out before you deploy. Workers times pool size is what the database sees (Connection Pools).
- Match the metric to the model: event-loop lag for a loop, busy-worker count and queue depth for workers, thread-pool saturation for a pool. The wrong metric shows a healthy service during an outage (The Metrics a Backend Must Emit).
- Keep CPU-heavy work off whatever the model uses to make progress — a worker thread, a separate process, or a job queue (Background Jobs).
- Do not migrate runtimes to fix a saturation problem you have not measured. The usual cause is an unbounded dependency or a blocking call, and both follow you to the new runtime.
What can go wrong
- Loop blocked: latency rises on every endpoint simultaneously, including health checks, and the process may be killed as unhealthy while doing useful work (Health Checks: Startup, Readiness, Liveness).
- Threads or workers all busy: new requests queue, latency climbs with no error, and the application looks idle because it is waiting rather than computing.
- Thread-pool exhaustion at a layer you did not know had a pool — file I/O, DNS resolution, or a driver's internal executor.
- Memory exhaustion from too many concurrent units: each thread's stack and each worker's heap are multiplied by the count you configured.
- The mitigation failing: adding workers to fix latency, which multiplies pool connections and moves the bottleneck into the database (Connection Pool Exhaustion).
- In-process shared mutable state races in every model that keeps requests in one process — including a single-threaded event loop, where an
awaitis a yield point at which another request can interleave and observe a half-updated value (Backend Races). - Multi-process models turn what would have been an in-process race into a cross-process one that no local lock can protect (Pessimistic Locking).
- Isolation differs by model, and it is a security property. Separate processes get separate address spaces, so a memory-disclosure bug is bounded by the worker; threads on one heap are not isolated from one another at all (Worker Processes).
- Request-scoped state stored in a module-level variable is shared by every concurrent request on the same thread or process — a straightforward path to serving one user's data to another (Stateless Services).
- A single-threaded runtime makes denial of service cheap: one request that occupies the loop for a long time affects every user, so input-size limits are a concurrency control as much as a validation rule (Transport Validation).
- Timing side channels behave differently per model; on a shared loop, one request's work is measurable in another request's latency.
- "Async is more scalable." Async is cheaper *per waiting request*. For CPU-bound work it adds scheduling and removes nothing, and a thread-per-request model with a bounded pool is often simpler and equally fast.
- "Single-threaded means it can only do one thing at a time." It runs one piece of *your code* at a time while holding thousands of connections. Waiting is not doing.
- "Threads are slow." Threads are expensive at high counts. At a few hundred concurrent requests, a thread-per-request server is entirely reasonable and much easier to debug.
- "The runtime handles concurrency for me." It provides a model. The limits, the blocking calls and the multiplied pools are yours.
- "CPU cores equal parallelism." Only if the runtime executes your code on more than one of them. Adding cores to a single-threaded process buys nothing directly (The Node Event Loop).
Operating it
- Instrument the saturation signal your model actually has: event-loop lag, busy workers versus total, active threads versus pool size, tasks queued.
- Graph concurrency alongside latency and throughput. When throughput flattens while concurrency climbs, you have found the ceiling — this is Little's Law in practice, not a metaphor.
- A CPU profile of a blocked runtime is diagnostic: a loop stuck in one synchronous function looks completely different from one that is genuinely busy (Why Is My API Slow?).
- Correlated latency across unrelated endpoints is a runtime-level signal. Uncorrelated latency on one endpoint is an application-level one, and the distinction saves hours.
- At 10x, the ceiling you were nowhere near becomes the thing you hit. The number of loop threads, workers or pool threads is a capacity plan you now have to write down.
- At 100x, the cost per *waiting* request dominates: models where waiting is nearly free hold far more concurrent connections per instance than models where waiting occupies a thread (Accepting Connections).
- Horizontal scaling papers over model differences up to the point where a shared dependency saturates. Then the multiplier — instances times units times pool size — decides everything (Horizontal vs Vertical Scaling).
- Event loops make waiting cheap and make every CPU-bound mistake catastrophic. Threads make CPU work safe and make each waiting request expensive.
- Process isolation buys crash containment and memory safety boundaries, and costs memory per worker plus a multiplied connection footprint.
- Lightweight tasks give blocking-style code with cheap concurrency, at the price of a runtime scheduler between you and the OS — which is another thing to understand when it misbehaves.
- Any model can be made to work for most workloads. The cost of the wrong one is not "it does not work", it is that your failure mode is surprising and your fix is unintuitive.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- RUNTIME-SPECIFICThe whole lesson is a comparison of runtimes. Node runs application code on one thread per process, so CPU work blocks every in-flight request in that process; Go schedules goroutines across all cores, so one busy goroutine costs a fraction of capacity rather than all of it; a synchronous Python worker serves one request at a time, so a slow request costs exactly one worker and nothing else.
- GENERALThe organising question — what happens to the other requests while this one waits — is universal, and the answer is what a runtime model is.
- SIMPLIFIEDFour clean categories over a landscape of hybrids. Every production runtime mixes them: loops with thread pools, thread pools with non-blocking I/O underneath, workers that are internally threaded.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.