The question this answers
Given what I now know about this work, which execution model should carry it — and what am I signing up for?
A decision, applied to three concrete candidates: a 40-line CSV import script, a payments API handler that calls four upstreams, and a video transcoder that runs ffmpeg for 90 seconds per job.
It depends on the answer, and that is the point: sequential shares nothing because there is one actor; threads share the entire heap by default; processes share only what you explicitly send; message passing shares only ownership that has been transferred.
Whatever model is chosen, the work completes exactly once, its result is observed by whoever asked for it, and no failure in one unit of work silently discards another's.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
Seven questions, in order
The order matters because early answers eliminate whole branches. If the work is small, nothing else in the list is worth asking. If it needs isolation, threads are out no matter what the other six say. Run them top to bottom and stop at the first one that decides.
The most load-bearing question is the third: is the work independent? Independence is what makes parallelism cheap and shared-state bugs impossible, and engineers routinely assume it where it does not hold. Two "independent" jobs that both touch the same row in the same table are not independent; they are two writers with a race and a database in the middle. See [[database-concurrency]].
The last question — isolation — is the one that overrides everything above it. Running untrusted code, code that can segfault, or code with a native dependency that leaks means a crash must not take the parent down, and that requirement points at processes regardless of how convenient threads would have been. See [[processes]].
The seven answers, with their bills
Each row is a complete recommendation: why it wins, what it costs and how it fails. A model chosen without the third column will be chosen again, six months later, during an incident.
Two rows deserve a note. Message passing is the only model here that removes shared mutable state rather than protecting it — its bugs are about queue depth, ordering and delivery rather than about interleaving, and that is a genuinely easier class of bug. And the sequential row is not a consolation prize: it is the only row with an empty failure-modes column, and it is the correct answer far more often than a domain called "Concurrency & Parallelism" makes it look.
| Model | Choose it when | Why it wins | What it costs | How it fails |
|---|---|---|---|---|
| Sequential | The work is small, or ordering between items is the requirement | No coordination, no interleavings, fully deterministic, trivially debuggable | Wall clock is the sum of everything, including every wait | It does not — it is just slow, visibly and predictably |
| Async / event loop | Waiting-bound, tens of thousands of concurrent units, short handlers | Thousands in flight per thread; memory per unit is a task frame, not a stack | Colours the whole call graph; every await is an interleaving point | One CPU-bound handler stalls every task on the loop; unbounded tasks grow the heap until OOM |
| Threads + explicit synchronisation | Compute-bound units that must share a mutable structure | True parallelism with direct shared-memory access and no copying | Every shared field needs an invariant and a lock; the memory model is now yours to know | Data races, lost updates, deadlock, lock convoys, false sharing |
| Bounded worker pool | Many short compute-bound or blocking tasks; you need a cap | Bounds resource use, amortises thread creation, makes the queue visible and measurable | A sizing decision with no universal formula; a queue that must be bounded too | Saturation with a growing queue; pool deadlock when a pooled task waits on another pooled task |
| Processes | Isolation, crash containment, untrusted or leaky code, or threads that cannot use cores | A crash, a leak or a segfault stays inside one address space; separate interpreters and heaps | Startup cost in the tens of milliseconds; everything shared must be serialised explicitly | IPC serialisation dominates for chatty work; orphaned children; memory multiplied by the process count |
| Message passing / channels | Units that exchange data rather than share it — pipelines, actors | No shared mutable state to protect; ownership moves with the message | A copy or a move per message; buffering policy becomes a design decision | Unbounded queue growth; deadlock on a full channel; message loss on shutdown |
| Parallel algorithm | One large partitionable computation over data already in memory | The highest ceiling available; the parallel phase needs no synchronisation at all | Partitioning and merge logic; nondeterministic float results; bandwidth ceiling | Off-by-one chunk boundaries; load imbalance; false sharing; slower than sequential on small inputs |
What choosing wrong actually looks like
The payments handler calls four upstreams and is plainly waiting-bound, so it went on the event loop — correct. Then a requirement arrived: sign the request body with RSA-4096 before dispatch, about 35 ms of pure computation. Nobody reclassified. The signing went inline, on the loop.
The schedule below is what 35 ms on a shared loop does at 300 requests per second. It is not a race in the classical sense; nothing is corrupted. The invariant it breaks is a liveness one, and liveness invariants are the ones that page you: the health-check handler is *ready* for 180 ms and never runs, so the orchestrator kills a process that is doing exactly what it was asked to do.
The fix is the same one classification would have produced: signing goes to a small worker pool, the loop keeps the socket work, and there is now a bounded queue you can put a graph on. See [[worker-threads]] and [[bounding-concurrency]].
| # | Event loop | Payment requests (queued) | Health check (queued at t=10ms) | State |
|---|---|---|---|---|
| 1 | accept 6 payment requests; all 6 ready | · | · | ready=6 loopBusyMs=0 |
| 2 | run request 1: sign body (RSA-4096) | · | · | ready=5 loopBusyMs=35 |
| 3 | · | · | health check arrives, marked ready | ready=6 loopBusyMs=35 healthWaitMs=0 |
| 4 | run request 2: sign body | · | · | ready=5 loopBusyMs=70 healthWaitMs=35 |
| 5 | run request 3: sign body | · | · | ready=4 loopBusyMs=105 healthWaitMs=70 |
| 6 | run request 4: sign body | · | · | ready=3 loopBusyMs=140 healthWaitMs=105 ✕ The health check has been ready for 105 ms without running. The liveness budget of 100 ms is blown while the process is entirely healthy and doing useful work. |
| 7 | run request 5: sign body | · | · | ready=2 loopBusyMs=175 healthWaitMs=140 |
| 8 | · | · | orchestrator liveness probe times out; SIGTERM sent | ready=2 loopBusyMs=175 healthWaitMs=140 |
| 9 | run health check — replies 200 OK | · | · | ready=1 loopBusyMs=178 healthWaitMs=0 |
Key points
- Run the seven questions in order and stop at the first that decides: size, bound, independence, ordering, count, duration, isolation.
- Sequential is the only model with an empty failure-modes column. It should be beaten, not skipped.
- Independence is the question people get wrong. Two units that touch the same row are not independent.
- Isolation overrides everything above it: untrusted, crash-prone or leaky code means processes, whatever the other answers say.
- Message passing is the only model that removes shared mutable state instead of protecting it, and it trades interleaving bugs for queueing bugs.
- Every model choice must come with its failure mode, or the choice will be re-litigated during an incident.
- Liveness invariants — "a ready task runs within N ms" — are broken by CPU work on a shared loop even when nothing is corrupted.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • Estimate the work: if the unit is under a few milliseconds and not repeated at volume, choose sequential and stop.
- • Classify each phase as compute-bound or waiting-bound. See
[[classifying-the-work]]. - • Test independence honestly: list every piece of state two units both touch, including rows, files, caches and counters.
- • Decide whether output order is a requirement; if so, plan to restore it at the join rather than to preserve it during execution.
- • Estimate the concurrent unit count. Tens points at a bounded pool; tens of thousands points at async, because stacks do not scale to that.
- • Ask whether a unit is long-running, which turns pool sizing into a queueing problem and makes cancellation mandatory.
- • Ask whether a crash must be contained. If yes, choose processes and pay the serialisation cost deliberately.
- • Async chosen for a waiting-bound handler: request 1 suspends on an upstream, request 2 runs, request 1 resumes — the model working as intended.
- • Async chosen and a CPU phase added later: request 1 signs for 35 ms while the health check sits ready; nothing is corrupted and the process is killed anyway.
- • Pool chosen, and a pooled task submits another task to the same pool and waits for it: with all slots occupied by waiters, nothing can ever complete. This is pool deadlock, and it needs no locks at all. See
[[pool-saturation]]. - • Threads chosen for units that turn out to share a cache entry: thread A reads the entry, thread B evicts and replaces it, thread A writes back the stale object — a lost update through a structure nobody thought of as shared.
- • Processes chosen for isolation: the child segfaults on a malformed video, the parent observes exit code 139, marks the job failed and continues. Under threads the same bug takes down every in-flight job in the process.
- • Choosing a model guarantees which failure classes are *possible*, not which occur. Async removes data races and keeps race conditions; processes remove both and keep protocol races over the channel.
- • A bounded pool guarantees an upper bound on concurrent work. It guarantees nothing about latency once the queue is non-empty, and nothing at all if a pooled task can block on another pooled task.
- • Processes guarantee memory isolation and crash containment. They do not isolate shared files, shared databases, shared caches or shared locks in the filesystem.
- • Message passing guarantees no shared mutable state along the channel. It does not guarantee delivery, ordering across multiple channels, or that a full channel will not deadlock the sender.
- • No model guarantees that the work needed to be concurrent at all. That question is answered by
[[concurrency-has-a-cost]], not by the model.
- • Sequential: none, by construction.
- • Async: contention is for the single loop thread, and it is total — one busy handler is every handler waiting.
- • Threads: contention is for locks, cache lines and memory bandwidth, in roughly that order of visibility and reverse order of diagnosability.
- • Pool: contention is for slots, and it is visible as queue depth and queue age, which is exactly why the pool is worth having. See
[[queue-age]]. - • Processes: contention is for memory and for whatever external resources they all open — 16 workers times 10 connections is 160 connections, and the database has a limit.
- • Message passing: contention is for buffer space, and a full buffer is backpressure, which is a feature until nobody handles it.
- • Event-loop starvation: CPU work on a shared loop breaks liveness for everything, including health checks, without breaking correctness.
- • Pool deadlock: pooled tasks waiting on pooled tasks with every slot occupied — no lock involved and no cycle in any wait-for graph you drew.
- • Thread oversubscription: hundreds of runnable threads on eight cores, where throughput falls as concurrency rises.
- • IPC dominance: a process model chosen for isolation, applied to chatty fine-grained work, where serialisation costs more than the work.
- • Unbounded queue: any model plus an unbounded intake converts an overload into an out-of-memory kill. See
[[unbounded-concurrency]]. - • Silent order loss whenever a parallel model is applied to work whose consumers assumed input order.
- • At design time, when the choice is free. Reversing an async decision after it has coloured 300 functions is a project.
- • During an incident review, as the question "was this the right model?" rather than "which lock is missing?" — the second question assumes the first was answered.
- • When a service has grown a phase it did not have originally, which is the single most common way a once-correct model becomes wrong.
- • When it becomes an architecture exercise for a script that runs nightly and takes four seconds.
- • When the tree is followed mechanically past the point where the answer was already "sequential".
- • When the model is chosen for the system rather than for the phase, which is how a correctly-async service acquires a 35 ms signing step.
- • Event-loop lag or scheduler run-queue delay: the direct measurement of "a ready task is not running", which no application-level timer captures.
- • Queue depth and queue age at every hand-off — depth alone is a snapshot, age is the one that tells you whether you are draining. See
[[queue-age]]. - • Per-core utilisation, to check that a parallel model is actually occupying cores rather than queueing on one.
- • Concurrent-unit gauge against throughput, to find where the model stops buying anything.
- • Child process exit codes and restart counts, which are the only evidence that isolation is being used rather than merely paid for.
- • Every model except sequential adds a coordination artefact — a loop, a pool, a channel, a join — that has its own configuration, its own metrics and its own failure modes.
- • Mixed models within one service double the vocabulary: a stall can be pool saturation, loop lag, channel backpressure or upstream latency, and telling them apart needs instrumentation for each.
- • The choice propagates into API shape: async colours callers, worker boundaries force serialisable payloads, and processes force everything shared to become a protocol.
- • Cancellation and timeouts must be designed per model, and they are the part teams skip, which is how orphaned tasks and abandoned children accumulate. See
[[structured-concurrency]].
- • Do not choose: keep it sequential and make the work smaller or the algorithm better. This wins more often than the tree suggests.
- • Let a framework choose. A server framework already picked thread-per-request or event-driven; fitting your work to it beats fighting it.
- • Scale out instead of in: N single-threaded processes behind a load balancer gets you core utilisation with no in-process concurrency at all. See
[[horizontal-vs-vertical-scaling]]. - • Move the work to a system built for it — a database aggregate, a stream processor, a transcoding service — rather than building a concurrency model inside your application.
Concurrency lab
They come from a queueing and contention model inside Engineer Atlas. What is faithful is the behaviour: work that waits benefits from more workers, work that computes does not, a wide critical section pins parallelism near 1 no matter how many cores you buy, and arrivals past capacity produce an unbounded queue rather than a large latency. Real arrivals are burstier than this model assumes, so real systems reach every one of these walls earlier than the sliders suggest. Do not quote a millisecond from this page.
Server model lab
Each model is the same simulator given a different worker shape: a thread per in-flight request, a fixed pool, or one task per core where waiting does not occupy a worker. Memory is a per-thread stack estimate. Real servers differ by orders of magnitude in all of these, and every runtime has its own hybrids. Concurrency in flight is the knob; offered load is derived from it as concurrency ÷ service time.
Thread pool: utilization and queue
capacity = workers / service = 8 / 50 ms = 160.0 req/s ρ = arrivals / capacity = 120 / 160.0 = 0.750 Little L = λ × W → 0.120/ms × 59.8 ms = 7.2 in flight engine status = healthy
Bounding concurrency with permits
| permits | goodput | mean latency | timeouts | failed of 10K |
|---|---|---|---|---|
| 1 | 25/s | 43 ms | 0.00% | 0 |
| 5 | 125/s | 43 ms | 0.00% | 0 |
| 10 | 250/s | 43 ms | 0.00% | 0 |
| 25 | 625/s | 43 ms | 0.00% | 0 |
| 50 | 1000/s | 53 ms | 0.00% | 0 |
| 100 | 1000/s | 103 ms | 0.00% | 0 |
| 200 | 1000/s | 203 ms | 0.00% | 0 |
| 350 | 1000/s | 353 ms | 0.00% | 0 |
| 500 | 0/s | 503 ms | 100.0% | 10K |
What people believe, and what is true
Threads are the general-purpose answer and everything else is a special case.
Threads are the model with the most failure modes in the table. They are the right answer specifically when compute-bound units must share a mutable structure — a narrower case than their popularity suggests.
Processes are just slow threads.
They buy crash containment, memory isolation and — in some runtimes — the only path to using more than one core. The startup and serialisation costs are what you pay for those, and for long-running units they round to nothing.
If CPU is low, the model is fine.
A single-threaded event loop pinned at 100% reads as 12% on an eight-core box. Aggregate CPU cannot see loop starvation; only loop lag can.
Go deeper
Overview
Seven questions in order: is it big, is it computing or waiting, are the units independent, does order matter, how many, how long, must a crash be contained. Stop at the first that decides.
Practical
Write down the chosen model together with its cost and its failure mode. A model recorded without those is a preference, and it will be argued about again in six months.
Advanced
Models are chosen per phase, not per service, and phases are added over time. The dangerous state is a service whose model was correct when written and whose workload has since acquired a phase that contradicts it — which is why loop lag and queue age belong on the dashboard from day one.