The question this answers
The queueing shape is identical whether a worker is a thread or a machine — so what actually changes?
Ten thousand PDF render jobs per hour, each 2–90 seconds of CPU and 400 MB of peak memory, submitted by a web tier that must answer in 200 ms.
The job queue, which for non-thread workers is external to every worker's address space — a broker, a database table or a managed queue. Nothing else is shared: separate processes and machines share no heap, so all coordination is message passing (Message Passing).
Every accepted job eventually reaches a terminal state (completed or dead-lettered), no job is lost when a worker dies mid-execution, and no job's externally visible side effects are applied twice.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
The same shape, four substrates
Once you see a pool as "queue plus bounded set of consumers", the thread version stops being special. A process pool, a container-based worker deployment and a fleet of machines pulling from a managed queue are all the same diagram. What changes is the distance between the queue and the worker, and everything expensive follows from that distance.
For a thread pool, the queue is an object on the same heap; hand-off is a pointer copy and costs nanoseconds. For a process pool it crosses a pipe or shared memory segment and the payload must be serialized. For containers and machines it crosses a network to a broker, and now the hand-off can fail *after* the job was taken and *before* the result was recorded — which is the entire source of the at-least-once problem below.
The PDF workload above is the case where escaping threads is correct rather than fashionable. 400 MB of peak memory per job means eight concurrent jobs is 3.2 GB in one address space, one leak or one malformed PDF takes the whole process down with every in-flight job, and a 90-second job cannot be interrupted safely. Separate processes give you memory isolation, kill-ability and a hard resource ceiling per worker, at the cost of serialization and startup.
- Same structure: bounded consumers, a queue that absorbs bursts, a rejection or dead-letter path.
- Different hand-off cost: pointer copy → serialization → network round trip.
- Different blast radius: a thread crash takes the process; a container crash takes one job.
- Different cancellation: a thread cannot be safely killed, a process can, a container definitely can.
What changes: startup, isolation, failure, cancellation
The four axes below are the decision. Startup cost sets the minimum sensible task duration — spending 3 seconds starting a container to do 40 ms of work is absurd, and spending 200 µs forking a process to do 40 ms of work is merely wasteful. Isolation sets what one bad job can destroy. Failure semantics set whether you need idempotency. Cancellation sets whether a runaway job is a nuisance or an outage.
The rule of thumb worth carrying: as the worker gets heavier, the failure model gets better and the hand-off gets worse. Threads are cheap and fragile; remote workers are expensive and survivable. There is no substrate that is cheap and survivable, and choosing one means choosing which of those two you need.
Note the row that surprises people: only the thread pool gives you shared memory, and that is a *liability* as often as a feature. Every lesson in Shared Mutable State applies to a thread pool and none of it applies to a process pool, because separate processes cannot race on a heap they do not share. Moving to processes deletes a whole category of bug and replaces it with serialization cost.
| Worker kind | Startup cost | Isolation | What a worker death costs | Cancellation | Hand-off |
|---|---|---|---|---|---|
| Thread | Sub-millisecond | None — shared heap, shared fate | Usually the whole process, including every other in-flight job | Cooperative only; no safe forced kill | Pointer copy |
| Process | Milliseconds to ~a second | Separate address space; OS-enforced memory limit | One job; parent respawns and requeues | A signal, then SIGKILL — genuinely forcible | Serialize over pipe or shared memory |
| Container | Seconds (image pull can dominate) | Process isolation plus filesystem and network namespace | One job; the orchestrator reschedules | Orchestrator terminates the container | Network to a broker |
| Remote machine | Seconds to minutes (boot, join, warm) | Complete — separate hardware and failure domain | One job, plus the capacity until replacement | API call, or the lease simply expires | Network to a broker |
The failure that only exists off-heap: at-least-once delivery
A thread pool loses a task only if you wrote the queue wrong. A distributed worker pool loses or duplicates tasks *by design*, because the queue and the worker cannot atomically agree on "taken" and "done" across a network partition. The standard mechanism is a lease: the broker hands the job out with a visibility timeout, and if no acknowledgement arrives before it expires, the job becomes visible again and another worker takes it.
That mechanism is correct and it is also the schedule below. Worker A did not die — it was slow, or its host paused, or its clock and the broker's disagreed. The job is now running twice, concurrently, and unless the side effect is idempotent, the customer gets two invoices.
The fix is not a longer timeout; a longer timeout only makes real failures slower to recover. The fix is making the side effect idempotent — an idempotency key on the write, a conditional update, or a compare-and-swap on a job status row so the second finisher loses. This is Optimistic Concurrency Control applied at the job level, and it is the reason "exactly once" is a property of your *effects*, not of your queue.
| # | Worker A | Broker | Worker B | State |
|---|---|---|---|---|
| 1 | receive job#412, lease 30 s | · | · | job#412=invisible lease holder=A invoices written=0 |
| 2 | begin render (host is under memory pressure; the process stalls) | · | · | A progress=stalled at 12 s |
| 3 | · | lease expires at t=30 s, job#412 becomes visible | · | job#412=visible lease holder=none |
| 4 | · | · | receive job#412, lease 30 s | lease holder=B concurrent executions=2 ✕ Two workers now execute job#412 concurrently; the exactly-once assumption is already dead. |
| 5 | · | · | render completes, write invoice | invoices written=1 |
| 6 | · | · | ack job#412 | job#412=deleted |
| 7 | unstalls, render completes, write invoice | · | · | invoices written=2 ✕ Second invoice written for one job — a duplicate side effect, not a lost one. |
| 8 | ack job#412 → broker reports unknown receipt handle | · | · | A sees=error it cannot act on |
Key points
- The queueing shape is identical across threads, processes, containers and machines; startup cost, isolation, failure semantics and cancellation are what differ.
- As the worker gets heavier the failure model improves and the hand-off gets more expensive — there is no cheap-and-survivable substrate.
- Only thread pools share a heap, so moving to processes deletes a whole class of race but adds serialization cost.
- Off-heap pools deliver at-least-once by design; a slow worker, not a dead one, is the common cause of duplicate execution.
- Exactly-once is a property of your side effects (idempotency keys, conditional writes), never of the queue.
- Startup cost sets the minimum sensible task size: seconds of startup demand tasks measured in minutes.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • A producer enqueues a serialized job description — for off-heap workers everything must survive serialization, which rules out closures, sockets and file handles.
- • A bounded set of workers polls or subscribes; the broker hands each job to one worker with a lease (visibility timeout).
- • The worker executes; the lease must be renewed (heartbeat) or the job returns to the queue.
- • On success the worker acknowledges, which is the point the job is finally removed.
- • On failure or timeout the job becomes visible again and is retried, with an attempt counter.
- • After N attempts the job is routed to a dead-letter destination rather than retried forever — the only place a poison job is visible.
- • Worker A stalls past its 30 s lease; the broker re-delivers to Worker B; both complete; the non-idempotent invoice write happens twice.
- • Worker A completes and writes the result, then crashes before acknowledging; the job is redelivered and the result is written again — the same duplicate from the opposite direction.
- • A worker acknowledges before doing the work (to "avoid duplicates"); the worker then dies and the job is gone forever — trading duplicates for loss, which is almost always the worse trade.
- • A container is terminated by the orchestrator mid-job with a 10 s grace period; the worker catches the signal, stops renewing the lease, and lets the job be redelivered promptly instead of waiting for expiry — graceful failure by design (Draining a Pipeline).
- • Two workers hold leases on two different jobs that update the same row; the queue guaranteed nothing about that, and ordinary database concurrency control applies.
- • Guaranteed: bounded concurrency, exactly as for a thread pool — the number of workers is the number of jobs in flight.
- • Guaranteed by most brokers: at-least-once delivery. Every accepted job is delivered until acknowledged or dead-lettered.
- • NOT guaranteed: exactly-once execution. It is unachievable across a network without idempotent effects or a transaction spanning broker and side effect.
- • NOT guaranteed: ordering. Most managed queues offer FIFO only within a partition or key, and retries reorder anything.
- • NOT guaranteed: that a worker holding a lease is alive. The broker knows only whether an acknowledgement arrived.
- • NOT guaranteed: that a killed container's partial work was undone. Cancellation kills the process, not the half-written file it left in object storage.
- • Workers contend on the broker: high poll rates against one queue become the bottleneck long before the workers are busy.
- • They contend on shared downstream resources far more visibly than threads do, because a fleet can be scaled to numbers a thread pool never reaches — 400 containers against a 100-connection database.
- • Long polls and leases contend on nothing locally, which is exactly why the local metrics look perfect while the system is stuck.
- • Container startup contends on the image registry and the node pool: a scale-out event is a thundering herd against both (Thundering Herd).
- • Duplicate execution from lease expiry under a slow worker — the headline failure, and the one that looks like a broker bug.
- • Job loss from acknowledging before completing, which is the "fix" people reach for after seeing duplicates.
- • Poison jobs retried indefinitely because no dead-letter destination was configured, consuming the whole pool.
- • Orphaned partial side effects when a container is killed mid-write (Orphaned Tasks).
- • Silent capacity loss: workers that die and are not replaced, so the pool shrinks and only queue age reveals it.
- • Serialization failures at the boundary — a job that carries something unserializable fails at enqueue time or, worse, at decode time on the worker.
- • When per-job memory or CPU is large enough that one bad job must not take the others down — 400 MB PDF renders are the archetype.
- • When jobs must survive process restart and deploy: an in-memory thread pool loses everything, a durable queue loses nothing.
- • When capacity must scale independently of the request path, so the web tier returns 202 and the fleet absorbs the burst.
- • When jobs need forcible cancellation, which threads cannot safely provide.
- • When the language runtime limits in-process CPU parallelism, making processes the straightforward answer (Python: Threads, Processes and the GIL).
- • When tasks are short. A 40 ms task behind a 3-second container start or a network round trip is dominated by the substrate.
- • When jobs need shared in-memory state — you have replaced a pointer dereference with a serialization round trip and a cache invalidation problem.
- • When effects cannot be made idempotent and duplicates are unacceptable; you have imported a hard distributed-systems problem to solve a local one.
- • When the operational surface (broker, dead-letter monitoring, fleet lifecycle, deploy coordination) exceeds the problem you were solving.
- • Queue age at p99 — for off-heap pools this is the primary health signal, far above worker CPU.
- • Redelivery rate and attempt-count distribution: a rising redelivery rate with a flat error rate is the lease-expiry signature.
- • Dead-letter depth, which should be alerted on at any non-zero value rather than graphed.
- • Duplicate-effect count from the idempotency layer — the number of times a key collided is direct evidence the schedule above is happening.
- • Worker fleet size against desired size, to catch silent shrinkage.
- • Time from enqueue to first attempt, split from time in execution: they have completely different remedies.
- • You have added a broker with its own availability, its own limits and its own failure modes to a problem that previously fit in one process.
- • Every job description must be serializable, versioned, and forward-compatible with workers running older code during a deploy.
- • Idempotency stops being optional: every side effect needs a key, a conditional write, or a status transition guarded by a compare-and-swap.
- • Debugging spans machines. The stack trace no longer contains the submitter, so correlation ids are mandatory rather than nice.
- • Deploys become rolling and mid-flight jobs straddle versions, so schema changes need two-phase rollouts.
- • A thread pool, when jobs are small, trusted, and losing them on restart is acceptable — vastly less machinery for the same shape.
- • A process pool on one machine, when you need memory isolation and forcible cancellation but not durability or horizontal scale.
- • Running the work inline in the request, when it is fast and the caller can wait — an async job pipeline for 30 ms of work is pure overhead.
- • A managed function platform per job, when the work is spiky and stateless — the provider owns the fleet lifecycle you would otherwise build.
- • A database table as the queue, when volume is modest:
SELECT ... FOR UPDATE SKIP LOCKEDgives leases with a system you already operate (The Database Solves Concurrency For Its Data, Not For Your Memory).
Server model lab
Each model is the same simulator given a different worker shape: a thread per in-flight request, a fixed pool, or one task per core where waiting does not occupy a worker. Memory is a per-thread stack estimate. Real servers differ by orders of magnitude in all of these, and every runtime has its own hybrids. Concurrency in flight is the knob; offered load is derived from it as concurrency ÷ service time.
Thread pool: utilization and queue
capacity = workers / service = 8 / 50 ms = 160.0 req/s ρ = arrivals / capacity = 120 / 160.0 = 0.750 Little L = λ × W → 0.120/ms × 59.8 ms = 7.2 in flight engine status = healthy
The producer is faster than the consumer
What people believe, and what is true
Our queue guarantees exactly-once delivery, so duplicates cannot happen.
Delivery and execution are different things. A worker can execute, then fail before acknowledging. Exactly-once *effects* require idempotency on your side, always.
Duplicates mean a worker crashed.
The common cause is a worker that was merely slow — a GC pause, memory pressure, a stalled host — past its lease. The broker cannot tell slow from dead.
A process pool is just a thread pool with more overhead.
It has a different correctness model: no shared heap means no data races and no shared caches, forcible cancellation is available, and everything must serialize.
Go deeper
Overview
A worker can be a thread, a process, a container or a machine. Same queue, same bound, very different costs when one dies.
Practical
Pick the lightest substrate whose failure model you can live with. Off-heap means at-least-once, which means idempotency keys and a dead-letter destination before you ship.
Advanced
The lease is the whole protocol. Heartbeat long jobs, stop renewing on shutdown so redelivery is prompt, and guard side effects with a conditional write so the loser of a duplicate race is harmless.
Internals
Fork-based pools share pages copy-on-write and are unsafe once other threads exist (locks can be inherited held); spawn-based pools re-import the program and pay full interpreter startup per worker.