The question this answers
What does running work in separate processes actually buy me, and what does it stop being able to protect?
A thumbnail service running an image decoder with a native C dependency, 40 jobs per second, where a malformed file can and does segfault the decoder.
Nothing in memory, by construction — no worker can reach another's heap. What is shared is everything outside memory: the input directory, the output bucket, the job table, the log file, and the operating system's file descriptors and ports.
Each job produces exactly one output object and exactly one terminal status row, and a crash while processing job N leaves jobs 1..N-1 completed and jobs N+1.. still processable.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
Two address spaces, and a hole you drilled on purpose
A process owns an address space. A pointer in worker A means nothing in worker B — not "means something dangerous", but genuinely nothing, because the hardware translates it through a different page table. Operating Systems covers how in [[process-anatomy]], [[process-isolation]] and [[process-memory-layout]]. The design consequence is the one that matters here: shared mutable memory is not something you must be disciplined about avoiding, it is something you cannot do by accident.
That inverts where the effort goes. Under threads, sharing is the default and every piece of it must be found and protected. Under processes, isolation is the default and every piece of sharing must be built — a pipe, a socket, a queue, a shared-memory segment you asked for explicitly. See [[ipc-overview]] and [[shared-memory]].
This is why "no shared state" is nearly true of a process model and never quite true. Every hole is visible in the code, which is the entire benefit — you can enumerate them in a review. The trap is that the enumeration usually stops at IPC and forgets the filesystem, the database and the object store, which are shared by all of them and protected by none of it.
The isolation is real, and it is exactly this wide
The demonstration people find surprising is the simplest one: a module-level variable mutated in a child is not mutated in the parent. Not "eventually", not "with a delay" — never. The child got a copy of the address space at fork time, or a fresh interpreter at spawn time, and the two have nothing to do with one another afterwards.
Everything you want to share has to be an object built for it — a queue, a value in a shared-memory block, a manager proxy — and every one of those costs serialisation on each access. A shared counter through a manager proxy is a round trip to another process; ten million of them will be several orders of magnitude slower than an in-memory increment. Fine-grained sharing is what a process model is bad at, and that is a design constraint rather than a tuning problem.
The start-mode caveat matters and bites people: with fork the child inherits a copy of everything including open descriptors and lock states; with spawn it gets a fresh interpreter that re-imports your module. Code that works under one can deadlock or double-execute under the other, and the default differs by platform and by version.
1import multiprocessing as mp2 3counter = 0 # module-level global4 5def bump(n):6 global counter7 for _ in range(n):8 counter += 1 # mutates THIS process's copy9 return counter # returned by value, serialised back10 11if __name__ == "__main__":12 with mp.Pool(4) as pool:13 results = pool.map(bump, [1000] * 4)14 15 print(results) # [1000, 1000, 1000, 1000] -- four separate copies16 print(counter) # 0 -- the parent's global was never touched17 18 # To actually share, you must ask for it, and then you own the invariant:19 shared = mp.Value("i", 0) # one integer in a shared memory segment20 def bump_shared(_):21 with shared.get_lock(): # NOT optional: shared.value += 1 is22 shared.value += 1 # read-modify-write, and processes race too23 # Isolation removed shared MEMORY. It did not remove shared FILES,24 # shared DATABASE ROWS or shared OBJECT KEYS -- see the schedule below.What process isolation does not isolate
The failure teams hit after moving to processes is always the same one, and it is the one the model quietly encourages: having removed shared memory, everyone stops thinking about shared state. But the two workers still write to the same bucket, still update the same job row, and still append to the same log.
The schedule below is a check-then-act race across two processes with no shared memory whatsoever. Both workers claim job 4471 because both read its status before either wrote it. The output object is written twice — harmless, since it is the same key — and the billing row is written twice, which is not. Separate address spaces contributed nothing, because the contended state was never in an address space.
The fixes are the ones from the database side of the world rather than the concurrency-primitive side: a conditional update that claims the row atomically (UPDATE ... WHERE status = 'pending' and check the affected-row count), a unique constraint, or a queue with visibility timeouts that does the claiming for you. [[database-concurrency]] and [[optimistic-concurrency-control]] are the relevant lessons, and [[local-lock-not-distributed]] is the trap of reaching for an in-process mutex here.
| # | Worker process 1 (pid 8801) | Worker process 2 (pid 8802) | State |
|---|---|---|---|
| 1 | SELECT status FROM jobs WHERE id=4471 → pending | · | jobs.4471=pending billingRows=0 |
| 2 | · | SELECT status FROM jobs WHERE id=4471 → pending | jobs.4471=pending billingRows=0 |
| 3 | UPDATE jobs SET status='running', worker=8801 WHERE id=4471 | · | jobs.4471=running/8801 billingRows=0 |
| 4 | · | UPDATE jobs SET status='running', worker=8802 WHERE id=4471 | jobs.4471=running/8802 billingRows=0 |
| 5 | decode + PUT thumbs/4471.jpg | · | jobs.4471=running/8802 billingRows=0 |
| 6 | INSERT billing (job 4471, 1 unit) | · | jobs.4471=running/8802 billingRows=1 |
| 7 | · | decode + PUT thumbs/4471.jpg | jobs.4471=running/8802 billingRows=1 |
| 8 | · | INSERT billing (job 4471, 1 unit) | jobs.4471=running/8802 billingRows=2 ✕ Job 4471 produced two billing records. Both processes claimed a row neither had exclusively, and neither has any memory the other could have corrupted. |
UPDATE jobs SET status='running' WHERE id=4471 AND status='pending', then act only if one row was affected — or a unique constraint on (job_id) in the billing table so the second insert fails loudly.Key points
- A process owns an address space; another process's pointers are meaningless in it. Shared mutable memory becomes impossible rather than merely discouraged.
- Every piece of sharing must be built explicitly — a pipe, a socket, a shared-memory segment — which makes the sharing enumerable in a code review.
- Crash containment is the headline feature: a segfault, an OOM or a native-library abort kills one worker and the supervisor reaps it.
- The cost is serialisation on every exchange, and startup in the tens of milliseconds rather than the tens of microseconds.
- Memory is multiplied by the worker count — separate heaps, separate runtimes, separate caches — and copy-on-write only helps until the pages are written.
- Isolation covers memory and nothing else. Files, database rows, object keys, ports and locks in the filesystem are shared by every worker.
- Explicitly shared memory (a
Value, ammapsegment) has exactly the same race problems as threads, with none of the language's help.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • The supervisor creates workers —
forkthenexec, or a fresh spawn — each receiving its own page tables and its own runtime. See[[process-creation]]. - • Work is dispatched by serialising a job descriptor onto a pipe or socket; the worker deserialises it into its own heap.
- • The worker executes with no reference to any other worker's memory, so its only failure surface is its own address space plus the external resources it opens.
- • Results are serialised back, or written directly to shared external storage and acknowledged by reference.
- • On a crash the kernel tears down the address space and the supervisor observes the exit status — signal 11 for a segfault, 137 for an OOM kill — and decides to retry, mark the job poisoned, or restart the worker.
- • P1 and P2 process disjoint jobs concurrently on separate cores: no shared memory, no locks, no interleaving to reason about — the model at its best.
- • P1 segfaults mid-decode on a malformed file; P2 is unaffected and completes normally; the supervisor sees exit code 139 and marks job 4471 poisoned. Under threads, the same segfault takes down every in-flight job in the process.
- • P1 and P2 both read job 4471 as pending, both claim it, both bill for it — the schedule above, with no shared memory involved anywhere.
- • P1 crashes after writing the output object and before writing the status row; a retry re-does the work and writes the object again. Idempotent by key, so harmless — which is the property that makes crash-and-retry safe at all. See
[[idempotency]]. - • Both processes hold a
Valuecounter and increment it without takingget_lock(): read 41, read 41, write 42, write 42. Explicit shared memory reintroduces the exact lost-update race the model was supposed to prevent.
- • The kernel guarantees one process cannot read or write another's memory without an explicit, permissioned mechanism.
- • It guarantees a crashing process is torn down completely, releasing its memory and its file descriptors, and that the parent can observe how it died.
- • It does not guarantee isolation of anything outside memory: files, sockets, database rows and object keys are shared and unprotected.
- • It does not guarantee the child is a faithful copy of the parent. Under
spawnthe module is re-imported; underforkin a multi-threaded parent, only the calling thread survives, which is how a child inherits a mutex that is locked forever. - • It does not guarantee ordering between workers, and it does not make explicitly shared memory safe — a
Valueneeds a lock exactly as a global would.
- • Contention moves outside the process: the database, the object store, the log file, the connection limit. Sixteen workers times ten connections is a hundred and sixty connections.
- • The dispatch pipe is a contention point at high job rates, and its buffer is a bounded queue whose full state is backpressure.
- • Memory is contended at the machine level: sixteen workers each with a 300 MB runtime is 4.8 GB before any work is done.
- • Explicitly shared memory is contended exactly like threaded shared memory, including false sharing on the same cache lines.
- • Orphaned children: the supervisor dies and the workers keep running, holding connections and writing output nobody expects.
- • Zombie accumulation when exit statuses are never reaped, exhausting the process table.
- • Double-claim races on external shared state, as in the schedule — the characteristic bug of a process model.
- • Fork-in-a-threaded-parent deadlock: the child inherits a mutex that was held by a thread that does not exist in the child, and the first allocation hangs forever.
- • IPC dominance: fine-grained sharing through a manager proxy, where serialisation costs orders of magnitude more than the work.
- • Memory exhaustion from worker count multiplied by per-worker footprint, with the OOM killer selecting the largest worker mid-job.
- • Untrusted or crash-prone code: native decoders, PDF parsers, user-supplied scripts, anything where a segfault is a realistic Tuesday.
- • Runtimes where threads cannot execute your code on separate cores, making processes the only path to CPU parallelism. See
[[python-threads-vs-processes]]. - • Long-running coarse-grained jobs where a 30 ms startup and a serialisation round trip round to nothing against 90 seconds of work.
- • Memory-leak containment: a worker recycled every N jobs bounds a leak you have not found yet, which is an unglamorous and extremely effective production tactic.
- • When you want the option of moving a worker to another machine later — a process boundary is already the shape of that move.
- • Fine-grained work: thousands of small tasks per second, where serialisation and startup dominate.
- • Large shared read-only data: every worker gets its own copy, and 2 GB of reference data times eight workers is 16 GB. Copy-on-write helps until a garbage collector touches the pages.
- • Chatty coordination: workers that must frequently agree on shared state spend their time in IPC rather than in work.
- • Tight memory budgets, especially in containers where the limit is per-container and the OOM killer is per-cgroup. See
[[resource-envelope]].
- • Child exit codes and their distribution — signal 11, signal 9, non-zero returns. This is the direct evidence that isolation is doing work rather than merely costing money.
- • Worker restart rate over time; a rising rate is a poison-input problem or a leak, and the two look identical until you correlate with job ids.
- • Time from dispatch to first byte of work, which isolates serialisation and startup cost from the work itself.
- • Total RSS across workers against the container limit, with a margin, because the OOM kill is a cliff rather than a slope.
- • IPC bytes per job. If it is comparable to the work being done, the process boundary is in the wrong place.
- • Everything shared must be serialisable, which constrains what a job descriptor can contain — no open connections, no callbacks, no live handles.
- • You now operate a supervisor: spawn, health, restart policy, backoff, graceful shutdown and reaping are all yours to write or configure.
- • Debugging spans processes: a stack trace covers one address space, and correlating across workers needs job ids threaded through every log line.
- • Configuration multiplies: pool size, per-worker memory limits, restart thresholds and IPC buffer sizes, each with its own failure mode.
- • Startup and warm-up costs are paid per worker — JIT warm-up, connection pools, caches — which is why worker recycling has to be tuned rather than set to 1.
- • Threads, when the code is trusted, the work is fine-grained, and the runtime can actually use cores. See
[[threads]]and[[thread-vs-process]]. - • One process with a bounded worker pool inside it, if crash containment is not a real requirement — most business logic does not segfault.
- • Containers or sandboxes when the isolation requirement is about security rather than crashes; a process boundary is not a security boundary on its own. See
[[containers-and-the-os]]. - • A queue plus separate deployments, when the workers do not need to be children at all — this gets isolation, independent scaling and independent deploys for the same conceptual price.
Server model lab
Each model is the same simulator given a different worker shape: a thread per in-flight request, a fixed pool, or one task per core where waiting does not occupy a worker. Memory is a per-thread stack estimate. Real servers differ by orders of magnitude in all of these, and every runtime has its own hybrids. Concurrency in flight is the knob; offered load is derived from it as concurrency ÷ service time.
Concurrency lab
They come from a queueing and contention model inside Engineer Atlas. What is faithful is the behaviour: work that waits benefits from more workers, work that computes does not, a wide critical section pins parallelism near 1 no matter how many cores you buy, and arrivals past capacity produce an unbounded queue rather than a large latency. Real arrivals are burstier than this model assumes, so real systems reach every one of these walls earlier than the sliders suggest. Do not quote a millisecond from this page.
What people believe, and what is true
Processes have no shared state, so there are no races.
They have no shared *memory*. The job table, the object store, the log file and the filesystem are shared by every worker, and the classic check-then-act race lives there.
Processes are just expensive threads.
They buy crash containment and, in some runtimes, the only access to multiple cores. For a 90-second job the extra 30 ms of startup is a rounding error and the containment is the product.
A process boundary is a security boundary.
It is a memory boundary. Without namespaces, seccomp, a distinct uid and filesystem restrictions, a compromised worker still reaches everything the user can reach.
Go deeper
Overview
Each process gets its own memory. Nothing leaks between them by accident, everything shared must be sent explicitly, and one crashing does not take down the others.
Practical
Use processes when code can crash, when threads cannot use cores, or when jobs are coarse enough that serialisation is noise. Then go and find the state that is still shared: rows, files, keys.
Advanced
A process model relocates concurrency risk rather than removing it. In-memory races become impossible and external-resource races become the only kind, which is a good trade because external stores offer atomic primitives — conditional updates, unique constraints, leases — that shared memory does not.