Thread per Connection
Give every client its own thread and let the scheduler interleave them: the code stays sequential and the OS supplies the concurrency — until the number of threads becomes the workload.
The problem
The model, and why it is so natural
The main thread does one thing: accept(), spawn, repeat. Each spawned thread owns one socket and one sequential handler: read the request, do the work, write the response, close. When a handler blocks in recv(), the kernel parks that thread and runs another. Concurrency is entirely the scheduler’s job (see The Scheduling Problem); the programmer writes straight-line code and gets isolation between clients for free.
It is also parallel when the work is CPU-bound: with eight cores and eight busy handlers, eight requests progress at once (see Concurrency versus Parallelism). This is why the model dominated server programming for two decades — Apache’s prefork/worker MPMs (processes, then threads), Java servlet containers, every early database server — and why it remains correct for services with hundreds of connections and real work per request.
The costs are all per thread, and they are the subject of the rest of this lesson: memory, kernel state, creation latency, scheduling overhead, and the shared-state bugs that appear when handlers touch the same data (see Race Conditions).
What a thread costs in memory — honestly
The number everyone quotes is the default stack size: 8 MB on Linux (ulimit -s), used by pthread_create unless overridden. 100,000 threads × 8 MB = 800 GB, which sounds like the end of the story. It is not, because that 8 MB is virtual address space, reserved with mmap and backed by physical pages only when touched (see The Virtual Address Space and Page Faults). A thread that runs a shallow handler touches perhaps 8–32 kB of its stack; that, plus a guard page, is its resident cost. The virtual reservation does matter — 100,000 × 8 MB exceeds a 47-bit address space’s comfortable use and 32-bit processes hit the wall at a few hundred threads — but "8 MB per thread" as RSS is wrong.
The honest per-thread bill on Linux is: the kernel task_struct and associated structures (~10 kB), a kernel stack (16 kB on x86-64, always committed), the touched pages of the user stack (tens of kB typical, up to the limit under deep recursion), thread-local storage, and the runtime’s per-thread state (Java’s default thread stack is 1 MB virtual with similar lazy commit; a JVM thread also carries a JIT/GC bookkeeping footprint). Call it 50–100 kB resident per idle thread: 100,000 threads is 5–10 GB. Feasible on a large box; expensive; and the memory is the *smaller* problem.
Set the stack size explicitly (pthread_attr_setstacksize, -Xss, threading.stack_size()) when you run many threads: it bounds virtual reservation, and a 256 kB stack makes an unexpected deep recursion fail fast with SIGSEGV on the guard page (see Stack Overflow) rather than eat 8 MB per thread first.
| Item | Virtual | Resident | Notes |
|---|---|---|---|
| User stack | 8 MB default | 8–64 kB typical | lazily committed; guard page below |
| Kernel stack | 16 kB | 16 kB | always committed; x86-64 |
| task_struct + kernel bookkeeping | — | ~10 kB | scheduler entity, signal state, fd table pointer |
| TLS + runtime state | varies | 1–50 kB | glibc TLS small; JVM/CPython per-thread state larger |
| Creation | — | — | ~10–30 µs for clone + stack mmap; more with runtime setup |
What a thread costs in time
The scheduler must choose among runnable threads; with a modern O(log n) run queue that choice stays cheap, but the switches do not. A context switch costs 1–5 µs directly (save registers, switch page tables or at least flush enough of the TLB to matter, restore) and more indirectly, because the new thread finds cold caches (see Context Switching). A server with 10,000 active connections each waking a thread for one small packet performs 10,000 switches per round; at 3 µs each that is 30 ms of pure switching per round, before any work.
Creation is the other latency. clone() plus a stack mmap is tens of microseconds; a runtime adds its own setup. Under a connection burst — a deploy, a cache flush, a retry storm — thousands of creations per second delay accept() itself, and the backlog from The Blocking Server returns. Thread caches and pre-spawned pools exist to remove this, which is the first step toward The Thread Pool Server.
Finally, threads share the address space, so any shared data — a connection count, a cache, a metrics map — needs synchronization, and lock contention scales with the number of threads that touch it. A single hot mutex turns a 64-thread server into a serialized one plus scheduling overhead (see Mutexes and Critical Sections). Independent per-thread state is fast; shared mutable state is where this model spends its remaining performance.
- Direct switch cost ~1–5 µs; indirect cost (cache and TLB refill) often larger.
- Wake-ups per packet, not connections per se, drive switching: idle connections cost memory, active ones cost switches.
- Every shared structure is a potential serialization point; measure with
perf lockor the runtime’s contention profiler.
Where it still works, and what runtimes changed
Thread-per-connection is a good design when connections number in the hundreds to low thousands, requests do real CPU or blocking work, and handlers are mostly independent. Historically that describes most Java servers (Tomcat, Jetty in its thread-per-request mode), PostgreSQL (process per connection, with a pooler in front), and countless internal services. The failure at 100,000 is real; the failure at 500 is imaginary.
Go changed the arithmetic rather than the model. A goroutine starts with a 2 kB stack that grows by copying, is scheduled by the runtime onto a small number of OS threads, and blocks on a socket by parking in the runtime while the OS thread continues — the runtime’s netpoller uses epoll/kqueue underneath. The programmer writes a blocking handler per connection; the machine runs an event loop. Java 21 virtual threads do the same for the JVM, Erlang/BEAM processes did it decades earlier, and Python’s asyncio tasks are the same idea with the suspension points made explicit (await). See How C++, JavaScript and Python Map onto the OS and Threads versus Async versus Processes.
The caveats are runtime-specific and worth stating precisely. A goroutine that makes a blocking system call (not a socket read — a file read, a cgo call, a syscall the netpoller does not cover) does occupy an OS thread; Go spawns more threads to compensate, and enough of them recreate the OS-thread cost. Java virtual threads "pin" to their carrier inside synchronized blocks in early releases. Cheap threads change the cost of blocking on sockets; they do not make every blocking operation free.
1ln := net.Listen("tcp", ":8080")2for {3 conn := ln.Accept() // goroutine parks on epoll, OS thread stays free4 go func(c net.Conn) { // ~2 kB stack, grows on demand5 buf := make([]byte, 4096)6 n := c.Read(buf) // "blocks" the goroutine, not the OS thread7 c.Write(response(buf[:n]))8 c.Close()9 }(conn)10}Key points
- Thread per connection makes the scheduler do the interleaving; handlers stay sequential and a slow client stalls only itself.
- The 8 MB default stack is virtual; resident cost per idle thread on Linux is roughly 50–100 kB (kernel stack, task state, touched stack pages, runtime state). Set the stack size explicitly when running many threads.
- The binding constraints at scale are context switches (~1–5 µs each, plus cold caches), creation latency under bursts, and lock contention on shared state — not RAM.
- It is the right design for hundreds to a few thousand connections with real work per request; the failure is at 100,000, not at 500.
- Go goroutines, Java virtual threads and BEAM processes keep the model and move the cost: the runtime multiplexes cheap user-level threads onto an event loop. Blocking syscalls the runtime cannot intercept still cost an OS thread.
Why does this exist?
Mechanisms are answers to constraints. Open each question before reading the answer.
▸Why did this model dominate for so long if it does not scale?
Because it scales far enough for most services, the code is the easiest kind to write and review, and CPU-bound work parallelises across cores without any extra machinery. The C10K problem was specifically about mostly-idle connections in the tens of thousands, which most services never have.
▸Why is the stack reservation still a problem if it is lazily committed?
Address space is finite, overcommit policies may refuse large reservations, and a 32-bit process runs out at a few hundred threads. Also, a thread that once recursed deeply keeps its touched pages until it exits — a stack does not shrink. Reserving less is cheap insurance.
▸Why does a goroutine cost less than a thread if both block?
Because "block" means different things. An OS thread blocking in recv() costs a kernel sleep/wake and a context switch. A goroutine blocking on a socket registers interest with the runtime’s poller and parks in user space; the OS thread picks up another goroutine without a kernel switch. The kernel sees one epoll_wait for thousands of goroutines.
How it fails
What the failure looks like from inside real software.
- Unbounded thread creation under a connection flood:
java.lang.OutOfMemoryError: unable to create native threadorEAGAINfrompthread_createoncethreads-max/ulimit -uis hit; the accept loop dies and the service goes dark. - Per-thread RSS estimated at 8 MB leads to a "we can only run 400 threads" limit that starves a machine with 64 GB free; the opposite mistake — assuming threads are free — leads to swap under a burst.
- A shared
HashMapbehind one lock serializes 200 handler threads; CPU shows 20% busy and 80% idle while p99 latency is seconds.perf lock/jstackshow everyone waiting on the same monitor. - Deep recursion in a handler with a 256 kB custom stack:
SIGSEGVon the guard page kills the whole process, not the thread; the fix is a larger stack for that pool or an iterative algorithm. - Go service doing blocking file I/O per connection: the runtime spawns OS threads to cover the blocked ones, thread count climbs into the thousands, and the "cheap goroutines" bill arrives as OS threads after all.