C10K: Ten Thousand Connections, Then a Million

In 1999 a server that could hold 10,000 simultaneous connections was an engineering achievement; the phrase "C10K problem" named it. The connections were mostly idle — a chat client, a long-poll, a slow download — and the servers of the time gave each one a thread or process and polled with select. Both costs were per connection and both were large: a thread’s scheduling and memory, and select’s O(n) scan on every wake-up. The machine was busy doing nothing.

What moved the number was not faster hardware but two mechanisms: readiness APIs that cost O(ready) instead of O(n) (epoll in Linux 2.5.44, kqueue in FreeBSD 4.1) and architectures that do not park a thread per connection (The Event-Driven Server, and later runtimes with cheap user-level threads). With those, an idle connection costs a few kilobytes of kernel memory and nothing else, and the ceiling moves to the next resource. This lesson walks the ceilings in the order you meet them.

Thread per connection (Thread per Connection) hits the wall first: at ~50–100 kB resident per idle thread and microseconds of context switch per wake-up, 10,000 is workable and 100,000 is not. Either the architecture stops parking threads on sockets or the runtime makes the threads cheap (goroutines at ~2 kB, Java virtual threads); both are legitimate, and the second is often the right engineering answer because it keeps the code sequential.

Every connection is a file descriptor, and descriptors are limited twice (see File Descriptors). Per process: RLIMIT_NOFILE, shown by ulimit -n, historically 1024 soft — a server that forgets to raise it dies at 1,021 connections with EMFILE: Too many open files, a failure so common it has its own OS challenge. System-wide: fs.file-max (ENFILE), typically hundreds of thousands to millions by default on modern kernels. Raise the soft limit at startup (setrlimit) or in the service unit (LimitNOFILE=), and remember that the descriptor table itself is memory: a million entries is a few megabytes, fine, but each open socket behind it is not free.

Hitting EMFILE in accept() is uniquely nasty: the connection stays in the accept queue, epoll keeps reporting the listen socket readable, and a naive loop spins at 100% CPU failing to accept. The standard trick is to hold one spare descriptor, close it on EMFILE, accept, close the client, reopen the spare — or simply to stop watching the listen socket until a descriptor is freed.

A TCP socket in the kernel is a struct sock and friends — roughly 2–4 kB — plus its buffers. Idle, the buffers are empty and the socket is cheap: on the order of a few kilobytes. Active, each direction can hold up to its buffer limit: with Linux autotuning the receive buffer grows toward tcp_rmem max (6 MB by default) and the send buffer toward tcp_wmem max (4 MB) *if the connection needs it*; a typical active connection sits at tens of kilobytes per direction. net.ipv4.tcp_mem caps total TCP memory system-wide in pages, and when it is exceeded the kernel starts refusing to grow buffers and logging "TCP: out of memory". See The Buffer Chain.

Then the application’s state: a parser, a per-connection object, TLS state (OpenSSL keeps ~20–50 kB per TLS connection, which is why TLS terminators budget it separately), request buffers. A well-built event-driven server keeps total cost per idle connection under ~10 kB and per active one under ~100 kB; at a million connections that is 10 GB idle to 100 GB active. Memory, not CPU, is what a "C10M" box is sized for.

The calculator attached to this lesson lets you vary these per-connection numbers and see which ceiling arrives first. The numbers it uses are simulated defaults; the real ones come from ss -m (per-socket memory), /proc/net/sockstat (TCP memory in pages), and your process’s RSS divided by its connection count.

A connection is identified by the 4-tuple (source IP, source port, destination IP, destination port). From one client IP to one server IP:port, the only variable is the source port, chosen from the ephemeral range: on Linux net.ipv4.ip_local_port_range is 32768 60999 by default, giving 28,232 connections at once to a given destination. Load generators, proxies that open a new upstream connection per request, and services that talk to one database all hit this. Worse, a closed connection’s tuple is held in TIME_WAIT for 60 s on Linux (fixed), so a proxy that opens and closes 1,000 upstream connections per second needs 60,000 tuples — more than the range. The symptoms are EADDRNOTAVAIL: Cannot assign requested address on connect() and a ss -s with tens of thousands of timewait. See The Connection Lifecycle: Close, Reset, TIME_WAIT, CLOSE_WAIT and Ports: Addressing a Process, Not a Machine; the challenge is Connection errors to a healthy upstream at 400 requests per second.

The fixes are at the right layer: reuse connections (Connection Pooling, Keep-Alive and Connection Reuse), widen the range, add source IPs, or let the kernel reuse TIME_WAIT tuples for outgoing connections (net.ipv4.tcp_tw_reuse, safe with timestamps). Note that the server side has no such limit: a server holds a million connections on one port because the *client* side of each tuple differs.

Between client and server, every stateful device keeps a table entry per connection: a NAT gateway (NAT: Many Private Hosts Behind One Public Address), a stateful firewall (Firewalls), a load balancer (Load Balancers: L4 vs L7), a Linux host running conntrack (nf_conntrack_max; when it fills, nf_conntrack: table full, dropping packet and new connections time out while existing ones work). Cloud NAT gateways publish per-IP connection limits for exactly this reason. C10K on the server is only the beginning; the path has to hold the state too. See the challenge Outbound calls fail from the whole cluster whenever traffic peaks.

The event loop is the answer when connections are many and idle. A thread pool is the answer when work per request dominates. A runtime with green threads over a netpoller — Go, Erlang, Java virtual threads — gets close to event-loop cost with thread-style code, and for most teams that is the best trade available. WhatsApp famously held two million connections per box on Erlang; nginx does it with epoll in C; a Go service does it with goroutines. Each is thread-per-connection *from the programmer’s view* or event-driven *from the kernel’s view*, and the choice between them is about the code you want to maintain.

What every C10K-scale system shares is not an architecture but a discipline: no per-connection OS thread parked on a socket, no O(n) scan per event, raised and monitored descriptor limits, bounded buffers per connection, connection reuse on the client side, and knowledge of every stateful device on the path. Get those right and the architecture is a matter of taste; get one wrong and no architecture saves you.