HTTPGENERALRUNTIME-SPECIFICCLOUD-SPECIFIC
Accepting Connections
A connection is not a request. Listen, backlog, accept and file descriptors decide what happens to traffic before your code exists.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
The question
What happens to a request between the client's TCP handshake and the first line of my server code?
The requirement
The service must stay reachable during traffic spikes rather than refusing connections outright, and we need to know what "reachable" costs.
The obvious build
The server listens, connections come in, requests get handled. If we are overloaded, requests get slower — that is what overload looks like.
Why it breaks
Under burst, clients get connection resets or timeouts rather than slow responses, and your server-side latency graph looks perfect throughout because the refused connections never became requests.
How it breaks in production
- Under burst, clients get connection resets or timeouts rather than slow responses, and your server-side latency graph looks perfect throughout because the refused connections never became requests.
- A load balancer marks instances unhealthy during a spike because its health check connection sat in the accept queue behind three thousand others (Health Checks: Startup, Readiness, Liveness).
- The process hits its file-descriptor limit and
acceptstarts failing; simultaneously the database driver cannot open connections and the logger cannot open a file. Three unrelated-looking errors, one cause. - One long CPU-bound operation stops the accept loop entirely on a single-threaded runtime, so new connections queue in the kernel while the process looks idle in your application metrics (Blocking the Event Loop).
Requirement→API Contract→Application Logic→Data Access→External Deps→Concurrency→Failure→Security→Observability→Deployment→Scale
What is actually happening
bindattaches a socket to an address and port;listenmarks it passive and sets a backlog. From that moment the kernel, not your process, is handling arriving connections.- On Linux there are effectively two queues: incomplete handshakes (SYN received, waiting for the final ACK) and completed connections waiting for your process to call
accept. Overflow behaviour differs between them, and the completed-connection queue is the one your application starves. acceptremoves one completed connection and returns a new file descriptor. The listening socket stays; the returned socket is one client.- A connection is not a request. With keep-alive, one accepted connection carries many sequential requests (Keep-Alive and Connection Reuse); with HTTP/2 it carries many concurrent streams.
- Every accepted connection costs a file descriptor, kernel send and receive buffers, and whatever per-connection state your runtime keeps. That product is your real connection ceiling, and it is usually memory or descriptors rather than CPU.
- What happens after accept is the runtime's choice: a thread per connection, a thread pool, or registration with an event loop that watches thousands of descriptors at once (Backend Runtime Models).
Two queues before your code
SIMPLIFIEDLinux-shaped. The two-queue split and its overflow behaviour are tunable and differ across kernels and BSD-derived systems; the teaching point — a queue exists before your process and it can overflow — is universal.
The mental model that matters: by the time a connection is ready for you, the kernel has already done the handshake and put the connection in a queue. Your process's only job is to drain that queue quickly. When it does not, the queue fills and the kernel starts refusing work on your behalf — with no application-level event to record.
This is why a single-threaded runtime running one expensive synchronous operation produces a latency spike with a very odd shape: the affected requests were not slow to process, they were slow to be *noticed*.
UserLLMAgentToolDataDecisionHumanGuardrail
Accept is a loop you are competing with
Whatever the runtime, something is calling accept repeatedly. Whether that something can keep up while requests are being served is the entire difference between the concurrency models (Backend Runtime Models).
Writing the loop by hand makes the competition obvious: the two calls below are on the same thread, so time spent in one is time not spent in the other.
The accept loop, made explicit
1import socket2 3srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)4# Without SO_REUSEADDR, restarting the process fails while old5# connections linger in TIME_WAIT.6srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)7srv.bind(('0.0.0.0', 8080))8srv.listen(511) # <- backlog: how many COMPLETED connections9 # the kernel will hold for us before refusing10 11while True:12 conn, addr = srv.accept() # one connection off the queue13 conn.settimeout(10) # a client that never finishes its14 # headers must not hold this forever15 try:16 handle(conn) # <- every second spent here is a17 # second not spent in accept()18 finally:19 conn.close() # the fd is the resource; leak it20 # and accept() eventually failsThis server handles exactly one client at a time. The interesting question is not how to make handle faster — it is what you do so that accept keeps running while handle is busy. Every concurrency model is one answer to that question.
When the queue fills
These are the four symptoms that mean "the problem is before your code". None of them produce a stack trace, and three of them look like network faults from the client's side.
Symptoms that live above the application
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Traffic burst, accept loop busy or slow | Clients time out or get connection reset; server latency graphs look normal | Accept queue overflowed; those connections never became requests | Cap in-flight work and shed with 503; move CPU work off the accept thread; scale out on connections, not just RPS |
| Descriptor limit reached | EMFILE, plus simultaneous database and file-open failures | One descriptor budget shared by sockets, pool connections and files | Raise the limit, cap concurrent connections, and find the leak — unclosed sockets or an unbounded pool |
| Many clients opening one connection per request | High accept rate, requests-per-connection near 1, CPU spent in TLS | Keep-alive not in use or being closed by an intermediary | Check the client library and every hop's keep-alive settings (Keep-Alive and Connection Reuse) |
| Clients that connect and dribble headers | Connection count climbs, bandwidth and CPU near zero, capacity gone | No header-read timeout; connection slots held by incomplete requests | Set a header timeout and a per-IP connection cap; terminate at a proxy that already does both |
How to build it
Most important first.
- Raise the process file-descriptor limit deliberately and know the number. It is the ceiling on concurrent connections plus database connections plus open files, all sharing one budget.
- Keep the accept path free of application work. On a single-threaded runtime, anything that occupies the loop delays accept for every pending client.
- Set an explicit ceiling on concurrent connections or in-flight requests and shed load above it with a fast 503, rather than letting the kernel queue decide by dropping (Resource Limits).
- Give a header-read timeout: a connection that is accepted but never sends a complete request holds resources indefinitely otherwise.
- Let a proxy or load balancer terminate the internet-facing connections when you can. It absorbs slow clients and connection churn so your process sees a smaller, better-behaved connection set (Load Balancing, From the Backend's Side).
- Measure connections and requests separately. Autoscaling on requests per second while the constraint is concurrent connections scales the wrong dimension (Autoscaling a Backend).
What can go wrong
Failure modes
- Accept-queue overflow: connections silently dropped or reset. Depending on kernel settings the client sees a hang and then a timeout, which looks like a network problem rather than a capacity problem.
EMFILE/ "too many open files": accept fails, and on some runtimes the accept loop spins on the error instead of backing off, burning CPU while serving nothing.- Idle connections accumulating from clients that opened and left; each one costs memory whether or not it ever sends a byte.
- Slow-loris style exhaustion: many connections that send one header byte at a time, occupying connection slots with almost no bandwidth (Request Bodies and Streaming).
- The mitigation failing: a connection cap that returns 503 immediately can knock out your own health check too, so exclude it or give it a reserved path.
What can race
- In a pre-fork model, multiple workers share one listening socket, so an arriving connection can wake several of them and only one wins the accept — the thundering herd.
SO_REUSEPORTgives each worker its own queue instead, trading fairness characteristics for less contention (Worker Processes). - A connection can be accepted and then closed by the client before your handler runs, so the request you are about to process may have no reader left.
Security
- Connection-level exhaustion needs no valid request and no authentication. It is available to anyone who can reach the port, so limits belong at the edge as well as in the process (Rate Limiting).
- A per-IP connection cap is a blunt but effective control; note that behind a proxy or NAT, "IP" is the proxy or a whole office, so caps set per source address can punish real users.
- Never leave a debug or admin listener bound to a routable address. A second listening socket is a second attack surface with usually a fraction of the review.
Misreads
- "Connections per second and requests per second are roughly the same." With keep-alive they differ by an order of magnitude, and the ratio is the single best indicator of whether reuse is working.
- "The backlog is a performance setting." It is a failure-mode setting: it decides whether excess traffic waits or is refused, not how fast anything runs.
- "We are not overloaded, latency looks fine." Latency measured server-side excludes every client that never got accepted. This is coordinated omission and it makes overload invisible.
- "More instances always fix connection refusals." They do if the constraint is per-process. If the client is opening a new connection per request through a shared proxy, the constraint may be upstream of your instances entirely.
Operating it
How you see it in production
- On Linux,
ss -ltnshows Recv-Q and Send-Q for listening sockets: for a listening socket those are the current accept-queue depth and the configured backlog. A non-zero Recv-Q means connections are waiting for your process to accept them. - Track current concurrent connections, accepts per second and requests per connection. The third number tells you whether keep-alive is actually working.
- Track open file descriptors against the limit as a ratio, and alert on it. It is a hard ceiling with a confusing set of symptoms.
- Compare connection count at the load balancer with connection count at the instance. A gap means connections are being terminated or refused between them.
What changes at 10x and 100x
- At 10x, per-connection memory becomes a real budget line and idle connection cleanup starts to matter. Reuse ratio — requests per connection — becomes a metric worth graphing.
- At 100x with many idle connections, the classic C10K framing applies: the cost is no longer per request but per *waiting* connection, which is what pushed servers from thread-per-connection to event-driven I/O multiplexing.
- Horizontal scaling adds accept capacity linearly and does nothing for a single slow accept loop. If one process cannot accept fast enough because it is busy, more instances help; a bigger instance may not (Horizontal vs Vertical Scaling).
What this costs
- A large backlog absorbs bursts and hides overload: clients wait in a queue you cannot see instead of failing fast. A small backlog gives honest, early failure and drops traffic you might have served.
- Shedding load with 503 protects the service and produces errors that count against your availability numbers. Queueing produces latency instead. Both are real costs; pick which one you would rather explain.
- Terminating connections at a proxy adds a hop and an operational dependency in exchange for absorbing slow clients and connection churn.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALBind/listen/accept and the descriptor cost are the POSIX socket model, common to every server on Unix-like systems; Windows IOCP differs in the readiness-versus-completion model but keeps the same accept-and-descriptor economics.
- RUNTIME-SPECIFICNode accepts on the single loop thread, so CPU work in a handler delays accept for every pending connection; a Go server accepts in one goroutine and hands each connection to another scheduled across all cores, so the accept path keeps running while handlers are busy; a pre-fork Gunicorn setup has N processes racing on one listening socket instead.
- CLOUD-SPECIFICBehind a managed load balancer you rarely see raw internet connection patterns: the balancer keeps its own pool to your instances, so your accept rate reflects the balancer's reuse policy, not client behaviour — and its idle timeout, not yours, usually decides when a connection ends.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
Cloudcloud-load-balancers