TCP Debugging: Reading the Handshake on the Wire

A TCP connection starts with a three-packet handshake, and every connection failure is a specific deviation from it. Capture the packets on the client and there are only three shapes to recognise. SYN, SYN-ACK, ACK: the handshake completed; if the application still reports failure, the problem is above TCP (TLS, HTTP, the app) or later in the connection. SYN, then RST: a host received the SYN and rejected it — nothing is listening on that port, or a firewall configured to reject answered on its behalf. The client gets ECONNREFUSED immediately. SYN, retransmitted, silence: nothing came back. The SYN was dropped by a firewall or security group in DROP mode, black-holed by a bad route, lost on a dead host, or — subtly — dropped by a listener whose accept queue is full. The client waits through exponential backoff (1, 2, 4, 8, 16, 32 s on Linux with tcp_syn_retries=6) and reports ETIMEDOUT, or the application gives up earlier with its own timer.

The The Three-Way Handshake lesson has the mechanics; here the point is diagnostic. RST is a positive signal — a host is there and you know which. Silence is absence of a signal and needs a second capture to locate the drop.

Step one, on the server: ss -tlnp (Linux) or lsof -iTCP -sTCP:LISTEN -nP (macOS) or netstat -ano | findstr LISTEN (Windows). Is the port listed? Which address? 127.0.0.1:443 or [::1]:443 explains a refusal from every other host — including from the Docker host to a container that bound loopback. No listing at all: the process is down, crashed on start, or listening on a different port than the config says.

Step two, on both ends at once: tcpdump -ni any "tcp port 443 and host CLIENT_IP" on the server and tcpdump -ni any "tcp port 443 and host SERVER_IP" on the client, then reproduce. Four outcomes. The server never sees the SYN: the drop is upstream — security group, network ACL, the client’s own egress firewall, a VPN that does not route this prefix, NAT. The server sees the SYN and sends a SYN-ACK that the client never sees: return-path problem — asymmetric routing, a stateful firewall that did not see the SYN and drops the "unsolicited" SYN-ACK, or the server’s default route pointing somewhere the client is not. The server sees the SYN and replies RST: nothing listening (step one lied? check the address family — IPv4 SYN to a socket bound only to ::1). The server sees the SYN and replies nothing: host firewall in DROP (iptables -L -n, nft list ruleset, ufw status), or listen-queue overflow (next section).

Step three: nc -vz from a third host. If a host in the same subnet connects and the client across the internet does not, the problem is between them — and you have bounded the search to the path, not the endpoints.

A socket bound to 127.0.0.1 accepts connections only from the same network namespace via loopback. That is a feature for databases and admin ports, and the number-one cause of "works with curl on the box, refused from anywhere else". It is worse in containers: localhost inside the container is the container’s own loopback, so a process that binds 127.0.0.1:8080 inside is unreachable even through docker run -p 8080:8080 — the port mapping forwards to the container’s eth0 address, where nothing listens. The fix is to bind 0.0.0.0 (or :: for dual-stack) inside the container and restrict access with the firewall or the orchestrator instead.

Address family is the second trap. A listener on [::]:443 on Linux accepts IPv4 too (via IPv4-mapped addresses) unless IPV6_V6ONLY is set; a listener on 0.0.0.0:443 accepts IPv4 only, and an IPv6 client to the same host gets RST. Check what the client actually resolved and connected to (curl -v prints it) against what ss shows.

A listening socket has two queues: the SYN queue (half-open, waiting for the ACK) and the accept queue (handshake complete, waiting for the process to call accept()). The accept queue is bounded by the listen() backlog (capped by net.core.somaxconn, 4096 on modern kernels). When it is full, Linux by default (tcp_abort_on_overflow=0) drops the final ACK or the SYN rather than resetting — so the client sees the handshake stall and retransmit, and the symptom is intermittent connect timeouts or one-second and three-second connect latencies on a server that is otherwise up. ss -tln shows Recv-Q at the backlog limit for the listener; nstat -az TcpExtListenOverflows TcpExtListenDrops counts the drops (netstat -s | grep -i listen on older tools). The fix is on the server: accept faster (more workers, an event loop that is not blocked — see The Event-Driven Server) and raise the backlog.

On the client side the equivalent exhaustion is ephemeral ports. Each outbound connection to the same destination IP:port needs a unique local port from net.ipv4.ip_local_port_range (default 32768–60999, ~28k ports), and every closed connection leaves a TIME_WAIT entry for 60 s on Linux. A client — or a proxy, or a NAT — opening more than ~470 connections per second to one backend runs out and connect() fails with EADDRNOTAVAIL, with nothing on the wire at all. ss -tan state time-wait | wc -l shows the pile-up; the fix is connection reuse (Connection Pooling, Keep-Alive and Connection Reuse), tcp_tw_reuse, or a wider port range. The Connection Lifecycle: Close, Reset, TIME_WAIT, CLOSE_WAIT explains why TIME_WAIT exists and why simply disabling it is wrong.

A RST during an established connection is a different animal from a RST answering a SYN. Common sources: the server process crashed or was killed, and the kernel reset every open socket (the client’s read() returns ECONNRESET); a load balancer or NAT expired an idle flow and, depending on configuration, either silently dropped subsequent packets (client sees a stall then timeout) or reset them; a firewall’s connection-tracking table filled up; or one side received data for a connection it no longer knew about (after a reboot) and answered RST. Wireshark’s tcp.flags.reset==1 finds them; the sender of the RST is the device to investigate, and its position (server address vs an intermediate) tells you whether it was the peer or a middlebox.

A connection that opens but then stalls — small requests work, large ones hang — is often not TCP at all but path MTU: the large segments are silently dropped and retransmitted forever. tcpdump shows retransmissions of the same 1448-byte segment with no ACK progress. That is the MTU black-hole challenge; Packet Loss: Duplicate ACKs, Fast Retransmit and the RTO covers what loss recovery looks like on the wire when the loss is real.