Debuggingdebuggingtimeoutconnection refusedbisectionlayers

Why Can’t I Connect?

Every connection failure lives in exactly one layer, and the fastest way to find it is to bisect the ladder — DNS, route, host, port, handshake, TLS, HTTP, application — reading the failure signature at each step instead of guessing.

ConceptualLinux

▶ Interactive Interview question

Progress

The problem

Your service logs ECONNREFUSED, the frontend shows a spinner, and the on-call engineer has nine layers between the browser and the process that could be responsible. Which one, and how do you prove it in under five minutes?

Progressive depth

The same mechanism at different altitudes — start where you are.

Nine rungs, first failure wins

Name → address → route → host → port → handshake → TLS → HTTP → application. Check each in order with its own tool; the first one that fails is the fault, and everything below it is a consequence.

The ladder: one question per layer

A connection is a chain of dependencies: a name must resolve to an address, the address must have a route, the host must be up, the port must have a listener, the transport handshake must complete, the TLS handshake must verify, the HTTP exchange must succeed, and the application must do the right thing with it. Every one of those is a separate mechanism with a separate failure signature, and each one only becomes relevant once the one above it works.

That ordering is the entire method. Do not read logs top to bottom; walk the ladder top to bottom and stop at the first rung that fails. A DNS failure makes every later layer look broken (nothing to connect to), so testing TLS first wastes the five minutes you have. The interactive lets you inject a fault and practise finding the rung.

Each rung has a canonical tool (The Tools, and Which Layer Each One Answers maps the whole set), a canonical success, and a canonical failure signature. Learn the signatures, not the commands — commands vary by platform, signatures do not.

The connection ladder — stop at the first rung that fails

Application error`ECONNREFUSED`, `ETIMEDOUT`, `ENOTFOUND`, 502, "certificate invalid" — the error text already names a layer
↓
DNS works?`dig api.example.com` → NOERROR with an A/AAAA answer; NXDOMAIN / SERVFAIL / no answer means stop here
↓
Route exists?`ip route get 203.0.113.10` shows a next hop; "no route to host" / "network is unreachable" means the kernel has nowhere to send it
↓
Host reachable?`ping` / `traceroute` — but ICMP may be filtered; treat a failed ping as inconclusive, not fatal
↓
Port open?`nc -vz host 443` — refused (RST) vs timeout (silence) are two different diagnoses
↓
Transport handshake?SYN → SYN-ACK → ACK; `tcpdump` shows which of the three is missing
↓
TLS?`openssl s_client -servername host -connect host:443`; a TLS alert names its own cause: expired, unknown CA, handshake failure
↓
HTTP?`curl -v` — a status code proves every layer below it works; 5xx means a server-side hop, 4xx means the request
↓
Application?A 200 with the wrong body, a 500 with a stack trace — now it is the code, and the network is exonerated

Timeout and refused are different diagnoses

The single most common debugging mistake is treating "cannot connect" as one symptom. It is two. Connection refused means a packet reached a host and that host answered with a TCP RST: the machine is there, the route works, and nothing is listening on that port (or a firewall on the host is configured to reject rather than drop). Connection timed out means the SYN was sent and nothing came back at all — for Linux with the default tcp_syn_retries=6, roughly 127 seconds of retransmissions with no reply.

Refused is good news: you are one ss -tlnp away from the answer, because the fault is on the destination host and it is almost always "the process is not running", "it is listening on 127.0.0.1 instead of 0.0.0.0", or "wrong port". Timeout is a search: the SYN was dropped somewhere between you and the listener — a cloud security group, an on-host firewall in DROP mode, a routing black hole, a NAT that lost state, or a host that is simply off. The Firewalls lesson explains why security teams prefer DROP (it reveals nothing) and why that makes your life harder.

There is a third signature: the connection succeeds and then hangs. That is not a connection problem at all — the handshake worked. It is a transport or application problem: an MTU black-hole path issue where large packets vanish, a server that accepted but never reads, or a backend waiting on a lock. TCP Debugging: Reading the Handshake on the Wire shows how to tell them apart on the wire.

Refused = RST came back = host up, port closed. Look on the server.
Timeout = nothing came back = dropped on the path or host down. Look at firewalls, security groups, routes.
Connects then hangs = handshake fine, data is not flowing. Look at MTU, the application, and the read side.
`ENOTFOUND` / `EAI_AGAIN` = never got as far as a socket. It is DNS.

Reading the failure signature at each rung

Every rung produces a specific error when it fails, and the error names the rung. A resolver answers NXDOMAIN (the name does not exist) or SERVFAIL (the resolver could not get an answer) — both are DNS, but they have different causes. The kernel answers ENETUNREACH (no route) or EHOSTUNREACH (the next hop could not be reached, often an ARP/ND failure on the local segment). TCP answers ECONNREFUSED or ETIMEDOUT. TLS answers with an alert: certificate_expired, unknown_ca, handshake_failure, protocol_version. HTTP answers with a status code, and a 5xx points at a hop behind the one you are talking to.

The matrix below is the whole lesson in one table. Keep it near you during incidents; the "next" column is what stops you from staring at the same output.

Rung → tool → signature → next move

Rung	Ask with	Healthy	Failure signature	Next
DNS	`dig` / `nslookup` / `Resolve-DnsName`	NOERROR + answer	NXDOMAIN, SERVFAIL, empty answer, wrong IP, stale TTL	Which resolver answered? Compare `@8.8.8.8` vs default; check `/etc/hosts`
Route	`ip route get`, `route print`	a next hop and interface	"network is unreachable", "no route to host"	Routing table, VPN up/down, default gateway
Host	`ping`, `traceroute`	replies with stable RTT	silence (inconclusive), `* * *` at the last hops	Try a TCP probe before concluding the host is down
Port	`nc -vz`, `Test-NetConnection -Port`	"succeeded" / TcpTestSucceeded True	refused (RST) vs timeout (drop)	Refused → `ss -tlnp` on server. Timeout → firewall / security group
Handshake	`tcpdump -n port 443`	SYN, SYN-ACK, ACK	SYN repeated, SYN-ACK missing, RST	Capture on both ends to find where the packet died
TLS	`openssl s_client`, `curl -v`	"Verify return code: 0 (ok)"	expired, unable to get local issuer, hostname mismatch, alert 40/70	Check clock, chain, SAN, SNI, protocol versions
HTTP	`curl -v`, browser devtools	2xx/3xx	502 / 503 / 504, 4xx, hang after request sent	Which hop generated it? Server/Via headers, request id
Application	logs, traces, request id	correct body	500 with a stack trace, 200 with wrong data	It is the code now; the network is exonerated

Where it usually is

Across a lot of incidents the culprits cluster. The process is listening on the loopback address only, which works from the same host and refuses from everywhere else — endemic in containers where localhost is the container, not the machine. The cloud security group or host firewall allows port 80 but the service moved to 8080. DNS points at the old IP because someone changed the record and the TTL was a day. The certificate expired at 00:00 UTC and the monitoring checked HTTP, not HTTPS. The load balancer marked every backend unhealthy because the health check path returns 401 after an auth change.

Note what is not on that list: broken routers, corrupted packets, mysterious network gremlins. Those exist and traceroute: Discovering the Path Hop by Hop finds them, but they are rare compared with configuration on the two ends. Start with the ends; bisect the middle only when both ends look right.

Bisect from both ends when you can. If a tcpdump on the server never sees the SYN, the fault is upstream of the server and you can stop reading server logs. If it sees the SYN and sends a SYN-ACK that never reaches the client, the fault is on the return path — asymmetric routing, a NAT that lost its mapping, a return-path firewall. Two captures turn a search over nine layers into a search over one segment.

Bind address: 127.0.0.1:8080 serves the host only; 0.0.0.0:8080 (or [::]:8080) serves the network. ss -tlnp shows which.
Security groups are stateful per direction of initiation; a rule for inbound 443 does not help if the service listens on 8443.
A fresh deploy with a new IP plus a long DNS TTL is a "works for me, fails for the customer" classic — see DNS Debugging: Who Answered, and With What?.
Health checks are HTTP requests too: they fail through the same ladder, and a failed health check produces a 503 with a perfectly healthy backend.

Key points

Walk the ladder top-down and stop at the first failing rung; every rung below a failure only looks broken.
Timeout (silence, dropped SYN) and refused (RST, nothing listening) are different diagnoses with different next steps.
Each layer emits its own signature: NXDOMAIN/SERVFAIL, ENETUNREACH/EHOSTUNREACH, ECONNREFUSED/ETIMEDOUT, TLS alerts, 5xx.
A successful HTTP status proves every layer below it; a failed ping proves almost nothing.
Most faults are on the two ends — bind address, port, firewall rule, DNS record, certificate — not in the middle.
Capture on both ends to bisect the path in one step.

Why does this exist?

Mechanisms are answers to constraints. Open each question before reading the answer.

▸Why bisect by layer instead of reading logs?

Logs describe the symptom at the top of the stack; the fault is at one specific rung below it, and each rung has a cheap, decisive test. Nine tests in order beat an unbounded read of application logs that all say "connection failed".

▸Why does the OS distinguish refused from timed out at all?

Because TCP does: a host with no listener sends a RST, so the kernel can fail the connect() immediately with ECONNREFUSED. Silence carries no information, so the kernel retransmits the SYN with exponential backoff and only gives up after the retry budget — ETIMEDOUT.

▸Why do security groups drop instead of reject?

A RST tells an attacker a host exists and which ports are closed. Silence reveals nothing. The cost is that legitimate debugging sees a timeout instead of an immediate refusal — you pay two minutes per test for that opacity.