HTTP Debugging: 502, 503 and 504 Are Different Failures

A 5xx from a gateway is a proxy, load balancer or CDN reporting on its upstream — the next hop toward your application. The three codes map, in common practice, to three different upstream events. 502 Bad Gateway: the gateway got a response it could not use — the upstream closed the connection before or during the response (a crashed or restarting process, a keep-alive race, an out-of-memory kill), sent malformed headers, or reset the connection. 503 Service Unavailable: the gateway did not have a usable upstream to send to — every backend failed its health check, the pool was empty during a deploy, an overload circuit tripped, or the application itself returned 503 for maintenance or shed load. 504 Gateway Timeout: the gateway sent the request and waited the configured upstream timeout without getting a complete response — a slow database query, a lock, a thread pool that is full, an event loop blocked.

That mapping is a convention, not a law, and infrastructure behaviour varies by proxy, load balancer and CDN. nginx returns 502 when it has no live upstreams; an AWS ALB returns 503 for a target group with no healthy targets; some CDNs return 52x codes of their own for origin failures; Envoy annotates with response flags. Read your gateway’s documentation once and pin the mapping to a wiki page — then the codes become evidence rather than folklore.

A request may pass CDN → load balancer → ingress → sidecar → application, and any of them can generate a 5xx. Before any other question, identify the author. The Server header often names it (nginx, awselb/2.0, cloudflare, envoy). Via lists proxies that added themselves. The body is a fingerprint: nginx’s default 502 page, the ALB’s bare-text 502 Bad Gateway, a CDN’s branded error page, or your application’s JSON error envelope. CDN-specific headers (cf-ray, x-cache, x-served-by) tell you the CDN was reached; a request id header (x-request-id, x-amzn-requestid, x-amz-cf-id) is the thread you pull through every log downstream.

Timing is the second fingerprint. A 504 that arrives at exactly 60.0 s was generated by something with a 60 s timeout — nginx’s proxy_read_timeout default, the ALB’s default idle timeout — and the number identifies the hop. A 502 that arrives in 3 ms never reached a backend; one that arrives after 30 s of streaming did.

Take the request id from one failed response and search the gateway’s access log for it: the line includes the upstream address chosen, upstream_status, upstream_response_time and upstream_connect_time (nginx variable names; ALB and Envoy have equivalents). That single line tells you whether an upstream was chosen, whether it connected, how long it took, and what it returned. Then the gateway’s error log for the same second: upstream prematurely closed connection while reading response header from upstream (502), connect() failed (111: Connection refused) while connecting to upstream (502 — nothing listening, see TCP Debugging: Reading the Handshake on the Wire), no live upstreams while connecting to upstream (502 in nginx; a 503 elsewhere), upstream timed out (110: Connection timed out) while reading response header from upstream (504).

Cross-check the health-check state at that moment — the load balancer’s target health history, the ingress controller’s endpoint list, kubectl get endpoints — because a 503 wall coincides almost always with a health check flipping. Then the backend: was it restarting (deploy, crash loop, OOM)? Was it slow (p99 latency, DB slow-query log, lock waits)? Load Balancers: L4 vs L7 explains how health checks decide pool membership and why a check that passes on / and fails on /health matters.

curl -sv -w timings from a machine that can reach both the gateway and the backend directly reproduce the split: if the backend answers directly in 40 ms but through the gateway in 60 s then 504, the gateway is not reaching the backend it thinks it is (wrong upstream address, security group between LB and instance, a stale endpoint).

Gateways keep idle connections to backends open and reuse them. The backend also has an idle timeout, after which it closes the connection. If the backend’s idle timeout is shorter than the gateway’s, there is a window in which the gateway picks an idle connection at the same instant the backend is closing it, sends the request into a closing socket, and gets a FIN or RST back — a 502 with upstream prematurely closed connection. Node’s default server.keepAliveTimeout is 5 s; an ALB’s default idle timeout is 60 s; the race fires on a small fraction of requests and looks like a flaky backend that is perfectly healthy.

The rule is: the backend’s keep-alive idle timeout must be longer than the gateway’s (and the backend’s headersTimeout longer still), so the gateway is always the one to close. Set Node to 65 s behind a 60 s ALB, or nginx’s keepalive_timeout above whatever fronts it; and configure the gateway to retry idempotent requests on a connection that failed before any bytes were received. Keep-Alive and Connection Reuse and Connection Pooling cover the mechanism; the 502 challenge in this module is the crash-loop variant.

Every hop has a timeout, and they must decrease as you go inward: client timeout > CDN timeout > load balancer timeout > application request timeout > database statement timeout. When the order is broken — a 30 s client with a 60 s gateway with a 120 s query — the client gives up and retries while the gateway is still waiting and the database is still working. The retry starts a second query; the client gives up again; the database now has three copies of the same slow query competing for the same lock. That is how a single slow endpoint becomes an outage: the retries amplify load exactly when capacity is gone.

Ordered timeouts fail inward-first: the statement timeout kills the query at 20 s, the application returns a clean 500 or 503 at 21 s, the gateway forwards it at 21 s, the client sees an honest error at 21 s and does not retry a request that is still running. The 504 disappears not because the backend got faster but because the failure is reported by the layer that owns it. Pair this with a retry budget and Retry-After on 503 so clients back off rather than pile on. The 504 challenge in this module walks a mis-ordered ladder.