How Networks Fail in Production
A catalogue: DNS, certificates, blocked ports, security group mistakes, connection exhaustion, NAT port exhaustion, packet loss and latency spikes — each with a symptom that identifies it.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Something in the network is wrong. What are the candidates, and which symptom belongs to which?
Network failures present as generic application symptoms — timeouts, connection errors, intermittent slowness — so the investigation usually starts in the application, which is the one place the fault is not.
The service cannot reach its dependency, so something is wrong with the service or the dependency. Check both applications and their logs.
The application on both ends can be entirely healthy while nothing gets between them. Neither log will contain the answer, because neither process saw anything except a timeout.
- The application on both ends can be entirely healthy while nothing gets between them. Neither log will contain the answer, because neither process saw anything except a timeout.
- The symptoms are shared. A timeout is produced by DNS failure, a blocked port, packet loss, connection exhaustion and simple overload, and the application cannot distinguish them (Why Can’t I Connect? in the networking view).
- Some of these fail partially: one destination, one zone, one protocol, one percentage of requests. Partial failures are read as flakiness and attributed to the application.
- Several of them are triggered by something that has nothing to do with your service: a security group edited for another workload, a NAT gateway shared by the whole subnet, a certificate on a dependency.
- The classic asymmetry: connection refused is fast and means something answered and said no; a timeout is slow and usually means nothing answered at all. Teams that do not distinguish these lose a lot of time.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Failures live at identifiable points on the path: name resolution, the route out, a filter in between, the handshake, the transport, and the destination. Each point produces a characteristic symptom, and the symptom is the diagnosis.
- DNS failure stops the request before any connection is attempted — the error names the host rather than an address (DNS in Production).
- Certificate failure happens after a successful TCP connection: the handshake completes at the transport level and is rejected at the TLS level (Certificates as an Operational Object).
- Blocked port or security group produces either a timeout (packets dropped silently, the common default) or a refusal (actively rejected). Which one tells you whether the filter drops or rejects.
- Connection exhaustion is a limit reached on one side: file descriptors, a connection pool, a load balancer's ceiling, or a database's maximum connections. It fails abruptly at a threshold, under load (Connection Pool Exhaustion in the backend view).
- NAT port exhaustion is the subtle one: outbound connections behind a shared translation address are limited by the tuple of source address, source port, destination address and destination port. Many short-lived connections to one destination exhaust the ports available for that destination, so connections to that one host start failing while everything else works perfectly (NAT: Many Private Hosts Behind One Public Address in the networking view).
- Packet loss does not fail — it retransmits, so the symptom is latency and variance rather than errors, and only above a threshold does it become visible failure (Packet Loss: Duplicate ACKs, Fast Retransmit and the RTO in the networking view).
- Latency spikes move the tail without moving the median, which makes them invisible to any dashboard built on averages (Percentiles: Which One, and How Many Users Is That? in the observability view).
Where on the path each failure lives
Every failure in this lesson sits at one identifiable point between a caller and a callee. Locating the point is the diagnosis; the fix follows from it almost mechanically.
The catalogue
Read the symptom column first, because the symptom is what you have. The most valuable rows are the ones whose symptom is not obviously a network symptom at all.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| DNS resolution fails | Error names the hostname; no connection attempted; fails instantly | Record missing, resolver unavailable, negative cache, or in-cluster DNS saturated | Resolve the name from the failing client's position (DNS in Production) |
| Certificate expired or chain incomplete | TCP connects fine; the failure is at handshake | Expiry, missing intermediate, or a name not covered | Inspect the served certificate from outside (Certificates as an Operational Object) |
| Port blocked by a filter that drops | Timeout with no response at all | A security group, network ACL or firewall silently discarding packets | Check the rules on both directions of the path; drops are the default behaviour |
| Port closed or actively rejected | Connection refused, immediately | Nothing listening, or a filter configured to reject | Refused is fast and informative — something answered and said no |
| Security group changed for another workload | Sudden failure with no deploy of the affected service | A shared rule edited elsewhere | Check infrastructure change history alongside deploy history (Change Correlation) |
| Connection pool or descriptor limit reached | Abrupt failure at a threshold, under load, then recovery | A ceiling on one side of the connection | Compare live connection counts to every ceiling you have (The Connection Budget) |
| NAT port exhaustion | Connections to ONE destination fail while everything else is fine; correlates with load | Many short-lived outbound connections behind a shared address exhaust the port range for that destination tuple | Reuse connections, add translation addresses, or spread destinations (NAT: Many Private Hosts Behind One Public Address in the networking view) |
| Packet loss on the path | Latency and variance rise; error rate barely moves | Retransmission hides loss as delay until it cannot | Look at retransmissions and jitter, not at error rates (Packet Loss Buys You a Timeout, Not a Retransmit in the observability view) |
| Latency spike, one path | Tail latency up, median flat, no errors | A congested link, a cross-zone hop, or a saturated middlebox | Compare per-zone and per-path latency; averages will show nothing (Percentiles: Which One, and How Many Users Is That? in the observability view) |
| Load balancer at its own ceiling | Connection failures with every backend healthy | The balancer is a shared capacity limit | Treat it as capacity to plan (Operating a Load Balancer) |
| Asymmetric reachability | A can reach B; B cannot reach A | Rules are directional and were only added one way | Test both directions explicitly; assumptions of symmetry are wrong more often than not |
| No timeout configured | One unreachable dependency stops everything the caller does | Workers blocked indefinitely on a connection that will never complete | Bound every network call (Timeouts in the backend view) |
Localise in four questions
Run these in order from the failing client's network position. Each one eliminates a layer, and the first that fails is where the problem is — which is usually reached before the third.
- 1Does the name resolve?
Resolve the hostname from the client's position, not from yours.
fails by It resolves for you and not for the client — different resolver, different view, different cache.
evidence An address is returned, and it is the address you expect (DNS in Production).
- 2Does a connection establish?
Open a TCP connection to the address and port.
fails by Refused means something said no; a timeout means nothing answered — a filter, a route or a drop.
evidence The connection completes, and you noted which of the two failures you got.
- 3Does the handshake succeed?
Complete TLS and inspect the certificate presented.
fails by Expiry, incomplete chain, or a name the certificate does not cover (Certificates as an Operational Object).
evidence A valid certificate covering the requested name, verified from a client without cached intermediates.
- 4Does the service respond?
Send a real request and read the status and the timing.
fails by A proxy-generated error means no ready upstream; an application error means you have left the network (Operating the Edge).
evidence A response whose origin you can identify — edge or application.
- 5If all four pass
The path works and the problem is intermittent: capacity, loss or a limit reached under load.
fails by Concluding "the network is fine" — an intermittent failure passes a point-in-time check every time.
evidence Connection counts against ceilings, retransmissions, and per-path latency over time.
The last step is the one that matters for the hard cases. NAT port exhaustion, connection limits and packet loss all pass a single check and fail under load, which is why a green check is not an answer.
How to do it properly
Most important first.
- Localise before diagnosing: does it resolve, does it connect, does it handshake, does it respond. Four questions, each eliminating a layer (Reading a Broken Workload has the same structure for workloads).
- Distinguish refused from timed out on every report. They point at opposite halves of the possibility space and cost nothing to check.
- Test from the same network position as the failing client. A check from your laptop, from a bastion or from another zone is a different path and can succeed while the real one fails.
- Suspect shared network resources when a failure correlates with load rather than with a change — NAT capacity, load balancer limits and connection pools all fail at a threshold (Building a Capacity Model).
- Reuse connections. Connection pooling is the direct mitigation for both NAT port exhaustion and handshake overhead, and it is usually a client configuration nobody has set (Connection Pooling in the networking view).
- Check what changed in the network, not only in the application. Security groups, route tables and firewall rules are production changes with no deploy record unless you made one (The Audit Trail).
- Set timeouts everywhere, so a network failure becomes a bounded error rather than an exhausted thread pool (Timeouts in the backend view).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
The shared components on this list — DNS, the edge certificate, the NAT gateway, the load balancer, a subnet's security group — have no containment at all: they are on the path for everything in their scope and they fail completely. What varies is the scope itself, which is why segmentation, per-zone redundancy and separate egress paths are the containment, decided long before the incident.
What can go wrong
- Investigating the application for a fault that is entirely in the path between two healthy applications.
- Testing from the wrong network position and concluding the network is fine.
- Retrying into a network failure, converting a partial problem into an overload of whatever is still working (Retry Storms: The Load You Generated Yourself in the backend view).
- Missing NAT port exhaustion because it presents as one destination being unreachable while every other check passes.
- Treating packet loss as an application performance problem because it produces latency rather than errors.
- A security group or firewall change made for an unrelated workload, with no record connecting it to the outage it caused.
- No timeouts, so one unreachable dependency exhausts the caller's workers and the outage spreads to everything that service does (Cascading Failure: When the Response to Failure Causes More Failure in the backend view).
- "It timed out, so the dependency is down." A timeout most often means nothing answered — dropped packets, a filter, a wrong route. The dependency may be perfectly healthy.
- "Connection refused means the service is down." It means something actively refused: nothing listening on that port, or a filter configured to reject rather than drop.
- "Packet loss would show up as errors." It shows up as latency and variance. The transport hides it by retransmitting, which is the whole point of the transport.
- "We are not near any connection limit." Which limit? Client pool, server maximum, load balancer ceiling and NAT translation capacity are four different ceilings and you are only watching one.
- "The network is reliable inside our own infrastructure." It is reliable enough that its failures are unfamiliar, which makes them slower to diagnose rather than less frequent.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Which layer failed: resolution, connection, handshake or response — established by a check per layer rather than inferred from an application error.
- Whether the failure is refused or timed out, and whether it affects one destination, one zone, or everything.
- Whether the failure correlates with load or with a change, which separates a capacity limit from a configuration mistake (Change Correlation).
- Connection counts against known limits: pool size, load balancer ceiling, database maximum connections, NAT translation capacity (The Connection Budget).
- Retransmission and loss indicators on the path, which distinguish a lossy network from a slow application (Packet Loss Buys You a Timeout, Not a Retransmit in the observability view).
- Network configuration changes — security groups, route tables, firewall rules, DNS records — are usually reversible immediately, and are frequently the fastest fix available once identified (The Plan: Desired vs Current).
- Capacity failures are not rolled back, they are relieved: fewer connections, more translation addresses, larger pools, or less traffic. That is mitigation rather than repair (Stop the Harm Before You Understand It).
- Connections dropped during the failure are gone. What protects the work is retry with a bound and idempotency, not the network recovering (Idempotency in Backends in the backend view).
- Automate reachability checks between the components that must talk to each other, from the network position that matters, continuously. A synthetic check catches a filter change before a user does (Dashboards an Operator Can Act On).
- Automate limit monitoring: connection counts against ceilings, translation capacity, pool utilisation. These fail at a threshold, so a trend is enough warning (Headroom).
- Keep network changes reviewed and human. A security group edit is a production change with a wide blast radius and, applied automatically from a broad policy, can partition a running system (Destructive Changes: What a Rename Really Does).
- Aggressive timeouts fail fast and turn a slow dependency into an error; generous ones preserve slow successes and tie up capacity while doing it.
- Connection reuse reduces handshake cost and NAT pressure and holds resources open on both ends, which is its own kind of exhaustion.
- Strict network segmentation limits lateral movement and makes every legitimate new connection a change request, which is a real operational cost and a real security benefit (Network Segmentation in the security view).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- CLOUD-SPECIFICSecurity groups, network ACLs, managed NAT gateways and load balancer limits are cloud constructs with per-provider semantics — in particular whether a filter drops silently or rejects, which decides whether you see a timeout or a refusal. On-premises the same failures exist behind firewalls and routers with different tooling and usually far less visibility.
- GENERALThe layer-by-layer localisation — resolve, connect, handshake, respond — and the refused-versus-timeout distinction hold on every network, including a laptop talking to a service on the same machine.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Distributed Systems — partial failure and why a timeout is genuinely ambiguous about whether the work happened.