DNS Failure Modes: What Each One Looks Like
DNS fails in a small number of distinct ways — cache expiry, a dead resolver, a wrong record, an over-long TTL, a change in flight — and each produces a different signature (NXDOMAIN, SERVFAIL, timeout, a stale or split answer) that tells you which cache or server to look at.
The problem
Expire cache
The simulator’s first button empties every cache in the chain, which is what happens on a fresh device, after a resolver restart, or when a TTL runs out on a rarely-used name. The next lookup is a cold resolution — root, TLD, authoritative — and costs tens to a hundred milliseconds instead of under one (Following One Lookup Through Every Cache). Nothing fails; the first request is simply slow, and if it is the first request of a page it delays everything behind it.
The operational version of this is a thundering herd on the authoritative servers: a very popular name with a very short TTL expires simultaneously in many resolvers, and they all ask at once. Large resolvers mitigate with prefetching (refreshing popular records shortly before expiry) and with serving stale while a refresh is in flight. If your authoritative servers see load spikes every *N* seconds, *N* is your TTL.
Diagnostic: dig shows the query time. A cold answer is tens of milliseconds and arrives with the full TTL; a warm one is ~1 ms with a partially counted-down TTL. Two consecutive digs where the second is fast and the TTL has decreased tells you the resolver is caching normally.
Resolver down
If the recursive resolver is unreachable, the stub sends its query and hears nothing. The default Linux stub behaviour (resolv.conf options timeout:5 attempts:2, per configured server) means an application waits roughly 10 seconds per name server before giving up with EAI_AGAIN ("temporary failure in name resolution"). Every hostname on the machine fails identically; addresses typed as IPs still work. Users describe this as "no internet". It is the single most common cause of that report, ahead of any actual loss of connectivity.
A resolver that is *reachable but broken* — it cannot reach the authoritative servers, its upstream link is down, DNSSEC validation failed — answers immediately with SERVFAIL instead of timing out. That distinction is diagnostic gold: a timeout means the packets to the resolver are not getting there (or back); SERVFAIL means the resolver got the question and could not answer it. Both are resolver-side; neither is your zone.
Fixes are configuration, not code: a second resolver in resolv.conf (queried after the first times out, so it helps availability but not latency), shorter timeout/attempts, a local caching daemon, or for an application, a resolver library with its own parallelism. dig @<resolver> name isolates the resolver: if dig @1.1.1.1 works and dig alone hangs, the configured resolver is the problem.
Wrong record
Two different wrongs. The record does not exist — a typo in the zone, or the record was never created — and the authoritative server says NXDOMAIN. Resolvers cache that for the SOA minimum (Following One Lookup Through Every Cache), so creating the record does not fix clients that already asked. Or the record exists with the wrong value: a stale IP after a server move, a CNAME pointing at a decommissioned platform host. The lookup succeeds and the connection then fails at a different layer — a TCP timeout or ECONNREFUSED if nothing is listening there, or a TLS certificate mismatch if something *else* is (the tls-hostname-mismatch challenge is often a DNS story underneath).
The tell for a wrong record is that the failure is deterministic and universal once caches expire, and that dig +trace (which bypasses every cache) returns the wrong answer directly from the authoritative server. If +trace is right and dig (via the resolver) is wrong, it is not a wrong record, it is a stale cache — the next two buttons.
High TTL
A record with a TTL of a day, changed at noon, is served at its old value by every cache that fetched it before noon until it expires there — up to a day later, with each cache expiring at its own moment. The result is split-brain: for a full TTL period, the population is divided between the old and the new answer according to when their resolver last asked. There is no "propagation" in DNS; nothing is pushed anywhere. There is only expiry.
This is the mechanism behind "it works for me": the engineer who made the change probably tested with a resolver that had not cached the old value, or flushed their own caches, and sees the new answer at once. A user on an ISP resolver that refreshed at 11:59 sees the old one for another day. Neither is lying, and no amount of restarting the user’s browser changes what their resolver holds. Serve-stale resolvers can extend this beyond the TTL when your old authoritative servers stop answering.
Diagnostic: query several resolvers and compare TTLs — dig @8.8.8.8, dig @1.1.1.1, dig @<isp-resolver>, and the authoritative server directly with dig @ns1... +norecurse. Different answers with counting-down TTLs is a high-TTL split; the authoritative answer is the truth; and there is no fix except waiting, which is why DNS Record Types and What They Are For tells you to lower the TTL a day before you need it.
Change IP
The last button changes the record at the authoritative server while caches are warm, which is a migration. Combine it with the previous button and you have the realistic case: some resolvers hold the old answer, some the new, browsers hold their own copies, and long-lived connections (HTTP keep-alive, WebSockets, database pools — see Connection Pooling) never look the name up again at all, so they stay on the old address regardless of DNS. A service migrated purely by DNS is not migrated until every cache and every connection has turned over.
The migration playbook: lower the TTL early, keep the old address serving (or proxying to the new one) for at least the old TTL after the change, watch the old server’s traffic drain rather than assuming, then decommission. Clients that pin an IP after the first lookup — some JVM configurations cached forever by default, some connection pools resolve only at construction — need restarting, and are the ones that "still hit the old server three days later".
One table for the diagnosis
Each failure has a signature in the status field of a dig response and in how long it took. The table maps them to where the fault lives. The one-command version is dig +trace name compared against dig name: +trace shows the authoritative truth, plain dig shows what your resolver thinks, and the difference is the cache.
dig name— what the configured resolver says.dig @1.1.1.1 name— what a different resolver says.dig +trace name— what the authoritative servers say, bypassing all caches.getent hosts name(Linux) /dscacheutil -q host -a name name(macOS) — what applications actually see, including/etc/hosts.
Status / symptom Meaning Look at
NOERROR, right answer fine the next layer (TCP, TLS)
NOERROR, wrong answer stale cache, or the zone really says that compare dig vs dig +trace
NOERROR, empty answer name exists, no record of that type (NODATA) AAAA-only? CNAME to a name with no A?
NXDOMAIN name does not exist (or negative-cached) the zone; then the SOA minimum
SERVFAIL resolver could not resolve resolver upstream, DNSSEC, broken delegation
REFUSED server declines to answer for this client you asked an authoritative server recursively,
or an ACL
timeout (no response) resolver unreachable / UDP blocked resolv.conf, firewall, VPN, captive portal
slow first answer, fast cold cache normal; TTL and prefetch if it recurs
after, TTL counting downKey points
- Cache expiry is not a failure, just a slow first lookup — and a load spike on authoritatives if the name is popular.
- Resolver down = timeouts on every name after ~10 s; resolver broken = immediate SERVFAIL; IPs keep working either way.
- NXDOMAIN is cached; a record created after it was first asked for is invisible for the SOA minimum.
- There is no propagation, only expiry: a high TTL means a split population for a full TTL period, and "works for me" is a cache statement.
- Changing an IP does not move existing connections; pools and keep-alive connections stay on the old address until they reconnect.
digvsdig +traceseparates "cache" from "zone" in one comparison.
Why does this exist?
Mechanisms are answers to constraints. Open each question before reading the answer.
▸Why does DNS fail differently for different people?
Because every resolver and every device holds its own cache with its own expiry moment; the "system" is thousands of independent caches with no invalidation channel between them.
▸Why does a DNS outage feel like a total outage?
The lookup runs before the connection, so every request to every name fails at step one, and the error surfaces in the application as "cannot connect" with no hint that nothing was ever sent.
▸Why can’t the authoritative server push the new value?
It does not know who cached the old one — resolvers pull; there is no registration. Pull-with-expiry is what lets DNS serve the planet with no state about its clients.
How DNS fails
- Browser cache↓
- OS cache↓
- Recursive resolver↓
- Authoritative server
How it fails
What the failure looks like from inside real software.
- Migration cut-over with a 24-hour TTL: for a day, roughly half of all requests reach the decommissioned server; the graph of errors decays slowly instead of dropping.
- A negative answer cached during deploy: the new service’s health check fails for 30 minutes with NXDOMAIN after the record was created; retries do not help because they hit the same resolver.
- Resolver timeout in a container with a wrong
resolv.conf: every outbound call takes 10 seconds to fail; CPU idle, latency enormous, no errors in the logs of the service being called because it never received anything. - Split answer between the corporate resolver and browser DoH: the intranet name resolves in one tool and not another on the same machine.
- Long-lived connection pool never re-resolving: three days after the IP change, one service still talks to the old database host that is about to be deleted.
- DNSSEC validation failure after a key rollover mistake: validating resolvers (most large public ones) return SERVFAIL; non-validating ones work; the site is "down for some ISPs".