DNS Debugging: Who Answered, and With What?

When code asks for api.example.com, the answer comes from the first place in a chain that has one. The browser has its own cache (Chrome: chrome://net-internals/#dns). The OS has a hosts file, which wins over everything. The OS has a resolver cache — systemd-resolved on many Linux distributions, mDNSResponder on macOS, the DNS Client service on Windows. Then the configured recursive resolver (your router, the ISP, 8.8.8.8, the corporate one pushed by VPN) has a cache. Only after all of these miss does anyone walk root → TLD → authoritative. Each link caches for the record’s TTL, independently.

This is why "DNS propagation" is a misnomer. Nothing propagates. Each cache serves the old answer until its own copy of the TTL runs out, and caches that were populated at different moments expire at different moments. A change to a record with a 3600 s TTL is fully visible only after every cache that held the old value has aged it out — up to an hour after the change, and longer for resolvers that clamp minimum TTLs.

Four things in a dig reply matter. The status: NOERROR (an answer or an empty-but-valid one), NXDOMAIN (the name does not exist — the whole name, at the authoritative level or cached as such), SERVFAIL (the resolver failed to get an answer: broken delegation, DNSSEC failure, unreachable authoritative servers), REFUSED (the server will not answer for you — usually asking an authoritative server recursively). The flags: aa means the answer is authoritative, its absence means it came from a cache. The answer section: the record chain, often CNAME → CNAME → A, each with its own TTL. And the SERVER line at the bottom, which says who you actually asked.

The TTL column is a clock. An authoritative answer shows the configured TTL (300, 3600). A cached answer shows the remaining TTL counting down; run dig twice and watch it decrease. A TTL that resets to the full value on every query means you are hitting the authoritative server or a cache that just refreshed; a TTL of 3412 tells you the cache fetched it 188 s ago and will hold the old value for 57 more minutes.

An empty NOERROR answer (status NOERROR, ANSWER: 0) is the "NODATA" case: the name exists but has no record of the type you asked for — typically AAAA for an IPv4-only host, or asking A for a name that only has a CNAME whose target failed. See DNS Record Types and What They Are For.

dig @ns1.example.com api.example.com asks the authoritative server directly — the answer is the truth as of now, with aa set and full TTLs. If it disagrees with your default resolver, the record changed and a cache is serving the old value; the remaining TTL tells you for how long. If it agrees, the problem is not staleness. dig @8.8.8.8 and dig @1.1.1.1 give two more independent caches for comparison; disagreement between public resolvers is staleness, agreement with the authoritative is fine.

dig +trace api.example.com performs the recursive walk itself from the root servers, printing each delegation. It shows a broken delegation (the TLD points at name servers that do not answer or do not know the zone — SERVFAIL at the resolver) and lame or mismatched NS records. It also tells you which authoritative servers exist so you can query each one; if ns1 and ns2 disagree, a zone transfer or deployment is half-done. The stale-cache challenge in this module is that incident end to end.

Corporate and cloud DNS is frequently split-horizon: the same name resolves to a private address from inside the network (or over VPN) and a public address from outside. Connecting the VPN typically pushes a resolver and search domains; disconnecting it restores the old ones — but the OS cache may keep answers from the previous world for their TTL, so internal.example.com keeps resolving to 10.0.4.2 after you disconnect, then fails. resolvectl status / scutil --dns show which resolver is active per interface and per domain.

Browsers add their own layer. Firefox uses DNS-over-HTTPS to a configured provider by default in some regions, and Chrome "upgrades" to DoH when the system resolver’s operator supports it. A DoH-enabled browser never consults /etc/hosts or the VPN-pushed resolver — so the browser sees the public answer while curl, which calls getaddrinfo() and therefore the OS chain, sees the private one. curl --doh-url reproduces the browser’s view; the browser’s about:networking#dns / chrome://net-internals/#dns show its cache. When two clients on the same machine disagree, the question is always "which resolver did each one use?".

Containers and Kubernetes add a fourth resolver: pods resolve through kube-dns/CoreDNS with search domains that turn api into api.default.svc.cluster.local, and a ndots:5 default that sends external names through five search-domain attempts before trying them as-is — slow, and a source of NXDOMAIN storms. Kubernetes Networking, Just Enough covers it.

NXDOMAIN is cached too. RFC 2308 says a resolver may cache a negative answer for the minimum of the SOA record’s TTL and its MINIMUM field — often 300–3600 s, sometimes much longer. The classic incident: a developer tests new-service.example.com before the record exists, gets NXDOMAIN, creates the record, and then cannot resolve it for an hour from that machine or that resolver, while a colleague who never looked it up resolves it instantly. The fix is to query a resolver that has not seen the failure, or flush the one that has; the prevention is to create the record before anyone queries it.

Negative caching also means a resolver outage leaves a scar: if the authoritative servers were unreachable and the resolver returned SERVFAIL, that is typically not cached long (seconds), but a spurious NXDOMAIN from a misconfigured zone is, for the SOA minimum.