DNS in Production
A DNS change is not an action, it is an expiry schedule. TTL decides how long the old answer keeps being used, and several caches between you and the user do not necessarily obey it.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
I changed a DNS record. Who is still using the old answer, and for how long?
Names are resolved through a chain of independent caches you do not control, so a change you make now takes effect at different times for different clients — and sometimes not at all.
Update the record and the change is live. DNS propagation takes a few minutes; wait a bit and everything will be pointing at the new address.
Nothing propagates. Authoritative records are updated instantly; every cached copy elsewhere expires on its own schedule, and the schedule was set by the TTL that was in effect before you made the change.
- Nothing propagates. Authoritative records are updated instantly; every cached copy elsewhere expires on its own schedule, and the schedule was set by the TTL that was in effect before you made the change.
- Lowering the TTL as part of the change does not help. Resolvers that already cached the old record cached the old TTL with it, so the shortening only affects lookups made after they expire.
- Several caches ignore the TTL entirely. Some runtimes cache resolved addresses for the process lifetime, some connection pools resolve once and hold the connection, and some corporate resolvers apply their own floor (Connection Pooling in the networking view).
- Negative answers are cached too. A name that did not exist when someone looked it up stays non-existent for them for the negative caching period, which is why a newly created record can appear broken for exactly the clients who tried it early.
- Because of all this, DNS is a poor failover mechanism and an excellent way to be surprised: the fraction of traffic that moves is a function of client behaviour you cannot inspect.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A record has a TTL, which is an instruction to every cache: you may keep this answer for this long. A cache that received the record starts its own timer, so the effective expiry is spread out across all the clients that looked it up at different moments.
- The chain is typically: application or runtime cache → local stub resolver → a recursive resolver operated by the network or a public provider → your authoritative servers. Each link caches independently (Following One Lookup Through Every Cache in the networking view).
- Lowering a TTL is therefore a change you make ahead of the change you care about, by at least the old TTL, so that every cached copy has expired and been replaced by one carrying the shorter value.
- The cost of a low TTL is query volume and dependency: every expiry is another lookup, and if your authoritative servers are unreachable when it happens, the client has no cached answer to fall back on.
- Inside a cluster, resolution has its own layer with its own behaviour — a search path that turns one lookup into several attempts, and a cluster DNS service that is itself a workload that can be saturated or unavailable (Service Discovery in Operation).
- DNS-based failover therefore has two independent delays: detecting the failure and changing the record, and the record change reaching clients. Only the first is under your control.
Changing a record is scheduling an expiry
The mental model that prevents most DNS incidents: you are not switching traffic, you are publishing a new answer and waiting for old answers to expire, on timers that started before you did anything.
- 1Lower the TTL
Publishes the same record with a shorter TTL so future caches hold it briefly.
fails by Skipped, or done at the same time as the change — in which case it has no effect on anything already cached.
evidence The record returns the new TTL when queried from an external resolver.
- 2Wait out the old TTL
Lets every cache holding the long-TTL answer expire and re-fetch the short-TTL one.
fails by Impatience. This wait is the entire mechanism; skipping it makes the rest theatre.
evidence Elapsed time exceeds the previous TTL, with margin.
- 3Publish the change
Updates the record to the new target.
fails by Typo, wrong record type, or applied to only one view in a split-horizon setup.
evidence Authoritative servers return the new value.
- 4Watch both endpoints
Observes traffic draining from the old target and arriving at the new one.
fails by Only watching the new one, so a stuck fraction of clients is never noticed (A Successful Deploy Is Not Evidence of a Healthy System).
evidence Request rate at the old endpoint falls towards zero.
- 5Keep the old endpoint alive
Serves the stragglers whose caches have not expired or who never honour TTLs.
fails by Decommissioning on the schedule the TTL implies rather than on the traffic you can see.
evidence The old endpoint has served no requests for a period you chose deliberately.
- 6Restore the TTL
Raises it again so normal operation does not pay for permanent low-TTL query volume.
fails by Forgotten, leaving a permanent dependency on your authoritative servers being reachable.
evidence Query volume returns to its baseline.
Five of the six steps are about time rather than about the record. That ratio is the lesson.
What DNS is and is not good at moving
DNS is frequently pressed into service as a failover mechanism because it is the one control that sits above everything. It moves traffic; it does not move it quickly or completely, and the difference matters most in exactly the situation you would reach for it.
Which layer do you move it at?
when Planned migrations, region moves, and anything where minutes are acceptable and clients are ordinary.
cost Bounded below by the TTL and unbounded above by clients that cache forever. Partially reversible on the same timescale (Region Failover).
when The endpoints sit behind one stable address you control.
cost Only works within the load balancer's reach — same region, same provider, usually same VPC (Operating a Load Balancer).
when You need failover to be a routing decision rather than a name resolution one.
cost Infrastructure most teams do not operate themselves; usually acquired through a provider, with its own opaque behaviour.
when The caller can be told where to go — a feature flag, a config push, a control-plane instruction.
cost Requires the callers to be yours and to be listening, which rules out anything with third-party clients (Feature Flags: Deploy Is Not Release).
The failures, and which cache caused each one
DNS failures are diagnosed by asking who answered and with what — the cause is almost always identifiable from which layer still holds the old answer.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Record changed without lowering TTL first | A stubborn fraction of traffic still hits the old endpoint | Caches are holding answers with the long TTL they were issued | Wait it out and keep the old endpoint serving; there is no way to force expiry |
| Name queried before it existed | Some clients cannot resolve a record that plainly exists | Negative caching of the earlier non-existent answer | Wait out the negative TTL; create records before publishing the name |
| Runtime caches resolution for process lifetime | One service never follows the change until it is restarted | A client-side cache that ignores TTLs entirely | Configure the runtime's cache; bound connection lifetimes |
| Cluster DNS saturated or unavailable | Every service fails to reach every other service | The in-cluster resolver is a workload and it is a shared dependency | Treat it as tier one: capacity, probes, alerting (An Alert Should Demand Action) |
| Record deleted or mistyped | Total, immediate, widening failure as caches expire | A one-line change with no blast-radius limit | Restore and wait; prevention is review and IaC, not speed (The Plan: Desired vs Current) |
| Split-horizon change applied to one view | Works internally, fails externally, or the reverse | Two zones for one name and only one was updated | Check both views explicitly; make the pipeline apply both |
| Domain or delegation expiry | Everything fails at once, including things that have not changed in years | A calendar event nobody owned | Auto-renew plus expiry monitoring with a long lead time (Renewal: Automating the Thing That Expires) |
| Low TTL plus authoritative outage | Resolution fails quickly and widely | No cached answers remain to fall back on | Redundant authoritative providers; raise the TTL when agility is not needed |
How to do it properly
Most important first.
- Lower the TTL well before a planned change — at least the current TTL ahead of it — and raise it again afterwards once the change has settled.
- Choose the TTL from what you intend to do with the record. Records you may need to move under pressure get a short one and pay for it in queries; stable records get a long one.
- Do not rely on DNS for fast failover. Use it to move traffic between endpoints on a timescale of minutes, and use a load balancer or anycast address for anything that has to move faster (Region Failover).
- Know your clients' caching behaviour, especially any runtime that caches resolution for the process lifetime — that is a property of your callers, not of your records.
- Keep DNS records in version control and apply them through the same pipeline as everything else. A hand-edited zone is a production change with no review and no audit trail (Infrastructure as Code).
- Alert on resolution from outside your network, from more than one vantage point. Resolution working from your laptop proves very little (Dashboards an Operator Can Act On).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Nothing you control once an answer is cached. A name is shared by every client of the service, and the only limiter is the TTL you set before the change — which means containment for a DNS change is decided in advance or not at all. Caching cuts both ways: it is why a mistake reaches everyone slowly, and why the fix does too.
What can go wrong
- A change made without lowering the TTL first, so a fraction of traffic keeps going to the old endpoint long after the migration is declared complete.
- A record deleted or mistyped, taking out every client that resolves it — the single highest-blast-radius one-line change most teams can make (Blast Radius: If This Is Wrong, How Much Does It Affect?).
- A negative answer cached widely because a name was queried before it existed.
- Cluster DNS saturated or unavailable, which presents as every service being unable to reach every other service — a total outage with no failing service.
- A registrar or delegation expiry, which no amount of TTL management protects against and which fails everything at once.
- Split-horizon configuration where internal and external clients get different answers, and a change is applied to only one view.
- A client-side resolver caching forever, so one caller is permanently pointed somewhere retired.
- "DNS propagation takes 48 hours." Nothing propagates and there is no global timer. Caches expire individually according to the TTL they were given.
- "I lowered the TTL, so the change will be fast." Only for lookups made after existing caches expire. Lowering it during the change is too late.
- "DNS failover gives us high availability." It moves most traffic on a timescale of minutes, for clients that respect TTLs. It is a migration tool, not a failover mechanism (RTO and RPO).
- "It resolves correctly for me." Your resolver is one of many, and yours is the one most likely to have been flushed.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Resolution checked from several external vantage points and from inside the network, showing the new answer.
- Traffic at the old endpoint falling to zero, which is the only real proof the change took effect — the record itself only proves what you published.
- Query volume against your authoritative servers after a TTL change, so a low TTL does not quietly become a load problem.
- Certificate and delegation expiry dates monitored on a schedule long enough to act on, rather than discovered by an outage (Renewal: Automating the Thing That Expires).
- Restoring the previous record is instant at the authoritative server and slow everywhere else — the rollback is subject to exactly the same TTL as the change was.
- This asymmetry is the reason DNS changes deserve more caution than their size suggests: you can make the change in one second and cannot unmake it for the length of the TTL.
- If the change was a deletion, the rollback also has to contend with negative caching, so recovery can be slower than the original TTL implies.
- Automate the record changes themselves through code review and a pipeline. Manual zone edits are the classic unreviewed production change (Manual Production Changes).
- Automate the TTL-lowering step as part of any planned migration runbook, because it is the step people skip and it has to happen first (Runbooks).
- Keep the decision to fail over by DNS human. It is slow, partially reversible and affects everyone; automating it means an automated system can move all your traffic on a signal it may have misread (The Automation Trap).
- A short TTL buys agility and costs query volume, and makes your authoritative servers a more frequent dependency for every client.
- A long TTL buys resilience against an authoritative outage and means you cannot move quickly when you need to.
- Managed DNS with health-checked failover removes a lot of operational work and moves an important control into a provider whose behaviour you cannot inspect during an incident (DNS in Cloud Infrastructure in the cloud view).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALTTL semantics, caching layers and negative caching are properties of DNS itself and behave the same on every platform and provider.
- CLOUD-SPECIFICHealth-checked DNS failover, alias records that resolve to a provider-managed endpoint, and private zones split by view are provider features with different semantics and different failure behaviour. In particular, an alias to a provider load balancer changes who owns the address and how quickly it can move.
- KUBERNETES-SPECIFICIn-cluster resolution adds a search-path expansion that turns a single external lookup into several queries, and a cluster DNS service that is a workload with its own capacity, probes and failure modes. Neither exists on a VM or a PaaS, where the node's resolver is the whole story.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.