The question this answers
One source record changed. How many cached things are now wrong, where are they, and can I actually reach them all?
Invalidation reduces *expected* staleness; the TTL bounds the *maximum*. The maximum can be tightened below the TTL only for cache tiers that are enumerable and that acknowledge delivery — which excludes browsers, most CDN edges, and any instance that joined the fleet after the message was sent.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The invalidating writer knows which source record changed and which keys it believes are derived from it. It does not know the full set of derived keys (nobody wrote that down), which cache tiers currently hold them, or which browsers are holding a copy. It is broadcasting a correction to an audience it cannot enumerate and receives no meaningful acknowledgement from.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Derived-key fan-out: one write, ten thousand wrong keys
A cached value is rarely a copy of a row. It is a *derivative*: a rendered fragment, a search result page, a recommendation list, a permission-filtered view, an aggregate. So the mapping from source records to cache keys is many-to-many, and it is usually implicit — encoded in whatever query built the value, and written down nowhere.
Change one product’s price and you have invalidated: the product page fragment, every category listing containing it, every search result page it appears on, every "similar items" list referencing it, every user’s cart summary containing it, and the homepage bestseller module. That is easily ten thousand keys for one write, and the set is not computable from the write alone. Nobody recorded which search result pages happen to contain product 42.
Two honest strategies exist. Tag-based invalidation: record, at fill time, which source entities contributed to each cached value, and invalidate by tag. It is correct, and you now maintain a reverse index whose size grows with the product of keys and entities. Key versioning: include a version or generation in the key (search:v83:query=shoes) and bump the version on change, so all derived keys become unreachable at once and eviction reclaims them lazily. One write invalidates everything logically, nothing has to be enumerated, and the price is a version read on the read path plus garbage in the cache.
Versioning at a coarse namespace level — one generation counter per entity type or per tenant — is the pragmatic answer most large systems land on. It over-invalidates, which costs hit rate, and it removes the enumeration problem entirely, which is worth a lot of hit rate.
The invalidation firehose is a load problem of its own
Invalidation traffic scales with writes × derived keys × cache tiers × instances, and each of those factors is one you have grown deliberately. A nightly price import updating five million rows, each touching ten derived keys, broadcast to five hundred instances, is 25 billion delivery events for a job that was described in the ticket as "refresh prices".
What breaks first is usually the messaging layer, not the cache: the topic backs up, invalidation lag exceeds the TTL, and the invalidations that eventually arrive are for entries that already expired — a great deal of work achieving nothing. Second to break is the origin, because every invalidation is a future miss and a bulk invalidation manufactures the stampede described in One Key Expires and Five Hundred Instances Miss at the Same Millisecond.
The mitigations are about *shape*, not throughput. Coalesce: within a window, collapse repeated invalidations for the same key. Batch: send one message listing a thousand keys rather than a thousand messages. Rate-limit deliberately: spread a bulk invalidation over minutes so that the resulting misses arrive at a rate the origin can serve. Use a generation bump for bulk changes: an import that touches everything should increment one namespace version rather than emit five million messages — a single write that logically invalidates the world.
The rule of thumb worth internalising: if an operation would emit more invalidations than the source emitted writes, it should probably be a version bump instead.
source writes 5,000,000 rows derived keys per row x10 cache tiers (near + shared + cdn) x3 instances holding a near cache x500 per-key broadcast => ~25,000,000,000 delivery events topic backlog hours; lag exceeds TTL resulting origin misses 5,000,000 within the import window namespace generation bump => 1 write logical effect every derived key unreachable origin misses spread naturally by real traffic cost hit rate falls until the cache refills
Ownership, and the tiers you cannot reach
Every cache key needs an owner: the service permitted to write it and responsible for invalidating it. Shared keys written by several services are the single most reliable source of long-lived cache bugs, because each writer knows only its own reasons for invalidation and none of them knows the whole rule. A cache key with two writers has no invalidation contract, only two partial ones.
Then there are the tiers you do not control. A CDN exposes a purge API, but a global purge is not instant — it propagates over seconds to minutes and its completion is often not observable. A browser cache has no purge mechanism at all; whatever Cache-Control you sent is the contract, and you cannot change your mind. An instance that started after your invalidation was published never received it, so any per-instance cache warmed from the source before the invalidation is fine, but one warmed from a stale shared tier is not.
This is the reason the guarantee at the top of this lesson is stated the way it is. The TTL you set is the promise you can actually keep. Invalidation makes the common case fresher. If a value would be harmful when 30 minutes stale, the answer is a TTL under 30 minutes — not a more reliable invalidation pipeline, because for the tiers that matter most (edge, browser) no such pipeline exists.
The practical consequence for design: choose TTLs from a statement of harm ("a wrong price is unacceptable after 60 seconds"), then use invalidation to make the typical case much better than that bound. Teams that do it the other way round — long TTLs justified by invalidation — are the ones with multi-hour staleness incidents, and the incident is always the day invalidation silently stopped working.
| Tier | Enumerable | Purge latency | Delivery confirmable | Real bound |
|---|---|---|---|---|
| Shared remote cachetypical | Yes (one place) | ~1ms | Yes | Invalidation |
| Per-instance near cachetypical | Only current instances | ms + broadcast lag | Per subscriber, if measured | Near-cache TTL |
| CDN edgetypical | No | seconds to minutes | Rarely | Edge TTL |
| Browser cacheprotocol | No | Never | No | Cache-Control you already sent |
| Client app in-memoryprotocol | No | Never | No | App session lifetime |
Key points
- Cached values are derivatives, so one source write invalidates a many-to-many set of keys that is usually not computable from the write.
- Invalidation traffic scales as writes × derived keys × tiers × instances, and a bulk update can exceed the messaging layer’s capacity by orders of magnitude.
- A namespace generation bump replaces an enumeration problem with one write, at the cost of over-invalidating and losing hit rate.
- Every key needs exactly one owning writer; keys with two writers have two partial invalidation contracts and no complete one.
- CDN and browser copies cannot be reliably reached, so the TTL is the only bound you can actually promise.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • A source write occurs and the owning service determines which cached derivatives are affected — by tag, by convention, or by bumping a generation.
- • Invalidations are coalesced within a window and batched into fewer, larger messages.
- • Delivery is rate-limited so that the resulting cache misses arrive at a rate the origin can absorb.
- • Tiers that cannot be reached — edge and client caches — are governed by the TTL and cache headers set at fill time.
- • Reconciliation samples cached values against the source, because delivery success is not evidence of freshness.
- • The derived-key set is incomplete, so some derivatives stay stale until TTL with nothing to correct them.
- • A bulk operation floods the invalidation channel and lag exceeds the TTL, making the whole pipeline pointless for that period.
- • A CDN purge partially completes, leaving some edges serving old content with no way to tell which.
- • A second writer updates a key without invalidating its derivatives, because the invalidation rule lives in the first writer’s code.
- • Invalidations arrive faster than the origin can serve the resulting misses, converting a correctness action into an overload event.
- • Regional inconsistency after a purge: users in one region see the old page for several minutes after a global purge reported success. CDN propagation is partial and its completion is not observable.
- • Invalidation backlog exceeding TTL: consumer lag on the invalidation topic sits above the TTL for hours after a bulk import, so entries expire on their own and the pipeline is doing pure waste.
- • Stale derivative with a fresh primary: the product page shows the new price and the category listing shows the old one. The primary key was invalidated and the derived set was not enumerated.
- • Import-triggered origin overload: a nightly job completes and the database saturates minutes later as the invalidated keys are re-fetched. The load looks unrelated to the job because it lags it.
- • Invalidation is one-way broadcast: cheap, unacknowledged, and correspondingly weak as a guarantee.
- • Tag-based invalidation requires maintaining a reverse index that both writers and fillers agree on — real shared state with real consistency requirements of its own.
- • Generation counters need only a monotonic counter per namespace, which is the cheapest coordination available and the reason the technique scales as well as it does.
- • If the invalidation pipeline stops entirely, the system silently degrades to TTL-bounded staleness with no error signal at all.
- • Under partial delivery, some tiers are fresh and others are not, so users observe inconsistency between pages rather than uniform staleness — which is far more confusing to diagnose.
- • Unreachable tiers are unaffected by any failure in the pipeline, because they were never covered by it in the first place.
- • Detect: monitor invalidation lag against the TTL. Lag exceeding the TTL means the pipeline is doing no useful work and nothing else will tell you.
- • Contain: rate-limit bulk invalidations, and use a generation bump for anything that would emit more invalidations than the source emitted writes.
- • Recover: bump the namespace generation to invalidate everything logically when the derived set is unknown or the pipeline is behind.
- • Reconcile: sample derived keys against freshly computed values and report divergence, per tier, since delivery metrics say nothing about freshness.
- • Verify: change one source record and confirm every intended derivative refreshes within its stated bound, including the edge. Most teams have never traced this path end to end.
- • Invalidation lag compared against the TTL — the ratio, not the raw number, is what tells you whether the pipeline is contributing anything.
- • Invalidations emitted per source write, which reveals derived-key fan-out and flags operations that should be generation bumps.
- • Origin miss rate following bulk operations, so the manufactured stampede is attributed to its cause rather than treated as a mystery spike.
- • Sampled divergence between cached derivatives and freshly computed values, per tier, including the edge.
- • Systems with expensive derived values and moderate write rates, where invalidation genuinely keeps the typical case far fresher than the TTL.
- • Content platforms with a CDN, where a purge on publish is the difference between seconds and hours of staleness for the common case.
- • Multi-tenant systems where a per-tenant generation counter turns an unbounded enumeration problem into a single write.
- • High write rates with wide derived-key fan-out, where invalidation traffic dwarfs the workload and the cache is mostly cold anyway.
- • When invalidation is used to justify long TTLs, which is precisely the configuration that produces multi-hour staleness the day the pipeline breaks.
- • When the derived-key mapping is guessed rather than recorded, giving the appearance of invalidation with silent gaps.
- • Short TTLs with no invalidation at all: simpler, predictable, costs origin load, and is the right answer far more often than it is chosen.
- • Key or namespace versioning, which removes enumeration and delivery entirely at the cost of hit rate and garbage.
- • Cache immutable content keyed by content hash, so nothing ever needs invalidating and new content is simply a new key.
- • Push updates to a materialised view maintained by the writer, converting invalidation into an update with defined semantics — see Materialized Views: A Read Model That Lags.
Invalidation at scale: TTL is the bound, invalidation is the optimisation
| Strategy | Messages/s | Hit rate | What it costs | Correct? |
|---|---|---|---|---|
| enumerate the keys | 80.00M/s | 95% | requires a mapping nobody recorded — which search result pages contain product 42? | only if the mapping is complete |
| tag-based | 800K/s | 95% | a reverse index whose size grows with keys × entities, maintained at fill time | yes |
| key versioning | 40K/s | 83% | over-invalidates, plus a version read on the read path and garbage left in the cache | yes |
| TTL only | 0/s | 91% | nothing to build; you accept staleness up to the TTL | eventually |
What people believe, and what is true
We invalidate on write, so the cache is never stale.
Only for tiers you can reach and enumerate. The edge propagates slowly, the browser cannot be reached, and derived keys you did not think of were never invalidated.
A long TTL is fine because invalidation keeps things fresh.
The TTL is the maximum staleness you have promised. The day invalidation breaks — silently, as it does — you serve that maximum, and it is a multi-hour incident.
Bulk imports should invalidate every affected key.
That emits more messages than the import had rows and manufactures a stampede. Bump a generation instead: one write, everything logically invalid.
Any service that changes the data can invalidate the cache.
Then the invalidation rule exists in several places and is complete in none. One owning writer per key, or the contract does not exist.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
One record changing can make thousands of cached things wrong, and some of those copies are in browsers and CDN nodes you cannot reach. Set a TTL you could live with if invalidation never fired, then use invalidation to do better most of the time.
Practical
Give every key exactly one owning writer, record derived-key relationships or use versioned keys, coalesce and batch invalidations, rate-limit bulk operations, and monitor invalidation lag against the TTL. For anything that would emit more invalidations than writes, bump a generation instead.
Advanced
Model the cache as a materialised view whose refresh channel is unreliable and whose consumers are not enumerable. That framing sets the guarantee correctly — maximum staleness is the TTL, expected staleness is the delivery latency — and tells you where each technique fits: tags maintain a reverse index for precision, generations trade precision for a single write, and content-addressed keys make invalidation unnecessary by removing mutability from the naming scheme.
Apply it
- 💬 One product price changes. List the cached things that are now wrong, and say which of them you can actually reach.
- 💬 A nightly import updates five million rows. What is wrong with invalidating each affected key, and what would you do instead?
- 💬 Why is the TTL, rather than invalidation, the bound you can promise?