The question this answers
Where should the cached copy live, and what did putting it there cost me?
A cache guarantees only *bounded staleness under a TTL*: a reader may observe any value written within the last TTL, and nothing stronger. It does not guarantee that two readers see the same value, that a reader sees its own write, or that the cache and the source ever agree at a given instant.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A node reading a cache knows what its own copy contains and when that entry was populated. It does not know whether the source has changed since, whether a peer instance holds a different value for the same key, or whether an invalidation for this key was published and lost. Every cache hit is an assertion about the past presented as a fact about the present.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Three topologies, three different guarantees
Architecture covers caching as a pattern and Database covers cache-aside and write-through as storage mechanics. The distributed question is narrower and comes first: where does the copy live, and how many copies are there? That single choice determines the guarantee, the failure modes, and everything in the rest of this module.
Per-instance local cache. Every service instance holds its own copy in memory. Fastest possible read — no network, no serialisation — and no shared dependency to fail. The cost is that you now have n independent replicas with no coordination between them: 200 instances mean 200 copies that can disagree, and a value that must be invalidated must be invalidated 200 times. Staleness surface grows linearly with your fleet, and it grows automatically when you autoscale.
Shared remote cache. One logical copy in Redis or Memcached. All instances see the same value, so invalidation is one write and coherence between instances is free. The costs are a network round trip on every read, and a new dependency whose availability multiplies with yours. It is also a new place for Sharding Does Not Help a Single Key to bite, because one key now lives on exactly one shard.
Two-tier (near cache + remote). A small local cache with a very short TTL in front of the shared one. Absorbs hot keys locally, keeps most values coherent through the shared tier, and gives you a bounded staleness window equal to the near-cache TTL. It is the common production answer and it inherits *both* sets of problems — you now have n local replicas *and* a shared dependency, and the reasoning about which copy is authoritative gets genuinely harder.
| Topology | Read cost | Copies | Coherence between instances | New dependency |
|---|---|---|---|---|
| Per-instance localtypical | ~100ns | n (one per instance) | None — each drifts independently | No |
| Shared remotetypical | ~0.5–2ms RTT | 1 logical | Free — everyone reads the same entry | Yes, on the read path |
| Two-tiertypical | ~100ns hit / ~1ms miss | n + 1 | Bounded by the near-cache TTL | Yes, but absorbed on hit |
| No cacheprotocol | full source latency | 0 | Perfect | No |
The 95% hit rate is a 20× dependency on the cache
Here is the calculation every cache design must survive. A service handles 10,000 requests per second at a 95% cache hit rate, so the origin sees 500 requests per second and is comfortably sized for it. Now the cache becomes unavailable. The origin sees 10,000 requests per second — 20× its normal load — arriving in the same instant as the cache failure.
The origin was never provisioned for that and cannot be, in general: the whole point of the cache was to avoid provisioning for it. So a cache outage is not a latency regression, it is an origin outage. And the multiplier is exactly 1 / (1 − hitRate), which means the better your cache is doing, the more catastrophic its loss: 99% hit rate is a 100× multiplier.
That reframes the fail-open versus fail-closed decision, which most teams make implicitly by writing try { cache.get() } catch { return db.get() }. Failing open converts a cache outage into a database outage. Failing closed — returning an error when the cache is down — keeps the database alive but makes you unavailable for the duration. Neither is right in general; what is wrong is not deciding.
The defensible middle is fail-open *with admission control at the origin*: on a cache miss storm, the origin admits only what it can serve and sheds the rest, so it degrades rather than dying. That composes the cache decision with Decide at the Door Whether the Capacity Exists, and it is the only version of "fail open" that survives the arithmetic above. Security’s treatment of fail-open versus fail-closed is the general framing; this is the load-shaped instance of it.
hit rate normal origin load origin load on cache loss multiplier 50% 5,000/s 10,000/s 2x 90% 1,000/s 10,000/s 10x 95% 500/s 10,000/s 20x 99% 100/s 10,000/s 100x 99.9% 10/s 10,000/s 1,000x The better the cache works, the larger the cliff behind it.
The cache read is a remote call, with everything that implies
Once the cache is on another machine, cache.get(key) is subject to every property of a remote call: it can be slow, it can time out, and a timeout tells you nothing about whether it succeeded. Teams write cache reads as if they were memory accesses and then discover their p99 is set by a cache node’s GC pause.
Three consequences follow immediately. A cache read needs a timeout, and it should be aggressive — a cache that takes longer than the origin has negative value, so a timeout of a few milliseconds with an immediate fall-through is usually correct. A cache read needs a budget check: if the remaining deadline is 40ms and a miss costs 200ms at the origin, then the miss path cannot succeed and the request should fail fast rather than reading the cache first. A cache write after a miss is fire-and-forget: nothing about correctness depends on it landing, so it must never be on the critical path or hold the response.
The deeper point is that adding a cache moves you from one dependency to two, and your availability is now the *combination* — better than either if you fail open on cache errors, worse than either if you do not handle them. Most cache incidents are not staleness bugs. They are availability bugs caused by treating a network hop as if it were a hash-map lookup.
1async function cachedGet(key: string, ctx: Ctx): Promise<Value> {2 const budget = remainingMs(ctx)3 4 // If a miss cannot be served inside the budget, do not even try: reading the5 // cache first would just spend budget on a lookup we cannot act on.6 if (budget < ORIGIN_P95_MS && budget < CACHE_TIMEOUT_MS + ORIGIN_P95_MS) {7 throw new DeadlineExceeded('insufficient budget for a possible miss')8 }9 10 try {11 // Aggressive: a cache slower than the origin has negative value.12 const hit = await withTimeout(cache.get(key), CACHE_TIMEOUT_MS)13 if (hit !== undefined) return hit14 } catch {15 cacheErrors.inc() // counted separately from misses — different problem16 }17 18 const value = await origin.get(key, ctx)19 20 // Fire and forget: nothing about correctness depends on this landing, so it21 // must not hold the response or consume the caller's remaining budget.22 void cache.set(key, value, jitteredTtl()).catch(() => {})23 return value24}Key points
- A cache is a replica; choosing its topology chooses how many copies exist and what can disagree with what.
- Per-instance caches give you one replica per instance and a staleness surface that grows with the fleet.
- Origin load on cache loss is
1 / (1 − hitRate)times normal — a 95% hit rate is a 20× cliff, and 99% is 100×. - Fail-open on cache errors converts a cache outage into an origin outage unless the origin has admission control.
- A remote cache read is a remote call: it needs a tight timeout, a budget check, and a fire-and-forget write-back.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • A read consults the nearest cache tier; a hit returns immediately with a value of unknown age bounded by the TTL.
- • A miss falls through to the next tier and ultimately to the source of truth.
- • The fetched value is written back asynchronously with a jittered TTL so that entries populated together do not expire together.
- • Cache errors are treated distinctly from cache misses: a miss is normal, an error is a dependency failure.
- • The origin protects itself with admission control, because a cache failure presents as a step change in load rather than a ramp.
- • The cache node becomes slow rather than unavailable, and cache reads consume the budget without returning anything.
- • A cache restart or eviction wave empties the cache and the origin sees full traffic instantly.
- • A per-instance cache holds a stale value that no invalidation reached, and one instance answers differently from its peers indefinitely.
- • Serialisation cost dominates: the cache is fast and the deserialisation of a large object is not.
- • The cache write-back path holds the response, so cache slowness becomes user-visible latency.
- • Origin collapse on cache restart: database CPU goes from 15% to saturation within one second of a cache node cycling. Nothing in the application changed and no deploy occurred.
- • Latency set by the cache: service p99 tracks a cache node’s GC pauses. Cache hit rate is unchanged and hit *latency* is bimodal, which is the tell.
- • Instance-level answer divergence: two users refreshing the same page see different data depending on which instance they land on, with no error anywhere. Local caches drifted apart.
- • Silent availability coupling: overall availability sits below the origin’s own, because cache errors are being converted into request failures rather than into misses.
- • A per-instance cache requires no coordination and gives no coherence — the two facts are the same fact.
- • A shared cache provides coherence between instances by centralising the copy, buying agreement with a network hop and a dependency rather than with a protocol.
- • Any attempt to keep local caches coherent requires messaging between instances, which is Invalidation Is a Messaging Problem, Which Is Why Cache Bugs Are Hard and is where the genuinely hard problems live.
- • On cache unavailability, correctness is preserved — every read falls through to the source, which is always right.
- • Capacity is not preserved: the origin receives the full multiplier, so the honest statement is that the system trades staleness for a capacity dependency.
- • On stale reads, the guarantee is TTL-bounded staleness and nothing more; no read-your-writes property survives without additional machinery.
- • Detect: alert on origin request rate, not just cache hit rate. A hit-rate drop and an origin-load spike are the same event and the second one is what hurts.
- • Contain: admission control at the origin, so a cache failure degrades service instead of destroying the database.
- • Recover: warm the cache before returning an instance to service. A cold instance behind a load balancer is an origin-load generator.
- • Reconcile: after a coherence incident, identify keys whose local copies diverged and force expiry rather than assuming the next TTL will fix it silently.
- • Verify: kill the cache in a load test and watch what the origin does. Most teams have never run this and are surprised by the multiplier.
- • Origin request rate as the primary cache metric — it is what actually falls over, and hit rate is only a proxy for it.
- • Cache errors counted separately from cache misses, since one is a dependency failure and the other is normal operation.
- • Cache hit latency distribution, not just hit rate: a bimodal distribution reveals a cache node with pauses.
- • Cold-start origin load per newly launched instance, which tells you whether autoscaling events are self-inflicted origin spikes.
- • Read-heavy workloads with high key reuse, where the hit rate is high enough that the origin can be sized far below peak demand.
- • Expensive-to-compute values — aggregations, rendered fragments, model outputs — where the compute saved dwarfs the coherence cost.
- • Workloads that tolerate bounded staleness explicitly, which is most read paths and almost no write paths.
- • Low-reuse workloads, where the hit rate is poor and you have added a hop and a dependency for nothing.
- • Data with strict freshness requirements, where the TTL that would be acceptable is short enough that the cache barely helps.
- • Systems that cannot survive the origin multiplier, where the cache becomes a hard dependency masquerading as an optimisation.
- • Make the source fast enough not to need a cache — an index, a denormalised column, or a smaller query. Fewer moving parts and no coherence problem at all.
- • Precompute into a materialised view that is itself the read path, converting a cache into a derived store with explicit update semantics. See Materialized Views: A Read Model That Lags.
- • Push the copy to the edge (a CDN) where the staleness contract is already explicit and the origin multiplier is absorbed by the provider.
- • Cache only the expensive minority of keys, keeping the coherence surface small and the benefit most of what a full cache would give.
Where does the copy live, and how many copies are there?
| Topology | Read cost | Copies | Coherence between instances | New dependency |
|---|---|---|---|---|
| per-instance local | ~100 ns | 200 | none — each drifts independently, and autoscaling adds more | no |
| shared remote | ~1 ms RTT | 1 logical | free — everyone reads the same entry | yes, on the read path |
| two-tier | ~100 ns hit / ~1 ms miss | 200 + 1 | bounded by the near TTL (2 s) | yes, absorbed on hit |
| no cache | 25 ms | 0 | perfect | no |
What people believe, and what is true
A cache is a performance optimisation, so it cannot cause an outage.
At a 95% hit rate the origin is provisioned for 5% of demand. Losing the cache is a 20× load step, which is an outage by any other name.
Local caches are simpler than a shared cache.
They are simpler to operate and much harder to reason about: you have one replica per instance, no coherence, and an invalidation problem that scales with your fleet size.
Cache reads are fast, so no timeout is needed.
A cache read is a remote call. Without a tight timeout your p99 is set by the cache node’s worst moment, and a slow cache is worse than no cache.
Always fall back to the database on cache errors.
That is a reasonable default only if the origin can absorb the multiplier or shed. Otherwise you have converted a cache incident into a database incident.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Putting a cache on another machine, or one on every machine, means you now have copies of data that can disagree — and a new dependency whose failure hands the full load to the origin at once.
Practical
Choose the topology deliberately, jitter TTLs, treat cache reads as remote calls with tight timeouts and separate error counters, keep write-back off the critical path, and put admission control at the origin. Then test a cache outage under load, because that is the failure that actually takes systems down.
Advanced
Treat the cache as an unmanaged replica and ask the replication questions of it: what is the staleness bound, what is the convergence mechanism, what happens on a partition between the replica and the source? Answering them in those terms turns fuzzy cache bugs into familiar replication reasoning, and it makes the honest limits obvious — TTL is your only enforced bound, and everything else is best-effort messaging on top of it.
Apply it
- 💬 You run at a 95% hit rate and the cache cluster restarts. What does the origin see, and what should have been in place?
- 💬 When is a per-instance local cache the wrong choice even though it is faster?
- 💬 Why does a cache read need a timeout and a deadline check?