A 95% Hit Rate Tells You Almost Nothing
Hit rate is a ratio, and the thing that hurts you is a volume weighted by cost. The right question is never "how high is the hit rate" — it is which objects miss, how expensive each miss is, and how much load the misses put on whatever is behind the cache.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
A ratio hides the number that matters
Two systems both report a 95% hit rate. The first serves 1,000 requests per second, so it passes 50 misses per second to the database. The second serves 40,000 requests per second, so it passes 2,000 misses per second — forty times the downstream load at an identical, unchanged, perfectly green hit ratio. The ratio did not move because the ratio cannot express volume, and the database behind the second cache is the one having a bad day.
This is why the primary cache metric should be absolute misses per second, with hit ratio as context rather than as the headline. Misses per second multiplied by the cost of serving a miss gives the downstream load in a unit that can be compared against downstream capacity, which is the only comparison that predicts an outage.
The second thing a ratio hides is *which* objects miss. If the 5% that miss are cheap, uniformly distributed lookups, the cache is doing excellent work. If they are the twenty most expensive aggregate queries in the system, the cache is passing through nearly all of the actual cost while reporting a number that looks like success. Weight the miss rate by miss cost or the metric is decorative.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| Cache A — hit ratio | 95% | Looks healthy. | normal |
| Cache A — misses/sec | 50/s | 1,000 req/s x 5%. Database serves 50 queries/s. Comfortable. | normal |
| Cache B — hit ratio | 95% | Identical ratio. Identical dashboard colour. | normal |
| Cache B — misses/sec | 2,000/s | 40,000 req/s x 5%. Database serves 2,000 queries/s — 40x cache A at the same ratio. | smoking gun |
| Cache B — miss cost p99 | 180 ms | Misses are the expensive aggregate queries, not the cheap lookups. | smoking gun |
| Cache B — implied DB concurrency | ~360 concurrent | 2,000/s x 0.18 s (Little's Law). Far beyond the pool. This is the incident. | smoking gun |
| Eviction rate | 1,400/s | Entries are being pushed out before their TTL — the working set does not fit. | suspect |
The five readings that actually characterize a cache
Eviction rate deserves particular attention because it is the signal that predicts the future. A cache evicting heavily is telling you the working set is larger than the memory allocated, which means hit rate is already lower than it could be and will fall further as data grows. Eviction rate rising while hit rate holds steady is a warning that the cache is about to stop working, delivered weeks in advance.
Key distribution matters for the reason Hot Keys: When Aggregate Metrics Hide a Saturated Node covers in depth: a cache can have an excellent aggregate hit rate while a single node holding one popular key is saturated. Aggregate metrics average that away completely, which is why per-node and per-key-prefix breakdowns are worth the cardinality cost on a cache that matters.
Finally, measure staleness deliberately rather than inferring it from TTL. TTL is the maximum age you configured; actual served age depends on write patterns and eviction. A cache serving data that is correct-but-old is not a performance problem at all until someone decides how old is acceptable — and that decision belongs with whoever owns the invariant, not with whoever tuned the TTL.
| Signal | Answers | Cannot answer | Read it with |
|---|---|---|---|
| Hit ratio | What fraction of lookups are served from cache | How much load reaches the backend | Absolute request rate |
| Misses per second | Actual load passed downstream | Whether that load is expensive | Miss cost p99 |
| Miss cost (p50/p99) | What one miss costs the backend | How often it happens | Miss rate |
| Eviction rate | Whether the working set fits in memory | Which keys are being lost | Memory used vs allocated |
| Memory used / allocated | Headroom before eviction pressure | Whether the right things are cached | Eviction rate, key count |
| Per-key or per-prefix rate | Whether load is skewed to a few keys | Aggregate health | Per-node utilization (Hot Keys: When Aggregate Metrics Hide a Saturated Node) |
| Served age of entries | How stale the data users see actually is | Whether that staleness is acceptable | A stated freshness contract |
The load a cache is hiding is the load it will hand back
The most important property of a cache is not its hit rate — it is what happens to the system behind it when the cache stops working. A cache at a 95% hit rate is absorbing 20× the traffic the backend sees. If it fails, restarts cold, or has its keyspace invalidated by a deploy, the backend receives 20× its normal load with no warning and no ramp. That is not a cache problem; it is a capacity problem that the cache has been quietly deferring, and Cache Stampede: Everyone Misses at Once is the acute version.
So the number worth writing on the dashboard next to the hit rate is the multiplier: "this cache is absorbing 20× backend capacity". It converts an abstract percentage into the question a capacity plan can act on — can the backend survive losing this cache, and if not, what is the plan? Options include limiting concurrency to the backend so a cold cache degrades rather than collapses, warming the cache before shifting traffic, and staggering TTLs so expiry never synchronizes.
A cold-start test is the honest validation. Flush the cache in a load environment at production-like traffic and watch what the backend does. Teams that have run this test know their multiplier; teams that have not are relying on a cache never having a bad day, which is not a property caches have.
Key points
- Hit ratio is a percentage and cannot express volume: 95% at 40k req/s passes 40× the load of 95% at 1k req/s.
- The primary metric is misses per second multiplied by miss cost — that product is the load reaching the backend.
- Eviction rate rising while hit rate holds is an early warning that the working set has outgrown memory.
- Aggregate hit rate averages away key skew; a cache can be 97% healthy overall and have one saturated node.
- A cache at 95% is absorbing 20× backend capacity; the real question is whether the backend survives losing it.
Progressive depth
Overview
A cache hit rate is a ratio, and ratios cannot express volume. What matters downstream is misses per second times the cost of a miss. Ask which objects miss, how expensive they are, and what happens to the backend if the cache disappears.
Practical
Put absolute misses per second on the dashboard as the headline, with hit ratio beside it. Track eviction rate as the early warning for working-set growth. Publish the absorption multiplier — total requests divided by misses — so everyone knows how much backend load the cache is hiding. Then run a cold-start test and find out whether the backend survives it.
Advanced
Cache behavior under skewed key popularity does not follow the aggregate. With a Zipf-like distribution, a small cache captures most of the *requests* while leaving most of the *keyspace* uncached, which is why hit rate rises steeply with the first megabytes of memory and then flattens — each additional megabyte buys progressively less. That curve is why "double the cache memory" often produces a disappointing hit-rate improvement, and why cost-weighted caching of the expensive tail can beat uniform caching of everything.
Internals
The eviction policy determines which keys survive under pressure, and the classic trade-off is between recency and frequency: LRU is cheap and vulnerable to scans that flush the working set, LFU resists scans and adapts slowly to changing popularity — the same trade-off databases make in Buffer Replacement: LRU, Clock and Scan Resistance. The data-structure mechanics live in LRU Cache and LFU Cache; the invalidation problem, which is where correctness rather than performance is decided, is Cache Invalidation, Stampedes and Hot Keys.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Dashboard → false comfort: cache hit ratio steady at 95% for six months; no cache alert has ever fired.
- 2Traffic → volume: request rate grew from 1k/s to 40k/s over those months while the ratio stayed constant.
- 3Ratio → absolute: misses grew from 50/s to 2,000/s — a 40× increase in backend load that the ratio could not express.
- 4Misses → cost: the misses are aggregate queries at 180 ms p99, so implied backend concurrency is ~360 against a pool of 20.
- 5Concurrency → root cause: the cache has been silently deferring a capacity problem, and the backend is now saturated whenever the hit rate dips even slightly.
- • "95% hit rate, the cache is doing its job." At sufficient volume that 5% is more load than the backend can serve.
- • "Hit rate is flat, nothing has changed." Flat ratio with growing traffic means growing absolute miss load.
- • "The cache is fine, the database is the problem." The database is receiving exactly what the cache passes through.
- • "Eviction rate is high but hit rate is fine, so ignore it." Eviction is the leading indicator; hit rate is the lagging one.
- • "We can lose the cache, it is just an optimization." At a 20× multiplier, losing it means 20× instantaneous backend load.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Absolute misses per second as the headline series, with hit ratio shown beside it for context rather than instead of it.
- • Miss cost p50 and p99 — the latency of serving a miss from the backend — so misses can be weighted by what they cost.
- • Eviction rate and memory used against allocated, together, to see working-set pressure before hit rate falls.
- • Per-node request rate and per-key-prefix rate on caches where skew is plausible, accepting the bounded cardinality cost.
- • The absorption multiplier (total requests / misses) published on the dashboard as a capacity number, not a performance one.
- • Change the dashboard first: misses per second and the absorption multiplier as headline metrics, hit ratio demoted to context.
- • Weight by cost: identify which keys miss and what those misses cost, then cache the expensive ones deliberately rather than caching uniformly.
- • Address eviction pressure with memory, a smaller cached representation, or a shorter tail — an evicting cache has already lost hit rate you are paying for.
- • Bound the blast radius: limit concurrent backend calls on miss so a cold cache degrades gracefully instead of collapsing ([[cache-stampede]]).
- • Stagger TTLs and add jitter so expiry never synchronizes across a popular keyset.
- • Run a cold-start test at production-like traffic and record the actual backend peak; compare it against measured backend capacity.
- • Confirm the absorption multiplier on the dashboard matches the observed spike during that test — if not, the metric is wrong.
- • After cost-weighted caching changes, check backend total time attributable to misses fell, not merely that hit ratio rose.
- • Verify eviction rate dropped after any memory change, and that hit rate improved as a consequence rather than independently.
- • Per-key metrics catch skew and cost cardinality; bound them to prefixes or a top-N sample rather than raw keys ([[cardinality]]).
- • More cache memory improves hit rate and costs money that might buy backend capacity instead — compare them directly.
- • Longer TTLs raise hit rate and increase staleness, which is a correctness decision rather than a tuning one.
- • Limiting backend concurrency on miss protects the backend and makes cold-start slower for users, deliberately.
- • An alert on absolute misses per second crossing a fraction of measured backend capacity — it fires while hit ratio is still green.
- • An alert on eviction rate, which predicts hit-rate decay weeks ahead of it appearing.
- • A scheduled cold-start drill in a load environment, so the multiplier is a known number rather than an assumption.
- • A dashboard panel pairing cache misses with backend utilization, making the victim-versus-cause question a glance.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe two-cache comparison and the miss-cost figures are constructed to show the ratio-versus-volume distinction.
- SIMULATEDThe cold-start recovery curve is produced by a decay model in this app, not captured from a real flush. Real recovery depends on key popularity distribution, backend capacity and whether concurrency is limited.
- WORKLOAD-SPECIFICWhether a given hit rate is good depends entirely on request volume, miss cost and backend capacity. There is no universally good hit rate.