The question this answers
One key is taking more traffic than any single node can serve. What can I actually do about it?
Read throughput for a hot key can be raised to replicas × per-node capacity (key splitting) or effectively unbounded (near caches), in exchange for a bounded staleness window and, for splitting, a period during which different readers may observe different values. Write throughput for a single key is not improved by any of these techniques.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A cache node knows its own per-key request rate. A client knows which keys it requests. Neither can see the global popularity distribution without extra machinery, so a hot key is usually discovered from a saturated node rather than from the traffic pattern — which is why client-side detection with a sketch is worth building before the first incident rather than after it.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The constraint that makes this different from ordinary skew
Partitioning maps a key to a shard by hash, so all traffic for one key lands on one node. That is the property the whole design depends on — it is what makes lookups routable without coordination — and it is also an absolute ceiling. If a single Redis node serves on the order of 100,000 operations per second and one key is receiving 500,000, then no cluster size fixes it. A hundred-node cluster serves that key exactly as well as a one-node cluster.
This is why hot keys are qualitatively different from Hot Partitions: The Skew Hashing Cannot Fix, which is about data skew across ranges and is solvable by resharding. A hot *key* is the irreducible unit: it cannot be split by any partitioning scheme, because splitting it is precisely what partitioning cannot do.
The operational signature is unmistakable and worth recognising instantly: one node at 100% CPU while the cluster average sits at 10%. Every capacity dashboard says you have plenty of headroom. Adding nodes changes nothing, and the natural instinct — scale out — is the one action guaranteed not to help.
Where do these keys come from? A celebrity account in a social graph. A flash-sale product. A global feature flag or configuration key read on every request. A "trending" list. The shape is always the same: something that is per-request and shared by every request, which is exactly the access pattern partitioning cannot spread.
redis-cluster: 10 nodes node-07: cpu 99%, ops 98,400/s
others: cpu 9%, ops 9,100/s each
cluster ops total ~180,000/s cluster cpu avg 18%
add 10 more nodes -> node-07 still 99%. The key did not move.
increase maxmemory -> node-07 still 99%. It is CPU and network, not memory.
add read replicas -> helps ONLY if clients are routed to them for this key
near-cache the key -> node-07 drops to ~1% instantlyThree real mitigations, with what each one costs
Near cache the hot key. Every instance keeps a copy in memory with a very short TTL — one to five seconds. Five hundred instances each refreshing once per second is 500 requests per second to the shared cache, replacing perhaps 500,000. It is a thousandfold reduction from a few lines of code, and it works because hot keys are, by definition, requested often enough that even a one-second TTL yields an extremely high local hit rate.
The cost is exactly the near-cache TTL of staleness, applied to the key that is most visible to the most users. For a feature flag or a trending list, one second of staleness is nothing. For a live inventory count on a flash sale, one second may be the difference between selling ten items and selling four hundred. Choose the TTL from the harm, not from the load relief.
Key splitting (replication of the key). Write the value under k#0 through k#N, and have readers pick one at random. The N copies hash to different shards, so read capacity multiplies by N. The costs are real: writes must fan out to N copies, and during that fan-out different readers see different values — a divergence window with no error signal. Splitting is a reasonable answer for read-mostly hot keys and a poor one for anything written frequently, since write amplification and divergence both scale with N.
Dedicated capacity. Route the hot key to its own node or its own replica set, so its load does not compete with the rest of the keyspace. It does not raise the ceiling for that key, but it stops one hot key from degrading everything colocated with it — which is Bulkheads: Buying Independence by Giving Up Utilisation applied to a keyspace. Worth doing when the hot key is known in advance, as in a scheduled flash sale.
1// Count-min sketch: approximate per-key rates in fixed memory. Cheap enough2// to run on every request, which is the point — you want detection before the3// node saturates, not after.4class HotKeyDetector {5 private readonly sketch = new CountMinSketch(4, 2048)6 constructor(private readonly promoteAboveRps: number) {}7 8 observe(key: string): boolean {9 const estimated = this.sketch.increment(key) // over-estimates, never under10 return estimated > this.promoteAboveRps11 }12}13 14async function get(key: string): Promise<Value> {15 const local = nearCache.get(key) // TTL 1-5s, only for hot keys16 if (local !== undefined) return local17 18 const value = await sharedCache.get(key)19 20 // Promote on detection. 500 instances refreshing once a second is 500 rps to21 // the shared tier, replacing however many hundreds of thousands were arriving.22 if (detector.observe(key)) nearCache.set(key, value, NEAR_TTL_MS)23 return value24}What none of this fixes, and the honest failure modes
Writes. Every technique here multiplies read capacity. A single key receiving a very high write rate — a global counter, a live view count, a shared rate-limit bucket — cannot be helped by replication, because every copy must receive every write. The answer for hot *writes* is structural: shard the counter into N sub-counters and sum on read (trading read cost and momentary inaccuracy for write throughput), batch updates and flush periodically, or move to a CRDT counter that merges without coordination. That is CRDTs: Deterministic Merge, Not Correct Merge and Coordination Avoidance: Restructuring the Problem Instead of Paying for It, not caching.
Read-your-writes. A near cache on a hot key breaks it in a very visible way. A user toggles a setting, their request is routed to a different instance, and they see the old value for up to the near-cache TTL. On a hot key this affects the maximum number of users, so it is the one most likely to be reported. Where it matters, bypass the near cache for the writing session for a window after the write.
Divergence under key splitting. With N copies, a write updates them sequentially and readers sampling different copies see different values during that window. There is no error, no lag metric, and no way for a reader to know. This is why splitting suits read-mostly data and is a bad fit for anything where two users comparing screens would notice.
The final honest note: the best fix is often to stop reading the key. A configuration value read from the cache on every request can be loaded once at startup and refreshed on a schedule. A per-request feature-flag lookup can be a periodic bulk fetch of the whole flag set. A large share of hot keys are not a distribution problem at all — they are an access-pattern problem, and moving the read out of the request path removes it entirely.
| Technique | Read capacity | Write capacity | Staleness introduced | Divergence risk |
|---|---|---|---|---|
| Near cache (short TTL)typical | Effectively unbounded | Unchanged | Near-cache TTL | Across instances, bounded by TTL |
| Key splitting k#0..k#Nassumption | × N | Worse (× N writes) | None if writes complete | Yes, during write fan-out |
| Read replicas for the keytypical | × replicas | Unchanged | Replication lag | Yes, bounded by lag |
| Dedicated node for the keyprotocol | Unchanged | Unchanged | None | None |
| Remove the read from the pathprotocol | N/A — no reads | N/A | Refresh interval | None |
Key points
- A key hashes to one shard, so cluster size does not raise the throughput ceiling for a single key — adding nodes is the intuitive move that cannot work.
- The signature is one node at 100% while the cluster average is low; every capacity dashboard reports healthy headroom.
- A near cache with a one-to-five-second TTL is the highest-leverage mitigation, and it trades exactly that much staleness on your most-read key.
- Key splitting multiplies read capacity by N and costs N× write amplification plus a divergence window with no error signal.
- None of this helps hot *writes*: those need sharded counters, batching, or CRDTs, which are coordination techniques rather than caching ones.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Clients track approximate per-key request rates with a count-min sketch, cheap enough to run on every request.
- • Keys exceeding a threshold are promoted into a small per-instance near cache with a short TTL.
- • Subsequent reads are served locally, collapsing fleet-wide load on that key to roughly one refresh per instance per TTL.
- • For read-mostly keys that cannot tolerate near-cache staleness, the value is written under N suffixed keys and readers sample one at random.
- • Known-hot keys are given dedicated capacity so their load does not degrade colocated keys.
- • Detection is too slow and the node saturates before promotion happens.
- • The near-cache TTL is too long for the data, and users act on stale values at the highest-visibility point in the system.
- • Key splitting writes complete out of order or partially, leaving copies divergent with no signal.
- • The hot key is hot for writes, and every read-side technique is irrelevant.
- • Promotion thresholds are static, so a key that becomes hot during an event is not promoted until after the event.
- • One node at 100% CPU with the cluster at 10%: latency for every key on that node degrades, including keys unrelated to the hot one, because they share the node.
- • Read-your-writes violation reported at scale: a user changes a setting and sees the old value roughly half the time. It is the hot key, so it affects the most users and generates the most reports.
- • Divergent split copies: two users comparing the same page see different counts. No error, no lag metric, and the difference resolves on its own within a second, which makes it maddening to reproduce.
- • Timeout cliff on the hot shard: p99 for all traffic to that node crosses the client timeout at once, so unrelated features fail while the cluster reports capacity to spare.
- • Near caching is coordination-free and gives no coherence, bounded by the TTL — the same trade as every local cache.
- • Key splitting needs the writer to update all copies, which is a small replication protocol you now own, including its partial-failure behaviour.
- • Detection can be entirely local (each client sketches its own traffic) or centralised (the cache reports its hot keys), and the local version is preferable because it keeps working when the shared tier is struggling.
- • Under near caching, the shared tier can be unavailable for up to the TTL without user-visible impact for that key — an incidental availability benefit.
- • Under key splitting, losing one copy reduces capacity proportionally and readers sampling it fall through to the origin.
- • The hot key’s own consistency is weakened in every mitigation except dedicated capacity, so the trade must be stated in terms of what a stale read of *that specific key* costs.
- • Detect: per-key request rates via sampling or a sketch, and per-node utilisation compared against the cluster average. The gap between them is the alert.
- • Contain: promote the key into near caches immediately; it is the fastest lever and needs no data migration.
- • Recover: for anticipated events like flash sales, pre-promote the known keys and pre-warm the near caches before traffic arrives.
- • Reconcile: after key splitting, verify copies agree once writes settle; divergence that persists indicates a failed fan-out that nothing else will report.
- • Verify: load-test a single key past one node’s capacity and confirm the mitigation engages before the node saturates rather than after.
- • Per-node utilisation versus cluster average — the ratio is the hot-key detector of last resort and should be a standing alert.
- • Top-N keys by request rate, sampled or sketched, which is the only proactive signal available.
- • Near-cache hit rate for promoted keys, which confirms the mitigation is doing what you think it is.
- • Age of served values for hot keys, so the staleness you accepted is measured rather than assumed.
- • Workloads with a strong popularity skew — social graphs, catalogues with a bestseller, anything with a trending list.
- • Global configuration and feature flags read on every request, which are hot keys by construction and almost always tolerate seconds of staleness.
- • Scheduled high-traffic events where the hot keys are known in advance and can be pre-promoted.
- • Uniform access distributions, where the detection machinery finds nothing and the near cache only adds staleness.
- • Strongly consistent reads, where any near-cache staleness is unacceptable and only dedicated capacity remains as an option.
- • Write-hot keys, where every read-side technique is beside the point and the effort is misdirected.
- • Remove the read from the request path: load configuration at startup and refresh periodically. Frequently eliminates the hot key entirely.
- • Push the value to clients (long-poll, SSE, a config stream) so they hold it rather than fetching it, converting reads into pushes.
- • Serve the hot key from a CDN or edge tier with a short TTL, moving the load off your infrastructure altogether.
- • Accept degradation for that key alone: rate-limit reads of it and serve a default, protecting the rest of the keyspace.
Hot key: one key lives on one shard
| Mitigation | Load on the hot shard | What it costs | When it is wrong |
|---|---|---|---|
| nothing | 500K/s | — | always, once the key exceeds one shard |
| near cache the key | 500/s | 1 s of staleness on your most-viewed value | a live inventory count on a flash sale — one second may be four hundred oversells |
| split into 4 copies | 500K/s | writes fan out 4×; readers diverge during the fan-out with no error signal | anything written frequently — amplification and divergence both scale with the copy count |
| dedicated capacity | 500K/s | reserved hardware, and the ceiling for the key is unchanged | when the key alone exceeds a node — it isolates, it does not scale |
What people believe, and what is true
Add more cache nodes to handle the hot key.
The key hashes to one node. A hundred nodes serve it exactly as well as one, and the added capacity sits idle while that node saturates.
Our cluster is at 18% CPU, so we have headroom.
Cluster averages hide hot keys completely. Compare per-node utilisation against the average; a single node at 99% next to a 10% average is the whole story.
Key splitting is a general fix for hot keys.
It multiplies read capacity and makes writes worse by the same factor, and it introduces a divergence window with no error signal. It suits read-mostly keys only.
Near caching a hot key is safe because the TTL is only one second.
One second on the most-read key in the system is the largest staleness surface you have. For a flash-sale inventory count that second is the incident.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
One key lives on one machine, so adding machines does not help. Either keep a copy near every reader, make several copies of the key under different names, or stop reading it so often.
Practical
Alert on per-node utilisation versus cluster average, sketch per-key rates on the client, and promote hot keys into a near cache with a one-to-five-second TTL chosen from what staleness costs. Pre-promote known keys before scheduled events, and check whether the read can be removed from the request path entirely.
Advanced
Separate the read-hot case from the write-hot case, because they have disjoint solutions. Read-hot is solved by replication in any form — near caches, split keys, replicas — since copies are cheap when nothing changes. Write-hot cannot be solved by copies at all: every copy must take every write, so the answer is to change the data structure (sharded counters, batched flushes, CRDTs) and give up on a single authoritative value per key. Recognising which one you have is the whole diagnosis.
Apply it
- 💬 One Redis node is at 99% CPU and the other nine are at 9%. What is happening and why will adding nodes not help?
- 💬 What does key splitting cost you, and when is it the wrong choice?
- 💬 A single key is taking a very high *write* rate. Why do none of the read-side mitigations apply?