The question this answers
One key takes a huge share of my traffic. Why does hashing not help, and what actually does?
Partitioning bounds the *data* held by a node. It bounds nothing about the *requests* to a key: every operation on key k is served by the single partition that owns k. No hash function, token count or rebalancing policy changes that, because it is a property of the mapping being a function.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The overloaded node knows its own queue depth and request rate. It does not know whether it is hot because it owns many warm keys or one scorching one, unless it keeps per-key counters — which most storage engines do not by default. That gap is why hot-key incidents are typically diagnosed hours after they start.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Why the hash is powerless here
Partitioning is a function from key to partition. A function maps each input to exactly one output. So all traffic for one key goes to one partition — that is not an implementation weakness, it is the definition of what partitioning is. The only way to spread one key across partitions is to stop it being one key.
This is worth stating bluntly because the expectation is so widespread. Teams add nodes, raise vnode counts, switch hash functions, and enable auto-rebalancing, and the hot partition stays hot, because none of those change the mapping of a single key.
The shapes it takes in practice:
- Celebrity: one account with a hundred million followers, in a system sized for the median account with two hundred.
- Viral item: a post, product or video that goes from ordinary to 10,000× ordinary in minutes. Unlike the celebrity case, it is unpredictable and short-lived.
- Global singleton: a counter, a sequence, a feature-flag row, a rate-limit bucket keyed on something coarse. Hot by construction, always, from the first day.
- Coordination row:
SELECT ... WHERE status = 'pending' FOR UPDATErun by every worker — a queue implemented as a hot row, which is also a lock convoy. - Fat tenant: in a multi-tenant system, one customer who is a hundred times the size of the next. Not one key, but one partition-key *value*, which amounts to the same thing.
- partition 1 — 250k keys · 1.2k req/s · 11% CPU
- partition 2 — 249k keys · 1.1k req/s · 10% CPU
- partition 3 — 251k keys · 84k req/s · 99% CPU — 78k of them for one key
- partition 4 — 250k keys · 1.3k req/s · 12% CPU
- p3believes “I am overloaded; the rebalancer should move some of my keys away”✕ and it is false
- p1believes “the cluster has spare capacity”✓ and it is true
Every node above is acting on what it believes. Nothing in the cluster tells the mistaken one that it is mistaken.
The mitigations, and what each one actually costs
There are five real options and a sixth that is really a redesign. None is free, and the cost is the part usually left out.
| Mitigation | What it fixes | What it costs | When it is the right answer |
|---|---|---|---|
| Cache in fronttypical | Read load on the hot key, almost entirely | Staleness bounded by TTL; a stampede when the entry expires; a cache tier to operate; nothing at all for writes | Read-dominated hot keys where seconds of staleness are acceptable — the default first move |
| Key splitting (shard suffix)typical | Write and read load, by turning one key into `k#0 … k#S` | Every read becomes a scatter-gather over S shards and a merge; deletes and uniqueness break; the value must be decomposable (a sum, a set, a log — not an arbitrary blob) | Counters, append-only collections, and anything whose value is an aggregate |
| Read replicas of the hot partitiontypical | Read throughput, linearly in the number of replicas | Replication lag, so reads are stale and monotonicity can break; extra copies of the whole partition, not just the hot key; writes are unhelped | Read-heavy hot keys where the value changes rarely and staleness is tolerable |
| Request coalescingtypical | Duplicate concurrent reads for the same key, collapsing N in-flight requests to one | Added latency for requests that arrive just after a batch starts; the coalescing tier is itself a per-key bottleneck; helps only identical concurrent reads | Very high concurrency on a small number of keys, especially in front of a cache miss |
| Dedicated placementtypical | Noisy-neighbour damage — the hot key stops harming the other keys on its node | Does not reduce the hot key’s own load at all; needs explicit placement, which a ring cannot express; manual or semi-automatic operations | A known, persistent fat tenant that must be isolated rather than accelerated |
| Redesign the access patterntypical | The problem, rather than its symptoms | A second code path, a migration, and permanent complexity in the data model | Structural hot keys that will not go away: fan-out-on-write vs fan-out-on-read for celebrity followers is the canonical case |
Key splitting in detail, because it is the one people get wrong
Splitting means writing to k#r for a random r in [0, S) and reading all S shards. It is the only mitigation that helps *writes*, which makes it the go-to for counters and append-heavy collections. Three properties determine whether it will work:
The value must be mergeable. A sum splits (add the parts). A set splits (union). An append-only list splits (concatenate, with an ordering key). A single mutable blob does not split at all — you would have S conflicting versions and no merge rule. This is the same requirement as CRDTs: Deterministic Merge, Not Correct Merge, and for the same reason.
Reads get S times more expensive, and their latency becomes the max over S requests rather than one — the Fan Out to 100 and the Component’s Tail Becomes the System’s Median effect in miniature. A counter split 100 ways to survive writes now costs 100 reads to display. The usual resolution is asymmetric: split for writes, and maintain a periodically-rolled-up total for reads, accepting that the displayed number lags.
Uniqueness and deletion break. "Does this key exist" becomes S existence checks. Deleting means deleting S records, non-atomically. Any invariant that was per-key is now per-key-group and needs Cross-Partition Operations: Paying for What the Split Took Away machinery to hold.
The advanced version is *adaptive* splitting: keep per-key counters, and split only the keys that are actually hot, with S proportional to the observed rate. This gets you the write throughput without paying the read fan-out on the 99.99% of keys that are cold — at the cost of the read path needing to know which keys are split, which is a piece of metadata that must be consistent with the writers.
1const SHARDS = 642 3// Write: pick a shard at random. Contention drops by ~SHARDS.4async function increment(key: string, by: number) {5 const shard = Math.floor(Math.random() * SHARDS)6 await store.add(`${key}#${shard}`, by)7}8 9// Exact read: fan out. Correct, and 64x the cost.10async function readExact(key: string): Promise<number> {11 const parts = await Promise.all(12 Array.from({ length: SHARDS }, (_, i) => store.get(`${key}#${i}`)),13 )14 return parts.reduce((a, b) => a + (b ?? 0), 0)15}16 17// What production usually does instead: a background roll-up writes the18// total every second, and readers read one key. The number is up to a19// second stale — which is the price of not fanning out on every read.The failure mode that makes it worse: retries
A hot partition starts to throttle or time out. Its clients retry. The retries go to the same partition, because the key still hashes there. The only node in the cluster that cannot absorb more load is the only node receiving the extra load, and each retry consumes capacity that would otherwise have served an original request.
This is One Retry per Tier Is Not One Retry — It Multiplies in its purest form, and it converts a degradation into an outage on a timescale of seconds. The defences are the ones from the overload module and they belong in the client, not the server: a retry budget rather than a fixed retry count, backoff with jitter (Without Jitter, Every Client That Failed Together Retries Together), and a circuit that opens per-key rather than per-host so that a hot key does not trip the whole dependency.
The server-side complement is per-key Decide at the Door Whether the Capacity Exists: shed requests for the hot key specifically, so that the other 250,000 keys on that node keep working. Without it, one key’s traffic takes down every key that happens to share its partition — a blast radius set by the hash function, which is to say, set arbitrarily.
Key points
- Partitioning is a function, so all traffic for one key goes to one partition. Hashing cannot spread a single key.
- A single key is the atomic unit of partitioning; going below it means changing the data model.
- Cache fixes reads, splitting fixes writes but requires a mergeable value and costs a read fan-out, replicas fix reads at the price of staleness.
- Isolating the hot key protects its neighbours without helping the key itself — often the more valuable thing.
- Retries concentrate on precisely the node that cannot take them, turning degradation into outage.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Traffic for key k arrives and is routed, by the partition function, to the single partition owning k.
- • That partition’s node saturates on CPU, queue depth or a provisioned throughput limit.
- • Every key co-located on that partition degrades with it, regardless of its own traffic.
- • Clients observe timeouts or throttling and retry, adding load to the saturated node.
- • The rebalancer, if enabled, may move keys off the node — which does not help, because the load is not in the keys it can move.
- • The hot key is not identifiable because the storage layer keeps no per-key counters.
- • Splitting is attempted on a value with no merge rule, producing S divergent versions.
- • The cache in front of the hot key expires and every request misses simultaneously (One Key Expires and Five Hundred Instances Miss at the Same Millisecond).
- • A read replica lags far enough that the displayed value moves backwards for a user (Monotonic Reads: Never Let Time Run Backwards).
- • The roll-up job that maintains the aggregate for split counters falls behind or dies, and the displayed number silently freezes.
- • Throttling at low aggregate utilisation: the table is throttling requests while the cluster reports 20% capacity used. The operator sees a "provisioned throughput exceeded" error rate on a near-idle system, which is the single most confusing signature in this module.
- • Noisy neighbour: every user whose data happens to share a partition with the hot key sees p99 latency 20× worse than everyone else, with no pattern in the application layer and no error on any of their requests.
- • Retry collapse: throttling begins, client retries double the load within seconds, and the node goes from degraded to unresponsive. The operator sees error rate and request rate rising together — the signature that distinguishes amplification from a traffic spike.
- • Frozen counter: the split-counter roll-up job fails and the displayed total stops advancing while writes continue. No error is raised; the number is simply wrong and stays wrong.
- • Backwards-moving values: reads served from a lagging replica of the hot partition return an older value than a previous read on the same session. The operator sees user reports of numbers going down, and nothing in the logs.
- • The hot key itself is often a coordination point in disguise: a counter, a lock row, a sequence. When it is, the fix is usually Coordination Avoidance: Restructuring the Problem Instead of Paying for It rather than more capacity.
- • Splitting deliberately removes coordination — S independent shards that never need to agree — and pays for it in read cost and lost invariants. That trade is the same one made by CRDTs: Deterministic Merge, Not Correct Merge, and recognising it as such tells you exactly which values can be split.
- • Adaptive splitting reintroduces coordination in the metadata: readers and writers must agree on the current shard count for a key, or a read will miss shards a writer is using.
- • Per-key admission control needs no coordination if each node decides independently, which is why it is a practical defence and cluster-wide rate limiting usually is not.
- • The hot key’s partition is the one most likely to fail, and its failure takes every co-located key with it.
- • Under a cache outage, the hot key’s full load returns to the storage layer instantly — so the cache is load-bearing for availability, not just latency, and should be treated as such.
- • Split shards degrade gracefully: losing one shard of a split counter loses 1/S of the count rather than the whole key, which is a genuine availability improvement alongside the throughput one.
- • Detect: per-key request counters, sampled or sketch-based. Without them, hot-key incidents are diagnosed by narrative rather than data.
- • Contain: shed or throttle the hot key specifically, protecting its neighbours. Isolation first, capacity second.
- • Recover: put a cache in front for reads; for writes, split, accepting the read cost as a temporary state.
- • Reconcile: after a split, verify the aggregate matches the sum of shards before trusting the rolled-up value.
- • Verify: confirm the previously co-located keys have returned to normal latency — the hot key getting better is not the same as the partition getting better.
- • Per-key request rate, top-N by count. A heavy-hitters sketch costs little and is the single highest-value metric in this lesson.
- • Per-partition request rate plotted against per-partition key count. Divergence between the two is the definition of a hot partition.
- • Throttle or rejection rate alongside aggregate utilisation. The combination "throttling while idle" is diagnostic and nothing else produces it.
- • Ratio of retries to first attempts, per key. Rising ratio on one key means amplification is already underway.
- • Cache hit rate for the hot key specifically, and time-to-expiry — the stampede predictor.
- • Mitigations are worth their cost when the hot key is structural and persistent: a celebrity account, a global counter, a fat tenant.
- • Caching is nearly always worth it for a read-hot key, because it is the cheapest option by an order of magnitude.
- • Splitting is worth it when the value is naturally an aggregate — it is close to free for counters and sets.
- • Isolation is worth it whenever the collateral damage to co-located keys exceeds the damage to the hot key itself, which is most of the time.
- • Splitting a value that is not mergeable — you have created a conflict-resolution problem in exchange for throughput.
- • Splitting every key preemptively: you pay S× read cost across the whole dataset to protect against a handful of keys.
- • Caching a key whose staleness has real consequences — a balance, a permission, a rate-limit counter.
- • Adding read replicas for a *write*-hot key, which adds cost and lag and fixes nothing.
- • Make the key less coarse. A rate-limit bucket keyed on
tenantis hot; keyed on(tenant, minute, bucket)it is not. Often the entire fix. - • Move the hot value out of the partitioned store entirely — into a dedicated in-memory service sized for it, where one key at high throughput is the normal case rather than a pathology.
- • Approximate instead of counting exactly: probabilistic counters and sketches remove the contention by removing the requirement for an exact number.
- • Fan out on write rather than on read (or the reverse) so the expensive side falls on the less-hot path — the classic celebrity-timeline resolution, and a hybrid of the two is what large social systems actually run.
- • Accept and shed: serve the hot key at a fixed capacity and reject the excess, protecting everything else. Sometimes the correct engineering answer, and rarely the popular one.
Hot partitions: the skew hashing cannot fix
What people believe, and what is true
A better hash function will spread the hot key.
The hash maps one key to one partition. Changing the hash changes *which* partition is hot, not whether one is.
Adding nodes fixes a hot partition.
It adds capacity everywhere except where the load is. The hot partition still lives on one node, now with more idle neighbours.
Auto-rebalancing will move the load away.
Rebalancing moves keys. If the load is in a key it cannot split, moving that key just makes a different node hot.
Throttling means we need more provisioned capacity.
When one key is throttling at 20% aggregate utilisation, raising the limit buys headroom for a single node and wastes it everywhere else. The fix is in the key, not the quota.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Hashing spreads keys, not requests. One hot key lands on one partition, and no amount of hashing, rebalancing or node-adding changes that.
Practical
Get per-key counters first — you cannot fix what you cannot see. Then: cache for read heat, split for write heat if the value is an aggregate, and isolate the key so its neighbours stop paying for it. Add a per-key retry budget on the client before any of it.
Advanced
Split writes and roll up reads. Writing to k#random and maintaining a periodic total gives you S× write throughput at O(1) read cost, in exchange for a bounded staleness on the displayed value. The general principle: when a key is hot, buy throughput with staleness, because the alternatives cost correctness.
Internals
Adaptive splitting requires the read path to know each key’s current shard count, and that metadata must be at least as fresh at readers as at writers, or reads miss shards. Publish shard-count *increases* to readers before writers begin using the new shards, and never decrease it in place — retire a split by draining shards into shard 0 and only then lowering the count.
Apply it
- 🔧 Add a heavy-hitters sketch to a service and find your top ten keys by request rate. Most teams are surprised by at least one of them.
- 🔧 Take a counter in your system, split it 64 ways, and write the roll-up job. Then write the check that proves the roll-up equals the sum of shards.
- ⚡ A product launch makes one item’s inventory row the target of 30,000 decrements per second. Inventory must not go negative. What do you do, and what does each option cost in correctness?
- ⚡ A multi-tenant analytics system has one customer generating 40% of all writes. Their queries are slow and so is everybody else’s. Rank your options.
- 💬 A key is taking 60% of your traffic. Explain why re-sharding will not help.
- 💬 Design a like-counter for a post that may receive 50,000 likes per second. State what you gave up.
- 💬 Your table is throttling but the cluster is at 20% utilisation. Walk me through the diagnosis.