The question this answers
Why does a consistent hash ring need many positions per machine instead of one?
With V positions per physical node, a node’s share of the key space concentrates around 1/N with a relative spread that shrinks roughly as 1/√V. This is balance of *key space*, in expectation, under a uniform hash — not balance of bytes, and emphatically not balance of traffic.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A node knows its own token positions and the tokens it has been told belong to others. It cannot verify that the resulting distribution is even without gathering the whole token list; balance is a global property computed from local fragments, which is why systems ship a "describe ring" command and why operators are surprised by skew.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
One position per node distributes badly
The intuition that N random points cut a circle into N roughly equal arcs is simply wrong, and the error is large. For N uniform random points, the expected largest gap is about (ln N + γ)/N — for ten nodes that is roughly 29% of the circle, against a fair share of 10%. Meanwhile some node will hold under 1%.
So a plain ring at N = 10 typically has one machine holding three times its share of the data and taking three times its share of the traffic, purely from the randomness of where hashes landed. You cannot fix it by choosing a better hash: the unevenness is a property of random points on a circle, not of the hash function.
And it is not self-correcting. The positions are fixed by the node identifiers, so the skew is *permanent* until membership changes. An operator who adds a node hoping to relieve the hot one has roughly a 1-in-N chance of placing it where it helps.
node share of ring vs fair share (10.0%) n-03 28.4% 2.8x n-07 17.1% 1.7x n-01 13.9% 1.4x n-09 10.6% 1.1x n-05 9.2% 0.9x n-02 7.4% 0.7x n-10 5.5% 0.6x n-06 3.8% 0.4x n-04 2.9% 0.3x n-08 1.2% 0.1x max/min ratio: 24x. The hash is perfect; the geometry is not.
V positions per machine, and the 1/√V rule
Give each machine V positions on the ring instead of one. Its share is now the sum of V independent arcs rather than a single one, and sums of independent random variables concentrate. The relative spread of a node’s share falls roughly as 1/√V: at V = 1 the spread is on the order of 100%, at V = 100 about 10%, at V = 256 about 6%.
That is the whole idea, and it buys three distinct things:
Balance. Every machine holds close to 1/N of the key space, so disk and baseline request load are even without any manual placement.
Spread recovery. This is the underrated benefit. On a plain ring, a dead node’s entire arc is inherited by one successor, which must both absorb double traffic and stream in a full node of data from one source. With V positions, the dead node’s V arcs have V different successors — so every surviving node contributes a small slice of the rebuild, and the rebuild runs N-way parallel instead of one-to-one. Recovery time drops by roughly a factor of N, and no single node is overloaded during it. For large datasets this dominates the balance argument.
Heterogeneous capacity. Machines are not identical: you buy a bigger instance type, or you run a mixed fleet during a migration. Give the 2× machine 2× the tokens and it takes 2× the data and traffic. There is no other simple handle for this on a ring; without it, the cluster runs at the pace of its smallest member.
| V (positions per node) | Typical spread of node share | Ring entries to hold and gossip | Rebuild sources on node loss |
|---|---|---|---|
| 1assumption | ≈ ±100%, max/min often > 10× | 100 | 1 — the successor does everything |
| 16assumption | ≈ ±25% | 1,600 | up to 16 |
| 64assumption | ≈ ±12% | 6,400 | up to 64 |
| 256assumption | ≈ ±6% | 25,600 | up to 99 — effectively the whole cluster |
The costs, including one that is genuinely surprising
Metadata grows as N·V. The ring is now tens of thousands of entries. Every node holds it, every gossip round carries changes to it, and every new node must learn it. Cassandra clusters running 256 tokens per node at several hundred nodes hit exactly this: the gossip payload and the per-node ring bookkeeping become a real cost, and startup and schema propagation slow down measurably.
Every operation fragments. A repair, a rebuild, a backup or a range scan that was one contiguous stream per peer becomes V small streams per peer. Small streams are less efficient — more seeks, more round trips, more per-stream overhead — and the accumulated fixed cost can exceed the benefit. This is why Cassandra’s recommended default moved *down* from 256 tokens to 16, paired with a smarter token-allocation algorithm that gets good balance without brute-force randomness.
And the surprising one: high V makes multi-node failures more likely to lose a quorum somewhere. With replication factor 3, each token range has a replica set of three machines. With V = 1 there are only N distinct replica sets. With V = 256 there are potentially thousands of *different* three-machine combinations. Now ask: if two random machines fail simultaneously, is there at least one range whose replica set they both belong to? With few distinct replica sets, usually no. With thousands, almost certainly yes — so some sliver of data drops below quorum even though only 2% of the cluster is down.
That is the copyset problem, and it is a genuine inversion of intuition: **spreading replicas maximally makes small failures more likely to cause *some* data loss, while making large failures less likely to cause *total* loss.** Systems that care about it constrain replica placement into a bounded number of copysets rather than letting every token pick freely — see Fault Domains: What Fails Together and Correlated Failure: The Independence Assumption Is Usually False.
- Metadata: O(N·V) ring entries, gossiped and held on every node.
- Streaming: V small transfers per peer instead of one large one, with worse throughput per byte.
- Repair: tree construction and comparison happen per range, so Merkle Trees: Finding the Difference Without Reading the Data work multiplies by V.
- Availability: more distinct replica sets means a higher probability that any given pair of failures affects some range.
- Debuggability: "which node holds key k" stops being answerable by inspection.
Choosing V, and the alternative to choosing it
The naive answer — "make V large" — was the industry default for a decade and has been walking back ever since. The current shape of good advice: V should be large enough that random placement is acceptably even, and no larger. For clusters of tens of nodes, V in the range 8–32 with a token-allocation algorithm that considers existing tokens beats V = 256 with pure randomness on every axis.
The deeper point is that virtual nodes are a *statistical* fix for a problem that also has a *deterministic* fix. If you are willing to keep an explicit partition → node assignment map — the fixed-partition scheme from Hash Partitioning and the Modulo Trap — you can compute a balanced assignment directly, place replicas in deliberate copysets, drain a node gradually, pin a tenant to particular hardware, and weight nodes exactly rather than approximately. You pay for it with a small piece of agreed state.
That trade is the honest summary of this lesson: virtual nodes buy balance without a map; an assignment map buys balance, control and predictability with one. Systems built around an existing coordination service tend to choose the map; systems that want no coordination in any path tend to choose vnodes.
Key points
- N random points do not divide a circle evenly — at N = 10 the largest arc is typically ~3× the fair share.
- V positions per machine shrink the relative spread roughly as 1/√V; V = 100 gets you to about ±10%.
- The rebuild argument matters as much as the balance argument: V positions means V different peers contribute to recovery, in parallel.
- Weighted tokens are the only simple way to run a heterogeneous fleet on a ring.
- High V costs O(N·V) metadata, fragments every streaming operation, and increases the chance that a small correlated failure takes some range below quorum.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • For each physical node, derive V token positions —
hash(nodeId + ":" + i)for i in 0..V, or assign them from an allocation algorithm. - • Insert all N·V positions into the sorted ring; each position owns the arc back to its predecessor.
- • Route a key to the node owning the first position clockwise, exactly as before.
- • When walking clockwise for replicas, skip positions belonging to a physical node already chosen — and, if rack-aware, already-chosen fault domains.
- • Weight a node by giving it more positions in proportion to its capacity.
- • On node loss, each of its V arcs is inherited by the owner of the next position, so the rebuild is sourced from and destined for many peers at once.
- • Replica walk does not deduplicate physical nodes, so R "replicas" are fewer than R machines.
- • Token collisions — two nodes deriving the same position — leave an arc with an ambiguous owner.
- • Gossip cannot keep up with an N·V ring, so nodes hold divergent token lists for extended periods.
- • A node is decommissioned but its tokens are not fully reassigned, leaving arcs unowned.
- • Weighted tokens are assigned by capacity that later changes (an instance is resized) and the weights are never revisited.
- • Permanent unexplained skew at low V: one node at 2.5× the disk usage of another, unchanged by restarts or rebalances. The operator sees a stable imbalance with no application-level cause, because the cause is where the tokens landed.
- • Replication factor silently below target: the replica walk returned three token positions belonging to two machines. The operator discovers it only when one machine dies and a range is unreadable — the cluster reported RF=3 throughout.
- • Rebuild that never finishes at high V: thousands of small streams, each with fixed overhead, so a node replacement that should take two hours takes two days. The operator sees streaming sessions in the thousands and per-stream throughput in the kilobytes.
- • Gossip saturation: ring metadata grows with N·V until gossip traffic and CPU on every node scale with cluster size. The operator sees node startup time and schema-propagation time climbing as the cluster grows, with no change in data volume.
- • Two-node failure takes a range below quorum in a large cluster: 2% of machines are down and a small set of keys is unavailable. The operator sees a tiny, surgical availability hit that the naive "we can lose R−1 nodes" model says should not exist.
- • Token assignment is a one-time decision per node, and randomly derived tokens need no coordination at all — that is the appeal.
- • Allocation algorithms that pick tokens to *improve* balance must see the existing ring, so they need the membership view to be current at join time. A node joining with a stale view picks bad tokens permanently.
- • Weighted assignment for heterogeneous fleets is a policy decision that must be recorded somewhere durable; deriving it from an instance type at boot means it silently changes on a replacement.
- • None of this is in the request path. Virtual nodes add zero per-request coordination, which is precisely why they are preferred over an assignment map in systems that avoid coordination services.
- • A node loss degrades many ranges by one replica rather than a few ranges catastrophically — better expected availability, worse worst-case count of affected ranges.
- • Rebuild proceeds from many peers concurrently, so recovery is fast, but the load is spread over the whole cluster rather than isolated to one pair.
- • Under partition, each side computes ownership from its own token list; if those lists have diverged, the two sides disagree about arcs in a way that is harder to reason about than a single-position ring.
- • Detect: compare each node’s computed key-space share against 1/N. A deviation beyond the 1/√V expectation means tokens, not data, are the problem.
- • Contain: on discovering a replica-set collapse, stop treating the affected ranges as replicated and re-place them before anything else.
- • Recover: re-token a badly placed node by decommissioning and rejoining it with allocated rather than random tokens — expensive, and the reason token allocation deserves attention at cluster creation.
- • Reconcile: after any token change, run repair over the affected ranges; token movement is the same hazard as any other ownership change (The Ring: Keeping the Mapping Stable When Membership Changes).
- • Verify: assert that every range resolves to R distinct physical hosts and, where applicable, R distinct racks.
- • Per-node computed ring share versus per-node measured bytes. A gap between them means the *data* is skewed even though the *key space* is not — a signal for Hot Partitions: The Skew Hashing Cannot Fix.
- • Ring size (N·V) and gossip message size over time. Both should be boring; when they are not, V is too high for the cluster size.
- • Streaming session count and per-session throughput during a rebuild. Thousands of tiny sessions is the high-V pathology, visible immediately.
- • Number of distinct replica sets in the cluster — the copyset count. It is the number that predicts whether a two-node failure will cost you a range.
- • Token count per node against declared capacity, for heterogeneous fleets. Drift here is silent and permanent.
- • Any ring-based system at all: a plain ring is not usable in production, so this is closer to a requirement than an option.
- • Mixed-capacity fleets, where weighted tokens are the only straightforward lever.
- • Large datasets per node, where parallel N-way rebuild is the difference between hours and days of running under-replicated.
- • Clusters that grow incrementally, since a joining node with V positions takes a small slice from many peers rather than a large slice from one.
- • Very large clusters with high V, where metadata and streaming fragmentation start to dominate.
- • Systems that repair by building Merkle Trees: Finding the Difference Without Reading the Data per range: the repair cost multiplies by V, which is often the real reason to lower it.
- • Availability-critical deployments where the copyset effect matters more than balance.
- • When you already run a coordination service and could simply keep a balanced assignment map, getting the same balance plus deliberate control.
- • Fixed partitions with an explicit assignment map: deterministic balance, deliberate placement, controllable draining, bounded copysets — at the cost of a small piece of agreed state.
- • Token allocation algorithms instead of random tokens: keep V low (8–32) and choose each new node’s positions to minimise the resulting imbalance. Best of both for most cluster sizes.
- • Rendezvous hashing with weights: no tokens at all, naturally balanced, weights are first-class, O(N) per lookup.
- • Maglev-style lookup tables, when lookups vastly outnumber membership changes and you want near-perfect balance with a bounded table.
- • Simply running homogeneous hardware, which removes the heterogeneity motivation entirely and is often the cheaper answer.
Virtual nodes: how many positions per machine is enough
What people believe, and what is true
Virtual nodes exist to make rebalancing smoother.
That is a real benefit but not the founding reason. The founding reason is that a plain ring distributes the key space very unevenly, and the unevenness is permanent.
More virtual nodes is strictly better balance, so use as many as possible.
Balance improves as 1/√V with diminishing returns, while metadata, streaming fragmentation and repair cost grow linearly in V, and the number of distinct replica sets grows with it too.
Virtual nodes balance load.
They balance *key space*. If one key takes half the traffic, every position of every node is irrelevant — see Hot Partitions: The Skew Hashing Cannot Fix.
With V positions and RF=3, three positions means three machines.
Only if the replica walk skips repeated physical nodes. Implementations that forget this report RF=3 while storing one copy.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Give each machine many positions on the ring instead of one, so its share of the key space is a sum of many small arcs rather than one lumpy arc. Balance follows.
Practical
Never run a ring with one token per node. Pick V for your cluster size — small with an allocation algorithm, larger with random tokens — and set tokens proportional to capacity on mixed fleets. Then verify that every range resolves to R distinct physical hosts.
Advanced
The rebuild argument usually outweighs the balance argument at scale: V positions means a dead node’s data streams in from V peers concurrently, cutting time-under-replicated by roughly a factor of N. That window is your real durability exposure, not the nominal replication factor.
Internals
Raising V multiplies the number of distinct replica sets, which raises the probability that any given pair of simultaneous failures shares a replica set for at least one range. Maximally spread replicas minimise the chance of losing *everything* and maximise the chance of losing *something*. Which you want depends on whether partial unavailability or total loss is the worse outcome for your data — and for most user-facing storage, it is the former, which argues for constrained copysets rather than free ring walks.
Apply it
- 🔧 Generate N=10 random ring positions and compute each arc. Repeat for V=1, 16 and 256 positions per node and plot the max/min share ratio.
- 🔧 Write the replica-walk function and a property test asserting it always returns R distinct physical hosts, then run it against a ring where one node holds 60% of the positions.
- ⚡ A 200-node cluster at V=256 takes 30 hours to replace a failed node. Streaming throughput per session is tiny. What is the diagnosis and what are the two fixes?
- ⚡ Two machines fail in a 500-node cluster with RF=3 and a small set of keys becomes unavailable. Explain to an operator why "we can survive two failures" was not wrong, but was not what they thought it meant.
- 💬 Why does a consistent hash ring with one token per node distribute data unevenly, and roughly how unevenly?
- 💬 You are told to raise vnodes from 16 to 256 to improve balance. What do you push back with?
- 💬 Your fleet is half
m5.2xlargeand halfm5.8xlarge. How do you keep the small machines from being the bottleneck?