Rebalancing: A Load Spike You Schedule for Yourself

Ask an engineer what a rebalance costs and you get "network". That is the smallest of the four, and the easiest to throttle.

Network. Moving a 2 TB node’s worth of data is 2 TB across the fabric, plus checksums and protocol overhead. At 1 Gb/s effective, that is about five hours; at 10 Gb/s, thirty minutes. Real and bounded, and the only cost with an obvious knob.

Disk I/O at both ends, and more than you moved. The source reads the data — usually a large sequential read that evicts its own page cache. The destination writes it, and in an LSM engine that write is then compacted, often several times, so writing 2 TB costs 4–10 TB of physical writes. Write amplification means the destination does more I/O than the mover accounts for, which is why a rebalance can saturate a destination whose network is at 20%.

Cache coldness at the destination. This is the one nobody budgets for. A partition arrives with an empty page cache, an empty block cache and an empty in-memory index. Every read for a moved key now goes to disk. The moved keys’ p99 jumps by an order of magnitude and stays there until the working set is re-warmed — minutes to hours, depending on the access distribution. **Latency gets worse *after* the move completes, not during it**, which is what makes it so consistently surprising.

Ongoing serving, degraded. Every one of the above competes with live traffic for the same resources. A cluster at 60% utilisation running a rebalance is a cluster at 90%+ utilisation, and queueing is nonlinear near saturation (Coordination Couples Availability is not the issue here; simple utilisation is).

Copying is safe: it is a read on one side and a write on the other, and if it fails you retry it. The hazard is the instant ownership transfers, because for a moment the answer to "who owns this key" must change atomically for everyone, and it cannot.

The standard safe shape has four phases, and each exists to close a specific hole:

1. Copy while the source keeps serving. The destination pulls a snapshot. The source is authoritative throughout; the destination answers nothing.

2. Catch up on the delta. Writes accepted by the source during the copy are streamed to the destination — from a change log, or by a second pass. Repeat until the delta is small enough to close in one short step.

3. Freeze, drain, flip. The source stops accepting writes for that partition, drains what is in flight, ships the final delta, and only then does the ownership record change. The unavailability window is the length of this step, and it is why the delta must be small before you enter it.

4. Redirect stragglers. Routers with stale maps still arrive at the source. It must answer NotOwner, never serve and never 404 — the same discipline as The Ring: Keeping the Mapping Stable When Membership Changes and Range Partitioning: Scans You Keep, Hotspots You Inherit. Optionally the source forwards to the destination for a grace period.

The single most common bug: skipping step 3’s freeze and hoping the delta stream catches everything. It does not, because there is always a write in flight when you decided the delta was empty, and that write is accepted by a node that is about to stop being the owner. It is then lost with no error anywhere.

Here is the coupling that produces the worst incidents in this module, and it is worth stating as a rule.

A failure detector is imperfect: it will occasionally declare a live-but-slow node dead (Crashed or Just Slow: The Distinction You Cannot Make, No Heartbeat Does Not Mean Dead). An automatic rebalancer reacts to membership changes by moving data. Compose the two and you get: a node that is slow because the cluster is loaded gets declared dead; the rebalancer starts moving its data; the move adds load; more nodes become slow; more nodes are declared dead; more data moves. The cluster converts an overload into a stampede, and the harder it works the worse it gets. This is Cascading Failure: When the Response to Failure Causes More Failure with the rebalancer as the amplifier.

Three defences, in order of how much they help:

Do not rebalance on suspected failure — only on confirmed departure. Distinguish "we have not heard from it in 30 seconds" from "it has been gone for 15 minutes" from "an operator decommissioned it". Only the last two should move data. Cassandra and Elasticsearch both learned this and both expose a delay before reallocation begins.

Cap concurrency cluster-wide, not per node. Each node throttling itself to a modest rate still adds up to a saturated fabric when fifty of them do it at once. The cap that matters is the number of concurrent moves in the whole cluster.

Keep a human in the loop for large moves. Automatic recovery from a single node loss is worth having. Automatic rebalancing of a whole cluster because the failure detector had a bad minute is not. The asymmetry — small automatic, large manual — is a good default.

And a fourth that is really a design choice: move whole partitions, never individual keys. A partition-granular move is discrete, interruptible, resumable and countable. A key-granular reshuffle has no natural checkpoint and cannot be paused halfway.

The operational goal is that a rebalance is uninteresting. That is achievable, and the techniques are unglamorous.

Move small units. Many small partitions mean each cutover window is short and each failed transfer is cheap to retry. This is a strong argument for a higher partition count independent of balance.

Warm before you flip. If the destination has the data but not the ownership, it can serve shadow reads to populate its caches. Then the cutover does not cause a cold-cache cliff. Few systems do this automatically and it is the highest-value custom work available here.

Throttle by observed impact, not by a fixed rate. A fixed 100 MB/s is too much at peak and too little at 3am. Systems that adapt the move rate to observed p99 on the affected nodes finish faster *and* hurt less.

Rehearse the abort. Every rebalance should be pausable and reversible before cutover. The plan that has no abort path is the plan that gets completed during an incident because nobody knew how to stop it.

Prefer cheap sources. Stream from a follower rather than the leader where the engine allows it, so the move does not evict the cache of the node serving reads.