Partitioning & Sharding

Sharding is not one decision. Storage, write throughput, working-set memory and recovery time are four separate ceilings, and which one you hit determines the partition key. Picking the key before naming the ceiling is how teams end up with a split that solves nothing.

Q · One machine is no longer enough. What does splitting the data actually buy me, and what does it silently take away?

hash(key) % N is the obvious way to spread keys across N nodes, and it works beautifully until N changes. Then roughly 1 − 1/max(N, N′) of all keys move — about 80% when going from four nodes to five. That single fact is the entire reason the next two lessons exist.

Q · If I route with `hash(key) % N`, what exactly happens when N changes?

Keep keys in sorted order and a range query touches only the partitions covering that range. The price is that load now follows the shape of your data and the shape of your traffic — and the single most common key in software, a timestamp, sends every write to exactly one partition.

Q · I need ordered scans across a partitioned dataset. What does keeping keys in order cost me?

Place nodes and keys on the same circular hash space and let each key belong to the next node clockwise. Adding or removing a node then disturbs only its neighbours — about K/N keys move instead of nearly all of them. The distributed-systems question the pattern write-ups skip: what happens to the requests already in flight while ownership changes hands.

Q · When a node joins or leaves, how much data actually moves — and what happens to requests issued during the change?

A ring with one position per node distributes badly — with ten nodes, the largest share is typically three times the smallest. Giving each machine many logical positions turns a very lumpy random partition into a nearly even one, spreads recovery load across the whole cluster, and lets a bigger machine simply take more positions. It is not free: the metadata, the streaming and the failure probabilities all change.

Q · Why does a consistent hash ring need many positions per machine instead of one?

A hash spreads keys. It does not spread requests. When one key — a celebrity account, a viral post, a global counter, a status row every worker polls — takes a large share of the traffic, it lands on exactly one partition no matter how good the hash is. A single key is the atomic unit of partitioning, and you cannot split below it without changing the data model.

Q · One key takes a huge share of my traffic. Why does hashing not help, and what actually does?

Moving partitions between nodes is not a background chore. It is a sustained, self-inflicted load spike: terabytes across the network, doubled disk I/O at both ends, and a destination whose caches are empty for every key that arrives. Rebalancing has to run while the system keeps serving — and it competes with that serving for exactly the same resources.

Q · Data has to move between nodes while the system stays up. What does that actually cost, and what breaks at the moment ownership changes?

Joins, transactions, aggregations, secondary indexes and unique constraints were all cheap when the data sat on one machine, because one storage engine could see all of it. Across partitions each becomes a distributed protocol with its own latency, its own failure modes and its own consistency story. The lever that matters most is chosen long before any of them: the partition key.

Q · Which operations get harder once the data is no longer co-located, and what does each one cost now?