The question this answers
One machine is no longer enough. What does splitting the data actually buy me, and what does it silently take away?
Partitioning preserves whatever guarantee the storage engine gave you, but only *within* a partition. Any property that spanned the whole dataset on one node — a cross-row transaction, a global secondary index, a uniqueness constraint, an ordered scan — holds only inside one partition afterwards, unless you pay separately to restore it.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A node knows the partitions it owns and the routing map it was last told about. It does not know the global dataset, it does not know whether its routing map is current, and it cannot answer any question about a key it does not own without a network call whose outcome is ambiguous.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Name the ceiling before you name the key
Four different pressures all get described as "we need to shard", and they do not want the same partition key. Getting this backwards is the most expensive mistake in the module, because the key is the hardest thing to change later.
Storage. The dataset does not fit on one disk. Any key that spreads bytes evenly works — the requirement is only that no single partition grows past a node.
Write throughput. One node cannot absorb the write rate. Now you need the key to spread *writes* evenly, which is a different distribution from bytes: a table where 1% of rows take 90% of the updates is balanced by size and hopeless by write load.
Working set. The hot data no longer fits in RAM, so every read hits disk. Here the goal is that each node’s *active* subset fits its page cache — which sometimes argues for the opposite key to the one that spreads bytes, because you want related hot data co-located, not scattered.
Recovery time. A 20 TB node takes hours to rebuild after a loss, and during those hours you are running under-replicated. Twenty 1 TB nodes rebuild in parallel. This ceiling is invisible on every capacity dashboard and is the one that actually forces the split at many companies.
- Storage ceiling → spread bytes. Almost any hash works.
- Write ceiling → spread writes. Requires knowing the write distribution, not the row count.
- Working-set ceiling → co-locate what is read together, then spread the groups.
- Recovery ceiling → smaller shards, more of them, chosen for rebuild parallelism.
- If you cannot say which ceiling you are hitting, you are not ready to choose a key.
| Ceiling | Symptom you actually see | What the key must spread | Key that would be wrong |
|---|---|---|---|
| Storagetypical | Disk at 85% and growing linearly | Bytes | Anything is fine — this is the easy case |
| Write throughputtypical | Commit latency rising, write queue depth growing | Write operations per second | A key that spreads rows evenly but concentrates updates |
| Working settypical | Cache hit rate falling, read IOPS climbing with flat traffic | Hot bytes, while keeping co-read data together | A pure hash that scatters one tenant across every node |
| Recovery timeassumption | Rebuild after node loss measured in hours | Rebuild work across peers | Few very large shards |
The boundary is where guarantees stop
On one node, BEGIN; UPDATE a; UPDATE b; COMMIT; is atomic because a single storage engine sees both writes. Put a and b on different partitions and nothing about that statement is true any more. The same is quietly true of a uniqueness constraint, an ordered scan, a COUNT(*), a foreign key, and a secondary index.
The important framing: partitioning does not weaken guarantees, it re-scopes them. Everything still holds inside a partition. So the design question is not "how do I get transactions back" but "can I choose the partition key so that the operations that need to be atomic all land in the same partition?" When you can, you keep single-node semantics at distributed scale for free. When you cannot, you have signed up for Cross-Partition Operations: Paying for What the Split Took Away and its costs.
This is why the partition key is usually an *entity* rather than a column: the tenant, the account, the conversation, the order. Those are the boundaries that transactions and invariants naturally respect. Choosing created_at or region because it looked evenly distributed is how you end up with every business operation spanning shards.
- Atomicity: survives inside a partition, needs Two-Phase Commit: Buying Atomicity With a Promise or Sagas: Trading Isolation for Availability across them.
- Uniqueness: survives inside a partition, needs Distributed Uniqueness: One Name, Many Shards across them.
- Ordered scan: survives under range partitioning, becomes scatter-gather under hashing.
- Secondary index: local index per partition is cheap to write and expensive to read; a global index is the reverse.
- Aggregate counts: become partial aggregates plus a merge, exact only for associative operations.
Partitioning and replication are different axes
These get conflated constantly, and the conflation causes real outages. Replication makes copies of the same data for availability and durability. Partitioning makes different data live on different nodes for capacity. Almost every real system does both, and the composition is what you actually operate: each partition has R replicas, each node holds many partitions, some as leader and some as follower.
The composition is where the interesting failures live. Losing one node does not lose one partition — it degrades the replication factor of *every partition that node held a replica of*, which under a naive assignment can be a large fraction of the whole keyspace. That is the argument for constrained replica placement (rack awareness, replica sets, copysets) rather than "pick R random nodes per partition": it trades a little balance for a much smaller probability that any given multi-node failure loses a quorum somewhere.
See Why Replicate: What a Second Copy Buys You for the availability side and Correlated Failure: The Independence Assumption Is Usually False for why "three random replicas" is worse than it looks.
- node-1 — leader p1 p5 · follower p2 p3 p4 p6
- node-2 — leader p2 p6 · follower p1 p3 p5
- node-3 — leader p3 · follower p1 p2 p4 p5 p6
- node-4 — held a replica of all six — every partition is now under-replicated
- n1believes “node-4 is gone; begin re-replicating the ranges it held”✓ and it is true
- n4believes “I am healthy and still a replica for all six partitions”✕ and it is false
Every node above is acting on what it believes. Nothing in the cluster tells the mistaken one that it is mistaken.
The routing question nobody asks early enough
Something has to turn a key into a node. There are exactly three places to put that logic, and the choice determines how a membership change propagates.
Client-side routing. Every client holds the partition map. Zero extra hops, lowest latency, and the worst change story: a map update must reach thousands of processes, and until it does they are routing on stale information. See Discovering Services: The Registry Is a Distributed System Too.
A routing tier. A proxy layer holds the map and forwards. One extra network hop on every request, but the map lives in a handful of processes you can update quickly, and clients stay dumb. This is what most managed systems do.
Routing-aware nodes. Any node accepts any request and forwards to the owner. No extra tier, but a mis-routed request costs a full extra round trip inside the cluster, and every node needs the map anyway.
Whichever you pick, the map itself is state that must be agreed on — which makes it a coordination problem (Cluster Membership: A Belief, Not a Fact), not a config file.
Key points
- Sharding is four different problems wearing one word: storage, write throughput, working set, recovery time.
- Partitioning re-scopes guarantees rather than removing them — everything still holds inside a partition.
- Therefore the best partition key is the one that puts each transaction and each invariant inside a single partition.
- Partitioning and replication are orthogonal; the interesting failures live in their composition.
- The routing map is distributed state, not configuration, and it is stale somewhere at all times.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Choose a partition key — a function of the record, fixed at write time.
- • Choose a partitioning function that maps key → partition: a hash, a range, or an explicit directory.
- • Choose an assignment that maps partition → node, kept separately so partitions can move without rehashing keys.
- • Publish the assignment to whatever does routing: clients, a proxy tier, or the nodes themselves.
- • Replicate each partition R ways and record which replica currently leads it.
- • On every membership change, recompute the assignment, move the data, and republish — see Rebalancing: A Load Spike You Schedule for Yourself.
- • The routing map a client holds is older than the cluster’s: requests land on a node that no longer owns the key.
- • A partition grows past what a single node can hold and cannot be split, because it is one key.
- • The map is updated but the data has not finished moving, so the new owner returns "not found" for data that exists.
- • Two nodes both believe they own a partition during a change window, and both accept writes.
- • A cross-partition operation partially applies: one partition commits, the other times out ambiguously (A Timeout Tells You Nothing About Whether It Happened).
- • Skewed shards: one node at 90% disk and 80% CPU while the rest sit at 20%, with no application-level explanation. The operator sees per-node utilisation diverge while aggregate utilisation looks fine.
- • Silent cross-shard breakage: a uniqueness constraint that was enforced by the database before the split now allows duplicates. There is no error — the operator finds out from a customer, or from a nightly reconciliation job.
- • Fan-out latency inflation: a query that used to touch one node now touches all of them, and p99 rises even though every individual node got faster. The operator sees per-node latency flat and end-to-end latency up.
- • Under-replication after a single node loss: because replicas were placed randomly, one node’s death degrades every partition at once, and the rebuild saturates the network for hours.
- • Stale-map errors that follow a deploy curve: error rate rises when a partition moves and decays over exactly the client cache TTL, not the server’s recovery time.
- • No coordination is needed to *read or write within* a partition — that is the entire point of the split, and why partitioning is the cheapest scaling technique available.
- • Coordination is needed to agree on the partition map itself. Every node and client must eventually converge on the same assignment or requests go to the wrong place.
- • Coordination is needed for any operation spanning partitions, and the cost scales with the number of partitions involved, not with the size of the data.
- • The design goal is to make the first case the overwhelming majority. A system where most operations are cross-partition has been partitioned along the wrong axis — see Coordination Avoidance: Restructuring the Problem Instead of Paying for It.
- • A lost node makes its partitions unavailable (unreplicated) or degraded (replicated); it does not affect partitions it did not hold. This containment is a real benefit of partitioning, not a side effect — see Containment Is Decided by What Is Shared, Not by Where the Service Boundaries Are.
- • Blast radius is bounded by the partition map: an incident hits a nameable subset of keys, which is what makes a partitioned system debuggable.
- • A cross-partition operation in flight when a partition becomes unavailable is left in an indeterminate state until something reconciles it.
- • Detect: per-partition health, not per-node — a node can be up while one of its partitions is stuck.
- • Contain: mark the affected partitions unavailable explicitly and fail fast for them rather than letting requests queue against a dead shard.
- • Recover: re-replicate lost partitions from surviving replicas; prefer parallel rebuild across many peers over one source node.
- • Reconcile: re-run cross-partition invariant checks for keys the incident touched, because those are the ones whose guarantees were suspended.
- • Verify: confirm replication factor is restored for every partition, not just that the node count is back.
- • Per-partition size, write rate and read rate — the skew is invisible in any per-node or cluster-wide average.
- • Ratio of single-partition to multi-partition operations. A rising ratio means the key is drifting away from the access pattern.
- • Count of requests answered with "not the owner of this key", broken down by client version — this is your stale-map metric.
- • Number of partitions currently below target replication factor, and for how long.
- • Rebuild time for one partition, measured, not estimated. It is the number that determines your real durability.
- • A genuine capacity ceiling on one of the four axes, where vertical scaling has run out or become uneconomic.
- • Workloads that naturally decompose by tenant, user or account, so nearly every operation is single-partition.
- • When blast-radius containment is worth as much as capacity: a partitioned system fails in slices rather than all at once.
- • When rebuild time under a single-node loss has become the binding durability constraint.
- • Before you actually need it. A single node with a replica handles far more than most teams believe, and every distributed problem in this domain arrives the moment you split — see When Not to Distribute.
- • When the access pattern does not decompose: analytics that always aggregate everything, or a graph where every query walks arbitrary edges.
- • When you must preserve global constraints. You have not removed the coordination, only made it explicit and expensive.
- • When the real problem was a missing index or an N+1 query. Sharding a badly-queried dataset multiplies the bad queries.
- • Scale vertically first. Doubling a machine is a change to a purchase order; sharding is a change to your correctness model.
- • Read replicas, if the ceiling is read throughput rather than writes or storage. Cheaper, and it costs you only staleness (Asynchronous Replication: The Loss Window You Chose).
- • Move cold data out — archive to object storage and keep the hot working set on one node. Frequently the storage ceiling is 90% data nobody queries.
- • Functional decomposition: split *tables* onto different machines rather than splitting rows. Simpler, bounded, and it buys time — though it converts local joins into cross-service calls.
- • Caching in front, if the ceiling is read latency or working set rather than capacity (A Cache Across Machines Is a Replica With No Replication Protocol).
Sharding: where the keys go, and what a resize costs
| Membership change | Keys moved | Unavoidable | Excess |
|---|---|---|---|
| add node-5 | 82 · 16% | 102 | 0.80× |
| remove n4 | 104 · 20% | 104 | 1.00× |
What people believe, and what is true
Sharding makes the system faster.
It raises a capacity ceiling. Individual operations get *slower*, because routing adds a hop and anything cross-partition adds a fan-out. You buy throughput with latency.
You can change the partition key later.
Changing the key means rewriting and re-homing every row while serving traffic, usually with a dual-write and backfill period lasting weeks. It is the most expensive migration in this domain.
More shards is always better.
Each partition carries fixed metadata, connection, compaction and rebalance cost, and every cross-partition query fans out further. Past a point, adding shards raises p99 without adding usable capacity.
Replication and partitioning solve the same problem.
Replication buys availability and durability for the *same* data. Partitioning buys capacity for *different* data. Neither substitutes for the other.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Partitioning splits data across machines to raise a capacity ceiling. It keeps every guarantee inside a partition and gives you none across partitions.
Practical
Name the ceiling (storage, writes, working set, recovery) before choosing a key. Then choose the key so that the operations that must be atomic, unique or ordered fall inside one partition. Measure per-partition load, never per-node averages.
Advanced
Keep two mappings, not one: key → partition (stable, fixed by the data model) and partition → node (mutable, changed on every membership event). Systems that fuse them — hash(key) % nodeCount — cannot move data without rehashing it, which is the subject of the next two lessons.
Apply it
- 🔧 Take a schema you know and list, for each of its five most common queries, whether it is single-partition under a
tenant_idkey. Then do the same under acreated_atkey. - 🔧 Compute your rebuild time: node size divided by realistic rebuild bandwidth. Decide whether that number is acceptable before it is tested for you.
- ⚡ A team shards by hash of
order_id. Reporting queries that filter oncustomer_idnow scan every shard. What are the three ways out, and what does each cost? - ⚡ After sharding, aggregate CPU across the fleet drops to 30% but latency is worse than before. What happened?
- 💬 Your main table is 4 TB and growing 5% a month. Walk me through deciding whether and how to shard it.
- 💬 You shard by
user_id. What breaks on day one, and how do you know before your customers do? - 💬 What is the difference between adding a read replica and adding a shard, in terms of what each one fixes?