Distributed Storage
6 lessons. Every one names the guarantee it claims, what a node can know, and how it fails.
Open any distributed store and you find the same six layers: an API, a partitioner, a replicator, an agreement or conflict-resolution layer, a single-node storage engine, and a disk. Nothing in that stack is new to you — you have already met every layer separately. What is new is that the guarantees compose, and mostly compose downward.
Q · What is actually inside a "distributed database", and which layer owns which guarantee?
GFS and HDFS answered one question well: how do you store a file bigger than any single disk, on machines that fail weekly, and still read it at the speed of many disks at once? The answer — split the file into large chunks, replicate each chunk, and keep the map in one small service — is thirty years old and still the shape of the thing.
Q · How do you store a file that is larger than any one machine, on machines that keep dying?
No directories, no partial updates, no file handles. An object store takes a key and a blob of bytes, hashes the key to decide where the bytes live, and gives you back a guarantee about durability that is far stronger than anything you would build. The price is that everything you liked about a file system is gone.
Q · How does a key turn into bytes on a specific set of disks, and what does the flat namespace cost you?
A write returned 200. That fact alone tells you almost nothing. It may be in a memory buffer on one machine, on one machine’s platter, or on three machines in three zones — and the difference is invisible to the client, invisible on the dashboard, and decisive the moment a machine is replaced.
Q · The write returned 200 and then the machine died. Is the data still there?
There are exactly two ways to remember what a system knows: write down the current state, or write down every change. Every recovery scheme worth using does both — a periodic snapshot of state, plus the ordered log of everything since. The same pair, unchanged, is how a replica catches up.
Q · After a crash, how does a node rebuild what it knew — and why does the same structure make replication work?
You want a picture of what the whole system knew at one moment. There is no one moment — every node has its own clock and its own present, and messages are in flight between them while you look. Chandy and Lamport showed how to take a picture that is *causally* consistent anyway, without pausing anything.
Q · How do you capture a consistent global state of a running distributed system without stopping it?