Distributed File Systems: Chunks, a Metadata Service, and Where the Copies Go

The founding move is to split the file system in two. A metadata service holds the namespace — directories, file names, permissions, and for each file, the ordered list of chunks it is made of. Chunk servers hold the bytes and nothing else: a chunk is just a large opaque blob with an id, typically 64 MB or 128 MB, stored as an ordinary file on an ordinary local file system.

The chunk size is the design decision everything else follows from. Make chunks small and the metadata service must track billions of them; its map stops fitting in memory and every operation gets slower. Make them large and the map stays tiny — a petabyte at 128 MB per chunk is about eight million entries, which fits comfortably in RAM — but a small file wastes a whole chunk entry and a hot chunk cannot be spread across more machines. Large chunks also mean the metadata service is off the data path entirely: a client asks *once* where the chunk is and then talks to chunk servers directly for a hundred megabytes of reads.

That last property is the whole performance story. The metadata service handles a few thousand lookups a second; the chunk servers collectively handle tens of gigabytes a second, because there are a thousand of them and each read goes straight to a disk. Bandwidth scales with the number of machines because the coordinator is not in the way.

The replication factor says *how many* copies. The placement policy says *where*, and it is the one that decides whether the copies are worth anything. Three copies on three machines in the same rack survive three independent disk failures and do not survive the rack switch dying. The classic policy — one replica local to the writer, one on a different rack, one more on that second rack — is a deliberate compromise: it buys rack-failure survival while keeping two of the three writes off the expensive cross-rack link.

This is Correlated Failure: The Independence Assumption Is Usually False applied to bytes. A replication factor of three is a statement about *independent* failures; the moment the failures are correlated, three copies behave like one. Cloud deployments repeat the exercise one level up — the failure domain becomes the availability zone, and the policy has to spread across zones or the replication factor is decorative.

The second thing placement decides is re-replication cost. When a machine is lost, every chunk it held is now under-replicated, and the cluster must copy those chunks from their surviving replicas. If the surviving replicas are concentrated on a few machines, re-replication saturates those machines’ network links and the recovery itself becomes an incident. Spreading each machine’s chunks across *many* peers means the repair of one machine is a small amount of work for hundreds of machines rather than a large amount of work for three.

GFS ran one master. HDFS ran one NameNode for most of its life. This was not an oversight and it was not a mistake — it was a trade the designers made on purpose and documented clearly. A single master means the namespace has a single authority, so there is no distributed agreement in the metadata path at all: file creation, renaming and chunk allocation are ordinary operations on one machine’s in-memory data structures, with an operation log on local disk. That is dramatically simpler than the alternative and, for the workload these systems targeted — a few thousand large files, written once and read many times — it was fast enough.

What it costs is honest and severe. The master is a single point of failure for the namespace, and the recovery time is not instant: a restarting master has an operation log to replay and must wait for chunk servers to report which chunks they hold before it can safely allocate or re-replicate anything. On a large cluster that was historically tens of minutes. During that window the data is entirely intact and entirely unreachable, because nobody knows where any chunk is.

It also caps the file count. The whole namespace lives in the master’s memory, so the number of files is bounded by one machine’s RAM — which is why these systems are miserable for millions of small files, and why "small files problem" is a phrase every HDFS operator knows. Later designs answered this in two ways: standby masters with a shared or quorum-replicated edit log, which cuts failover from minutes to seconds without removing the single-active-authority model; and federated or sharded namespaces, which split the namespace across several masters and thereby give up a single global view.

The lesson generalises past file systems. A single-authority metadata service is a legitimate design when the metadata operation rate is small relative to the data operation rate. The question to ask is never "is a single master bad?" but "what is my recovery time when it is gone, and can the data plane keep serving reads while it is?"

These systems do not offer general random writes with useful semantics. A concurrent write to the same offset from two clients has no defined outcome, and the designers said so. What they *do* offer is atomic record append: many writers can append to the same file concurrently, each record lands at least once, and each record is contiguous. The offset is chosen by the system, not the client.

This is a beautiful example of matching the guarantee to the workload. The target workload was log ingestion and batch output — many producers appending records, one batch job later reading everything. "At least once, contiguous, at an offset we choose" is exactly enough for that and vastly cheaper than making arbitrary concurrent writes linearizable. The cost is that duplicates are possible and the reader must tolerate them, which drops you straight into Deduplication: Bounded Memory Against an Unbounded Stream and Where You Put the Acknowledgement Decides Everything.

Notice also what this makes the file: an The Log Is Not a Queue structure at the file-system layer. The same shape reappears at every level of this domain — the log is the primitive that makes both recovery and replication tractable, which is the subject of Recovered State Is a Checkpoint Plus the Log After It.