Move the Computation to the Data

A program that processes a file normally reads the file. Scaled up, that becomes: a thousand workers each pull their slice across the network from wherever it lives, and the aggregate read is the entire data set moving over the network once per job. If the data is a hundred terabytes and the job runs hourly, that is a lot of network for a computation whose code is a few hundred kilobytes.

Turn it around. Put the workers on the machines that already hold the data, ship the code to them, and the only thing crossing the network is the code and the results. For a filtering or aggregating job the results are far smaller than the input, so total network traffic drops by orders of magnitude. This was the founding insight of the chunk-based file systems in Distributed File Systems: Chunks, a Metadata Service, and Where the Copies Go: chunks are placed on machines, jobs are scheduled onto those machines, and reading is a local disk read.

The reversal holds when three things are true: the data is much larger than the code, the computation reduces the data substantially, and local storage is genuinely faster than the network. Every one of those was overwhelmingly true in 2004. Two of them are still usually true. The third is exactly where the modern exceptions live.

A scheduler that insists on locality will sometimes have no local slot free, and then it has a choice: wait for one, or run the task somewhere else now. Both are wrong some of the time. Waiting leaves capacity idle; not waiting reads over the network.

Delay scheduling is the standard answer, and it is a nice piece of engineering. When a task’s preferred machines are busy, wait a short bounded time — a second or two — for a local slot, then fall back to rack-local, then to anywhere. Because tasks are short and slots free up constantly, a very small wait converts most assignments to local ones. The scheduler gets most of the benefit of insisting on locality with almost none of the idleness.

The lesson generalises past scheduling: a preference with a bounded fallback usually beats both a hard requirement and no preference at all. A hard requirement turns a performance optimisation into an availability risk — if the machines holding the data are down or full, the job does not run. No preference at all leaves a large, free win on the table. The bounded version captures the win and degrades rather than failing.

The rule has real exceptions, and knowing them is more valuable than knowing the rule, because the exceptions are the growing case.

The data is small. Moving a hundred megabytes is nothing. Constraining placement to chase it costs scheduling flexibility and buys a rounding error. Locality reasoning only pays when the data is large relative to everything else in the job.

The compute is heterogeneous. If the job needs a GPU and the data sits on general-purpose storage nodes, there is no local option and never will be. The same applies to any specialised resource: the placement is dictated by the scarce resource, and the data comes to it. This is the normal condition for model training, where the accelerators are the constraint and feeding them is a pipeline problem.

Compute and storage are already separated. This is the big one. When your data lives in object storage, every read is a network read for every worker, and there is no local option to prefer. Locality has not been violated; it simply does not exist as a concept in that architecture. The industry moved this way deliberately, and for good reasons — storage and compute scale independently, you stop paying for idle disks attached to busy CPUs, and you can run ephemeral or spot workers that hold no state. The price is that the founding assumption of data locality is gone.

The "local" disk is not local. A cloud instance’s attached volume is frequently a network-attached block service. A "node-local" read is a network read wearing a filename. Reasoning about locality against a diagram rather than against the actual storage path is a good way to optimise something that does not exist.

Separating compute from storage does not repeal physics; it relocates the problem. If every read crosses the network, the techniques that matter become the ones that read less and the ones that read once.

Reading less is a data-layout question. Columnar formats let a query fetch three columns of two hundred; partition pruning skips whole directories; predicate pushdown and per-file statistics let a reader decline to fetch a file at all. These regularly cut bytes read by one or two orders of magnitude, which is a larger effect than locality ever provided.

Reading once is a caching question. A local SSD cache in front of remote storage restores something close to node-locality for repeated reads, and scheduling a task onto the node that already has its input cached is *locality again* — just against a cache rather than against the primary store. This is why cache-aware scheduling exists in engines that read from object storage: the concept survives, the thing it is measured against changes.

And the cost model changes shape. Locality used to be about time; with object storage it is also about money — per-request charges and, if the compute and the bucket are in different regions or clouds, egress. A job that reads across a region boundary can cost more in transfer than in compute, which is a category of bill that surprises people once each.