Distributed Compute
6 lessons. Every one names the guarantee it claims, what a node can know, and how it fails.
Parallelism on one machine is about using more cores. Distributed compute is about using more machines, and the difference is not a matter of degree: the moment work crosses a machine boundary, communication becomes the dominant term and every task acquires an independent way to fail.
Q · When does splitting a computation across machines actually make it faster, and what does the split cost?
Two functions and one sort. MapReduce is not how anyone should write a new batch job today, and it remains the clearest way to see what distributed computation actually costs — because it puts the expensive part, the shuffle, right in the middle of the picture where you cannot ignore it.
Q · How does a computation expressed as two simple functions become a fault-tolerant job across a thousand machines?
Map is cheap. Reduce is cheap. The step nobody writes — moving intermediate data from every producer to every consumer — is where distributed compute spends its time, its money and its incidents. N mappers times M reducers is N×M transfers, and that product grows faster than your cluster does.
Q · Why is the job network-bound when the computation is trivial?
The intuition every programmer starts with is that you fetch the data and then work on it. At cluster scale that reverses: the code is kilobytes and the data is terabytes, so you ship the code to whichever machine already holds the bytes. Except when you should not — and the exceptions are more common every year.
Q · Is it cheaper to move the data to the computation, or the computation to the data?
A scheduler places tasks, tracks capacity, and re-runs work it believes was lost. That last word is the whole problem: "believes". A scheduler cannot know a worker died — it only knows the worker stopped talking, and re-running a task that is still executing is how a batch job charges a customer twice.
Q · The worker stopped responding. Do I re-run its task?
Behind a barrier, a job finishes when its slowest task finishes. Nine hundred and ninety-nine tasks completing in four seconds buys you nothing if the thousandth takes an hour. The counter-intuitive fix is to do the work twice on purpose — and it is only safe under conditions worth stating carefully.
Q · Why is my job as slow as its worst task, and what can I do about it?