What a Build System Actually Is
A dependency graph of tasks with declared inputs and outputs, plus a rule for deciding what still needs doing — not a script that runs commands in order.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What is a build system doing that a shell script does not?
Turning source into a deployable artefact involves many steps with real dependencies between them, and re-running all of them on every change is too slow while re-running the wrong subset is incorrect.
A build is a sequence of commands. Write them in a script, run the script, and add a clean target for when things get weird.
A script cannot skip work, so every build is a full build and the feedback loop is set by the slowest possible path (CI Is a Feedback System).
- A script cannot skip work, so every build is a full build and the feedback loop is set by the slowest possible path (CI Is a Feedback System).
- Add skipping by hand — "only recompile if the file is newer" — and you have written a build system, badly, with staleness rules that are wrong in ways nobody has enumerated.
- A script encodes an order, not a structure, so nothing can be parallelised safely: the tool cannot tell which steps are independent (The CI Dependency Graph).
- The existence of a
cleantarget is the diagnostic. It means the incremental path is known to be unreliable and the team has a ritual for working around it. - Undeclared inputs — an environment variable, a tool on the PATH, a file the script reads but never mentions — are invisible to any staleness rule, so the build silently reuses output that no longer matches its real inputs (Reproducible Builds).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A build system models three things: targets (what can be produced), inputs (what each target consumes), and an action (how to produce it). The graph over those is a DAG.
- Given the graph, the system answers one question per target: is the existing output still valid? Everything that distinguishes build systems is how they answer it.
- Timestamp-based systems compare modification times. Cheap, and wrong whenever a clock is wrong, a file is restored, a checkout changes mtimes, or a change does not alter content.
- Content-hash systems hash the inputs. Correct against content, and still blind to inputs that were never declared.
- Action-hash systems hash the entire action — inputs, command line, tool binaries, declared environment — so the hash *is* the identity of the output. That is what makes a remote cache safe to share: a hit cannot be stale, because a different input produces a different key (Caching in CI).
- The strength of the guarantee scales with how much of reality is declared. That is the whole design axis, and it is also what makes strict systems demanding to adopt.
A build is a graph
The graph below is small and already has properties a script cannot express: two independent branches, a shared dependency built once, and a final target that consumes several inputs.
Everything a build system does — skipping, parallelising, caching, computing what a change affects — is derived from this structure. A script has none of it because it has no structure, only an order.
Four ways to decide something is stale
This is the axis along which build systems actually differ, and it decides how much you can trust incrementality and whether a shared cache is safe.
Read the last column as the list of things that will bite you. Every model has one; the question is whether it is a list you can live with.
| Model | Example tools | Rebuilds when | Blind to |
|---|---|---|---|
| Always rebuild | Shell scripts, most npm scripts | Always | Nothing — correct and slow, which is a fine trade at small scale |
| Timestamp | Make | Input mtime is newer than output | Clock skew, restored files, checkouts that reset mtimes, content-identical edits |
| Input content hash | Gradle, sbt, most modern task runners | A declared input's content changed | Anything not declared: env vars, tools on PATH, the network |
| Full action hash | Bazel, Buck2, Pants | Inputs, command, toolchain or declared env changed | Only undeclared inputs — and the tool actively fights those by sandboxing the action (Sandboxing Untrusted Workloads) |
| Layer cache | Docker / OCI builders | An earlier instruction or copied content changed | What a RUN instruction fetched from the network at build time (Layers and the Build Cache) |
Task runner or build system?
Both of these produce the same artefact from the same source. The difference is what the tool knows, and therefore what it can do without being told.
build.sh: generate-client schema.proto compile lib/ compile api/ compile worker/ docker build -t app . the tool knows: nothing can it skip? no can it parallelise? no — order might matter what does a change to api/ rebuild? everything
client: in [schema.proto] out [gen/] lib: in [gen/, lib/src] out [lib.a] api: in [lib.a, api/src] out [api] worker: in [lib.a, worker/src] out [worker] image: in [api, worker] out [image] the tool knows: the whole DAG can it skip? yes, per target can it parallelise? api and worker, automatically what does a change to api/ rebuild? api, image
The second form is not more elegant, it is more informative. Skipping, scheduling and affected-set computation are all mechanical consequences of the tool knowing which inputs feed which outputs — and none of them are available to a tool that was only given an order to follow (The CI Dependency Graph).
How to do it properly
Most important first.
- Know which staleness model your build uses and what it is blind to. "Make with mtimes" and "Bazel with action hashes" both build software and offer completely different guarantees.
- Declare inputs explicitly, including tools and environment variables. Anything undeclared is a correctness hole in caching, in incrementality and in reproducibility at once.
- Express dependencies rather than order, so the tool can parallelise. A build script with an implicit order cannot be scheduled.
- Keep outputs out of the source tree, in a directory keyed by configuration, so two configurations cannot overwrite each other's results.
- Treat any need for
cleanas a bug report about the graph, and go find the undeclared input. - Match tool strength to repository size. A single service does not need a hermetic build system; a repository where a full build is a serious cost usually does (Build Performance).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A build system that reuses stale output can put a wrong artefact into every environment. Containment is a clean-build check and comparing digests before promotion (Tags Versus Digests).
What can go wrong
- Stale output reused because the changed input was never declared — the defining failure, and it is silent.
- A build that only works incrementally: a clean build from a fresh checkout fails, and nobody notices until the release job runs one (Build Environments).
- Generated files committed to the repository and then edited by hand, so the generator's output and the checked-in copy diverge.
- A graph with a hidden cycle expressed through the filesystem — target A writes a file target B reads, without any declared edge.
- Non-deterministic actions inside a correct graph: the graph says nothing needs rebuilding, but the last build embedded a timestamp, so the artefact differs anyway.
- Adopting a strict build system for its speed and then adding escape hatches for every awkward target, keeping the cost and losing the guarantee.
- "A build system is just a task runner." A task runner runs tasks. A build system decides which tasks do not need to run, and that decision is the entire product.
- "Incremental builds are an optimisation." They are a correctness feature with a performance benefit. An incorrect incremental build produces a wrong artefact, not a slow one.
- "We use Docker, so the build is reproducible." A container fixes the operating system layer. It does nothing about unpinned dependencies, embedded timestamps or network access during the build (Reproducible Builds).
- "Everyone should use Bazel." Hermetic build systems pay off at a scale most repositories never reach, and cost real adoption effort at every scale.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- A clean build from a fresh checkout on a fresh machine produces a working artefact. Scheduled, not assumed.
- Touching a source file rebuilds exactly the targets that depend on it, and nothing else — verify by touching one and reading what runs.
- Nobody on the team has run
cleanthis month to fix something. - The build graph can be printed, and its edges match what a person would say the dependencies are.
- The same target built twice without changes does no work the second time.
- Build configuration is code and reverts like code — which is the argument for keeping it in the repository beside what it builds rather than in a CI system's settings.
- When a build change is suspected, the diagnostic rollback is to build clean. If clean succeeds and incremental fails, the defect is in the staleness rules, not in the source.
- Migrating build systems is not reversible in one commit. Run both in parallel and compare artefacts before switching, because the failure mode of a bad migration is a subtly different binary rather than an error.
- Automate the graph derivation from the source layout wherever the tool supports it; a hand-maintained dependency list is stale within weeks.
- Automate a scheduled clean build. It is the only thing that detects undeclared inputs before they matter (Caching in CI).
- Do not automate around a failing incremental build with a blanket
cleanstep in CI. That trades every build's speed for one unfixed defect and hides the defect permanently.
- Stronger guarantees demand more declaration. Hermetic build systems are correct and impose a build description language on everyone in the repository.
- Fine-grained targets give better incrementality and more per-action overhead; past a point the scheduling costs more than the work.
- Generated code speeds builds when cached and adds a class of failure — generator version skew — that hand-written code does not have.
- Keeping build config in the repository makes it reviewable and versioned, and means a broken build config can block its own fix.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- TOOL-SPECIFICMake decides staleness by mtime; Gradle and sbt hash task inputs and outputs; Bazel, Buck and Pants hash the full action including the toolchain and can share results remotely because of it; npm scripts and Makefile-shaped wrappers usually have no staleness model at all and rebuild everything. These are not variations on one design — the correctness guarantees genuinely differ.
- SCALE-SPECIFICFor a repository whose full build is quick, "rebuild everything, always" is the correct design and any incrementality model adds a way to be silently wrong. Incrementality earns its risk only when full builds are expensive enough to change behaviour.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.