Where the Compiler Ends and the Build System Begins
The compiler turns one set of sources into one artifact. The build system decides which of those invocations must run at all. Getting that division wrong — most often by trusting timestamps — produces both missed and spurious rebuilds.
What is the build system's job, and what is the compiler's?
Two representations meet here and neither knows about the other. The compiler operates on a program — tokens, trees, IR — and produces an artifact from a command line. The build system operates on a graph of *actions*: opaque commands with declared inputs and outputs, whose contents it never inspects. The build system's question is "which actions must run?"; the compiler's is "what does this source mean?". Neither can answer the other's.
A build system may skip an action only if it can establish that re-running it would produce the same outputs, which requires that every input be enumerated and compared by a property that actually changes when the content changes. Timestamps satisfy neither half reliably: a file can be rewritten with identical content, restored from a backup with an older date, or checked out fresh with a newer one. Content hashes satisfy the second half and leave the first — completeness of the input set — as the standing obligation of whoever wrote the build rules.
Key points
- The compiler answers semantic questions about one unit; the build system answers scheduling questions about a graph of opaque actions.
- They communicate through two narrow channels: a command line forwards, and a report of files actually read backwards.
- Timestamps produce both missed rebuilds — restored or checked-out files with old dates — and spurious ones, because checkout and code generators rewrite unchanged content.
- Content hashing removes both failure modes and costs I/O, usually mitigated by an mtime fast path or a filesystem watcher.
- A content-addressed key is machine-independent, which is what makes shared and remote caches possible — and which requires the compiler to be deterministic to be worth anything.
The division of labour
The line is sharper than it looks. The compiler knows what a symbol is, what a type is and what an interface change means; it knows nothing about which files exist or what was built yesterday. The build system knows the state of the filesystem, the previous build and the available cores; it treats clang -c a.c -o a.o as an opaque string. Neither can do the other's job, and every attempt to merge them has produced a build system that only supports one language or a compiler that only supports one project layout.
They communicate through two narrow channels. Forwards, the build system supplies a command line — the flags, the include paths, the target. Backwards, the compiler reports what it actually read, via -MMD dependency files or an equivalent. That backwards channel is what stops the edge set from being a fiction, and a build system that lacks it is guessing.
| Question | Answered by | Because |
|---|---|---|
| What does this name refer to? | Compiler | It has the symbol table; the build system has filenames |
| Did the public interface change? | Compiler | It produced the interface artifact and can hash it |
| Which files did this compile actually read? | Compiler, reported to the build system | Only the process that opened them knows |
| Does this action need to run? | Build system | It holds the previous state and the input hashes |
| In what order, and how many at once? | Build system | It owns the graph and the machine |
| Can this result come from another machine? | Build system | Cache keys and trust are its concern, not the compiler's |
| Is this program correct? | Compiler | The build system never looks inside an artifact |
Timestamps fail in both directions
The classic build system compares modification times: rebuild if any input is newer than the output. It is cheap — one stat per file — and it is wrong in two distinct ways, which is unusual for a heuristic.
Missed rebuilds. Restore a file from a backup or a stashed copy and its mtime may be older than the artifact built from a newer version. Check out an older branch and the same thing happens. Generate a file in the same filesystem-timestamp granularity as the build step that consumed it — one second on some filesystems — and the comparison sees equality and skips. In each case the build is stale and reports success.
Spurious rebuilds. git checkout rewrites files with fresh mtimes whether or not their content differs, so switching branches and switching back rebuilds everything. A code generator that rewrites its output unconditionally re-triggers everything downstream on every build, even when the bytes are identical. A touch invalidates the world.
Content hashing fixes both: hash the inputs, compare against the hash recorded for the last successful run of that action, and skip if they match. Rewriting a file with identical content is a no-op; restoring an old version is correctly detected as a change. The cost is real — every input is read and hashed on every build — and mature systems mitigate it with an mtime-and-size fast path that falls back to hashing, or with a filesystem watcher that knows what actually changed.
// timestamp rule
if mtime(a.c) > mtime(a.o) or any(mtime(h) > mtime(a.o) for h in headers):
run("clang -c a.c -o a.o")// content rule
key = hash(contents(a.c), contents(headers), command_line, compiler_id, target)
if key != recorded_key(a.o):
run("clang -c a.c -o a.o")
record(a.o, key)The content rule may skip an action only if key covers every input that can influence the output — sources, headers, the full command line, the compiler binary's identity and the target description. Under that condition an equal key means an identical output, so skipping is behavior-preserving.
The action reads something not in the key: a header found via an ambient CPATH, a generated file produced by a step outside the graph, an environment variable consulted by a build script, or the system clock. Then equal keys do not imply equal outputs and the build serves a stale artifact — which is exactly what [[hermetic-compilation]] exists to prevent.
Why content hashing unlocks sharing
-fdebug-prefix-map and -ffile-prefix-map exist; some versions also embed build timestamps unless SOURCE_DATE_EPOCH is set. A cache key is only as meaningful as the compiler's willingness to produce identical bytes twice.Once an action is keyed by a hash of all its inputs, the key is machine-independent. That means the output can be stored in a cache shared by a whole team or a CI fleet, and a developer who compiles a file nobody has changed can download the object file rather than produce it. This is the design of Bazel, Buck, Nix and sccache, and it is why they insist so hard on enumerated inputs: the value of the cache is exactly the trustworthiness of the key.
It also gives a much stronger definition of a correct build. If two machines compute the same key, they must produce the same artifact — which is a statement about the compiler, not the build system, and one that is only true if the compiler is deterministic. Embedded timestamps, absolute paths in debug info and randomized layout all break it, which is where this lesson hands over to [[reproducible-compilation]].
How it works
The steps, in the order the compiler takes them.
- The build system reads build rules to construct a graph of actions with declared inputs and outputs.
- For each action it computes a key: a hash over input contents, the command line, the compiler identity and the target description.
- It compares the key against the key recorded for the last successful execution of that action.
- If the keys differ, it runs the action — which is where the compiler is finally invoked, on one unit.
- The compiler reports the files it actually opened; the build system folds these into the graph so the next key is complete.
- The new outputs and key are recorded, and if a shared cache is in use, uploaded under the key for other machines to reuse.
How it breaks
What the engineer observes when it goes wrong — not what goes wrong internally.
- A branch switch rebuilds the entire project because checkout rewrote every mtime, and the team concludes the build is simply slow.
- A file restored from a stash is never recompiled because its date is older than the object file, and a bug that was supposedly fixed reappears.
- A generated file is rewritten identically on every build, so everything downstream of it rebuilds every time and incrementality is silently dead.
- A shared cache serves an object built with a different compiler patch version because the key omitted the compiler binary, and a link fails with an inscrutable symbol error.
- Two engineers get different binaries from the same commit, because an undeclared input differs between their machines and neither key reflects it.
When it helps
- Any repository where full builds are expensive enough that correctly skipping work is worth engineering effort.
- CI, where content-addressed keys let a fleet of ephemeral machines reuse each other's work rather than rebuilding from scratch every time.
- Multi-language builds, where the graph is the only thing that spans the toolchains and no single compiler could coordinate them.
When it hurts
- Small projects, where hashing every input costs more than the compiles it avoids and a two-line
makerule is genuinely the right answer. - Builds dominated by a few enormous actions, where per-action granularity is too coarse for skipping to help and the win must come from inside the compiler instead.
What it costs
Every one of these is paid by something.
- Content hashing buys correct skipping in both directions and pays in I/O — every input read and hashed on every build — plus the bookkeeping to store keys.
- Enumerating every input buys a trustworthy cache key and pays in build-rule verbosity and in friction whenever someone adds a dependency the rules did not anticipate.
- A shared remote cache buys other people's compile time and pays in network transfer, cache storage, and a trust decision about artifacts you did not build.
- Keeping the build system ignorant of language semantics buys support for every toolchain at once, and pays with a granularity floor: it can never skip part of an action, only the whole thing.
What else you could do
What a different compiler or language does instead, and when that is better.
- Timestamp comparison, as in classic
make: nearly free, universally available, and wrong in the two directions described above. Still the right answer for a small project. - Filesystem watching, as in Watchman-backed builds, which learns what changed from the OS rather than by polling — accurate and cheap, and unavailable in ephemeral CI containers.
- Compiler-internal incrementality, which achieves much finer granularity than any action-level system can, at the cost of being machine-local — see
[[incremental-compilation]]. - Sandboxed hermetic execution, as in Bazel and Nix, where an action runs with only its declared inputs visible so an undeclared read fails loudly instead of quietly succeeding.
See it for yourself
The flag, dump or tool that shows you this directly.
make -dexplains every rebuild decision it made and which timestamp comparison drove it.ninja -t depsandninja -t recompactshow the dependency records the compiler reported;ninja -ndry-runs the schedule.bazel aquery --output=text //app:binprints the concrete actions and their input sets — the key made readable.sccache --show-statsreports hit rates and, importantly, the reasons for misses, which is where key-completeness bugs surface.- Run a build twice with no changes. Anything that rebuilds is a key bug, and this thirty-second test catches most of them.
Plausible wrong readings
Stated the way a confident engineer states them.
- "The build system knows about my code." It knows filenames and command lines. Everything it appears to know about your code came from the compiler reporting it.
- "Timestamps are fine in practice." They fail on checkout, on restore, on same-second writes and on unconditional generators — all of which happen daily.
- "Content hashing makes builds slower." It adds I/O and removes rebuilds. On any project past trivial size the second dominates, which is why mature systems all moved to it.
- "If the key matches, the artifact is correct." Only if the key is complete. An undeclared input makes a matching key meaningless.
Misconceptions
The claim, and what is actually true.
Go deeper
The same idea at increasing depth. Stop wherever it stops being useful.
overview
The compiler takes source and produces an artifact. The build system decides which of those compiles need to happen, in what order, and how many at once. It does not understand your code at all — it sees commands with inputs and outputs. Its whole job is skipping work that would produce the same result.
practical
The most useful thing you can do to a build is run it twice and see what rebuilds the second time. Anything that does is a bug in the input set: an unconditional generator, an undeclared file, a timestamp embedded in an output. Fixing those is usually a bigger win than any compiler flag. After that, check that the cache key includes the compiler version and the flags — the failures from omitting those are rare, confusing and expensive.
advanced
The interesting boundary question is granularity. A build system can only skip whole actions, so its floor is one compiler invocation, and shrinking actions to get finer skipping runs into per-action overhead — process startup, sandbox setup, cache lookups — that quickly exceeds the compile. That floor is precisely why compiler-internal query systems exist: below one invocation, only the compiler can see the structure. The two mechanisms are complementary and their caches have opposite properties — the build system's is content-addressed, shareable and durable across versions, while the compiler's is machine-local and invalidated by every upgrade. Systems that tried to unify them ended up with a build system that understood exactly one language.
How much this depends on
Nothing in this domain is true of every compiler. These say how much.
make and most IDE builds remain timestamp-based; Bazel, Buck2, Nix and Gradle are content-based. Which model you are on determines which of the two failure families you will meet, and neither model is universal even within one organization.-ffile-prefix-map is used, and may embed build timestamps unless SOURCE_DATE_EPOCH is set. A shared cache built on a non-deterministic compiler produces keys that match while artifacts differ.If you were asked this in an interview
- Name a case where timestamps cause a missed rebuild and a case where they cause a spurious one.
- What has to be in a cache key for a shared build cache to be sound?
- Where is the boundary between what a build system can skip and what only a compiler can skip?
Connections
- DevOps / Production Engineering — Build pipelines, remote execution, artifact registries and cache trustEverything on the far side of the cache key — where artifacts are stored, who may write to the cache, how a CI fleet shares work, and what it means to trust a binary you did not build — is that domain's subject. We stop at the question of when reusing an artifact is behavior-preserving, which is the only half a compiler can answer.