CITOOL-SPECIFICGENERAL

Caching in CI

A cache key is a claim that two inputs are equivalent; when the claim is wrong the pipeline does not get slower, it gets wrong.

What actually happens How to do it properly Evidence and rollback

The question, the obvious approach, and why it breaks

Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.

The production question

What can safely be reused between CI runs, and what does the key have to include for that reuse to be correct?

The problem

Every CI run starts from nothing, so it re-downloads dependencies, recompiles unchanged code and rebuilds identical image layers — work that is genuinely repeated, on inputs that genuinely have not changed.

What teams do first

Cache the dependency directory. Key it on the branch name so each branch gets its own, and add a fallback so new branches start warm.

How it breaks

A branch name is not an input to the build. Two commits on the same branch with different lockfiles share a key, so the second run reuses dependencies resolved for the first.

How it breaks in production

A branch name is not an input to the build. Two commits on the same branch with different lockfiles share a key, so the second run reuses dependencies resolved for the first.
Fallback keys are prefix matches, not equality. A restore key of deps- will happily hand you an entry created three weeks ago from a different lockfile, and report a cache hit.
The failure is silent in the direction that matters. A stale cache does not error; it produces a build that passes tests against dependencies nobody declared, and the difference only appears when a clean environment — usually production — resolves them properly.
Caches accumulate. A directory that is restored, added to and re-saved every run monotonically grows, eventually costing more in restore time than the work it saved.
On a shared or self-hosted runner, a cache is shared mutable state between untrusted inputs. A pull request from a fork that can write the cache can put anything in it, and the next trunk build will execute it (CI Security).
And the thing everyone learns the hard way: a wrong cache produces a wrong answer, whereas a missing cache produces a slow correct answer. These are not comparable costs.

Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve

What is actually happening

Underneath the tooling, which is the part that survives a change of tool.

A cache key is a hash of everything that determines the cached content. If two runs share a key they must produce identical content from that content's inputs — that is the entire contract.
So the key has to include every input: the lockfile, the language runtime version, the operating system and architecture, the compiler flags, and any environment variable the build reads.
What people usually key on is a proxy: the branch, the lockfile alone, or a manually bumped version number. Each proxy is correct until the input it omits changes.
Restore keys — prefix fallbacks — deliberately break the equality contract in exchange for warm starts. That is a legitimate trade only where the cached content is *additive and verified*: a package manager that checks integrity hashes against the lockfile will re-fetch what does not match. It is not legitimate for compiled output, which nothing re-verifies.
Content-addressed build caches are the strong form. Bazel-style systems hash the full action — inputs, tool, command line, environment — and the key *is* the identity of the result, so a stale hit is not expressible.
Container layer caching is a third model again: layers are keyed on the instruction plus the parent layer plus, for COPY, the content copied. Reordering a Dockerfile changes which layers are reusable, which is why instruction order is a caching decision (Layers and the Build Cache).

The key is the claim

Writing a cache key is writing an assertion: "any two runs with this key would have produced the same bytes." Everything that can falsify that assertion has to be in the key.

The pair below is the single most common CI caching bug and the fix for it.

Two keys for the same cache

Keyed on identity

key: deps-${{ github.ref_name }}
restore-keys: |
  deps-

  hits whenever the branch name matches
  falls back to ANY key starting "deps-"
  changing the lockfile does not change the key
  upgrading the runner image does not change the key

Keyed on content

key: deps-${{ runner.os }}-node20-${{ hashFiles('package-lock.json') }}
# no restore-keys for anything that is not re-verified

  lockfile change    -> new key -> miss -> correct
  runner OS change   -> new key -> miss -> correct
  node upgrade       -> new key -> miss -> correct

The first key answers "whose cache is this?" and the second answers "what is in it?". Only the second can be checked against reality. The removal of restore-keys is not caution for its own sake — a prefix fallback returns a different entry and calls it a hit, which is exactly the event you cannot observe.

Four caches, four risk profiles

CI pipelines usually have several caching layers, and they are not equally dangerous. The column that decides how much review a key deserves is the last one.

Layer	Keyed on	Invalidated by	Risk if the key is wrong
Package download cache	Lockfile hash + runtime + OS	Any dependency change	Low — the installer re-verifies against the lockfile and re-fetches mismatches
Installed dependency tree	Same, and often only the lockfile	Nothing checks it after restore	High — a tree built for another platform or runtime is used as-is
Compiled build output	Should be the full action hash	Source, flags, toolchain, environment	Very high — links stale objects into the artefact with no error (What a Build System Actually Is)
Container layer cache	Instruction + parent layer + copied content	Any earlier instruction changing	Medium — deterministic, but a `RUN apt-get install` layer freezes package versions invisibly (Layers and the Build Cache)
Test result cache	Test inputs + code under test	Either changing	High — skips a test and reports it as passed (The CI Dependency Graph)
Remote shared cache	Whatever writers choose	Nothing, if a writer is compromised	Very high — a shared trust boundary across every build (Securing the Pipeline Itself)

When the cache is wrong

TOOL-SPECIFICThe fork-PR row is GitHub-flavoured: caches are scoped per branch with read-through to the base, and a fork's pull_request run gets a restricted token. GitLab protects variables and environments rather than caches, and a self-hosted runner shared across projects has no boundary at all unless you build one (Build Environments).

These are the observed shapes of cache defects. Notice how many symptoms are "works in CI, fails elsewhere" or the reverse — the signature of a build whose real inputs differ from its declared ones.

Trigger	Symptom	Cause	Response
Runner image updated by the provider	Link errors, or a binary that segfaults only in CI	Cache key omits the OS image, so objects built against the old system libraries are restored	Add the runner OS and image version to the key; rotate the key prefix once
Lockfile updated	Tests pass in CI, application fails on a clean install	Key covers the branch, not the lockfile, so the old dependency tree is reused	Key on `hashFiles` of the lockfile; verify by changing it and confirming a miss
New branch created	A build that has never run passes suspiciously fast	A restore-key prefix matched an unrelated entry	Remove prefix fallbacks for anything not re-verified; accept the cold start
Fork opens a pull request	Later trunk build runs unexpected code	Untrusted job could write a cache that trusted jobs read	Never let untrusted triggers write shared caches; isolate or disable caching on fork PRs (CI Security)
Cache grows past a threshold	Pipeline duration rises with no code change	Restore and save now dominate the work they replace	Bound the size, prune on save, measure restore time as its own step
Release build on a clean runner	The release fails on a commit CI approved	The cached path was masking an undeclared input the whole time	Scheduled cold build; treat its failure as a real defect, not as a CI hiccup (Build Environments)

How to do it properly

Most important first.

Key on content, never on branch or run number. The hash of the lockfile is the minimum; add the runtime version and the runner OS.
Use restore-key fallbacks only for caches whose contents are re-verified against a manifest — package manager download caches qualify, compiled output does not.
Cache the download, not the installed tree, where the ecosystem allows it. Re-running the install against a warm download cache is slower than restoring node_modules and it is verified against the lockfile.
Scope caches per branch for writes, with read access to the trunk cache — the shape most forges implement, and the reason a fork PR cannot poison trunk's cache.
Make one job in the pipeline run cold on a schedule. A nightly no-cache build is how you find out that the cached path had been hiding a broken build for a month.
Bound the size and give caches an expiry, so growth is a policy rather than a surprise.
When a build behaves inexplicably, clear the cache before investigating anything else — and treat the fact that it fixed things as a defect in the key, not as a resolution.

How much can this affect

Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.

Blast radius if this is wrongEveryone

One testEveryone

What contains it

A poisoned or stale cache can contaminate every artefact built from it, including released ones. Containment is a cold rebuild and provenance records that let you tell which artefacts came from the affected window.

What can go wrong

Failure modes, including of the mitigation

A key missing the toolchain version: the runner image is updated, the cache from the old compiler is restored, and the build links objects from two toolchains.
Restore-key prefix matching returning a much older entry and reporting a hit, so the run is neither cold nor correct.
Cache poisoning from an untrusted branch on a shared runner, executing attacker-controlled content in a trusted build (The Delivery Chain as Attack Surface).
A cache that is written but never invalidated because the invalidating input is not part of the key — the classic being an environment variable read at build time.
Cache restore becoming the critical path: a multi-gigabyte archive takes longer to download and unpack than the work it replaces.
A green pipeline that only passes warm. The clean build in the release job fails, and nobody can reproduce it locally because their machines are warm too (Reproducible Builds).

Misreads this invites

"A cache miss is a failure." A miss is the safe outcome. The dangerous outcome is a hit that should have been a miss.
"We hashed the lockfile, so the key is content-addressed." Only if the lockfile is the sole input. The runtime version, the OS image and build-time environment variables are inputs too.
"Clearing the cache fixed it, so it is resolved." It is diagnosed, not resolved. Something is missing from the key and it will happen again (Production Debugging).
"Caching is a performance concern." It is a correctness concern with a performance benefit. That is why the key deserves review.

Operating it

Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.

How you know it worked

A scheduled cold build passes. This is the single most valuable piece of evidence in this lesson.
Cache hit rate is visible per cache, and a change to the lockfile produces a miss — verified by making one.
The key string, printed in the log, names every input you believe determines the content.
Artefacts built from a warm cache and from a cold cache have the same digest (Tags Versus Digests).
Restore time per cache is measured, so growth is noticed before it dominates.

How you get back

Disable the cache. It is always safe, always available, and costs only time — which is precisely why a cache problem should never turn into an outage.
Rotate the key by adding a version prefix rather than deleting entries by hand; deletion races with in-flight jobs, a prefix bump does not.
After any suspected poisoning, treat the cache as compromised, purge it, and rebuild cold — and then check what was published from builds that used it (Build Provenance).

What to automate, and what stays human

Automate key construction from real inputs — hash the lockfile and interpolate the runtime and OS — so nobody has to remember to bump a manual version.
Automate the cold-build canary on a schedule, with a real owner for its failures.
Do not automate "clear the cache on failure" as a retry step. It converts a key defect into a permanent, invisible cost and removes the only symptom you had.

What this costs

Precise keys are correct and miss more often; loose keys hit more and are sometimes wrong. There is no setting that is both, only a choice about which error you can tolerate.
Caching the installed tree is faster than caching the download and skips the verification step the install would have performed.
A remote shared cache gives every runner warm starts and makes the cache a shared trust boundary, with the blast radius of anything that writes to it.
Storage costs money and restore costs time; past a certain archive size the cache is a net loss and nothing warns you.

Where this applies

This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.

TOOL-SPECIFICGitHub actions/cache uses exact-key lookup with restore-keys prefix fallback and scopes caches by branch with read-through to the default branch; GitLab cache:key:files hashes named files and has no prefix fallback; Bazel and Gradle remote caches are content-addressed on the full action hash and cannot return a stale entry by construction. These offer genuinely different safety guarantees and are all called "the cache".
GENERALThe contract — a key must include every input that determines the content — holds for every caching layer at every level, from a CI dependency cache to a CPU cache line. Only the consequence of getting it wrong differs.

Where the depth lives

This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.

Cloudcontainer-images

Observability & Performancecache-performance-signals

Securitysupply-chain-security artifact-integrity