The CI Dependency Graph
A pipeline should be a DAG of real dependencies; stages, sequential steps and path filters are approximations of it, and each approximation has its own way of being wrong.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Which jobs in this pipeline genuinely have to wait for which others, and which are waiting because of how the file is written?
Most pipeline definitions encode an order rather than a dependency structure. The order is a superset of the real constraints, so the pipeline is slower than it has to be — and nobody can tell which waits are real.
Group jobs into stages: build, test, deploy. Everything in a stage runs together, and the next stage starts when the previous one finishes. It is easy to read.
A stage is a barrier. The fastest job in stage two waits for the slowest job in stage one, even when it does not consume anything that job produced.
- A stage is a barrier. The fastest job in stage two waits for the slowest job in stage one, even when it does not consume anything that job produced.
- Barriers compound. Three stages with one slow job each cost the sum of the three slowest jobs, regardless of how the work actually relates.
- The structure hides real dependencies. When someone moves a job between stages, nothing states which other jobs relied on its output, so breakage is discovered by running it.
- In a monorepo the problem inverts: everything runs on every change, including the full service test suite for a documentation edit, because the pipeline has no notion of what a change affects.
- The usual fix — path filters saying "only run this job when files under
services/api/change" — is a hand-maintained approximation of the dependency graph. When it is wrong, a check that should have run does not, and the pipeline reports green. That is a silent false negative, which is strictly worse than a slow pipeline.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The real structure is a directed acyclic graph over *artefacts*. Job B depends on job A when B consumes something A produced: a compiled binary, a container image, a generated schema, a lockfile.
- Everything else is not a dependency. Sharing a language, a directory or a mental category is not consumption.
- When the graph is stated honestly the scheduler does the rest — it runs everything whose inputs are ready. Elapsed time collapses to the longest chain of genuine consumption (Parallelising CI).
- A monorepo needs a second graph: from a set of changed files to the set of targets that could be affected. Build systems that model inputs and outputs per target — Bazel, Nx, Turborepo, Pants — derive this from the build files themselves.
- Path filters approximate that derivation by hand. They fail in one direction quietly: an implicit dependency the filter does not know about means the affected target is not tested, and nothing reports an error.
- The distinction that matters: an over-broad graph costs compute, an under-broad one costs correctness. They are not symmetric mistakes.
Barriers you did not ask for
The diagram shows the same six jobs under two structures. Under stages, the docs job waits for the container image to build even though it reads only Markdown. Under a DAG, it starts immediately.
Note that nothing about the work changed. The only difference is whether the definition states relationships or sequence.
Is that a real dependency?
Most spurious edges come from one of a small number of confusions. Going through a pipeline row by row with this table usually removes half the waiting.
The test is mechanical: name the file, image or record that flows from the upstream job to the downstream one. If you cannot name it, the edge is not a dependency.
| Apparent dependency | Real? | Why |
|---|---|---|
| e2e waits for the container image | Yes | It runs the image. The artefact is the image digest |
| integration waits for unit tests | No | It consumes nothing unit tests produce; it is a preference about which failure you see first |
| deploy waits for all tests | Yes | The gate is the decision, and the decision consumes every verdict (Required Checks) |
| lint waits for build | No | Lint reads source. Sequencing it later only delays the cheapest signal |
| test waits for dependency install | Yes | It consumes node_modules — though a warm cache can make the edge nearly free (Caching in CI) |
| job B waits for job A because both write one cache key | Accidental | A shared mutable cache is an undeclared dependency; give them separate keys |
| publish waits for the security scan | Policy, not artefact | Real and worth stating as such — the edge exists because you decided it does (CI Security) |
| docs build waits for anything | Almost never | Classic stage artefact; docs consume Markdown |
Deciding what a change affects
paths and GitLab rules:changes are pure glob matching with no model of the code at all. The guarantees are genuinely different and are often described with the same word.In a repository with many services, the second graph — from changed files to affected targets — is where the real risk sits. All four approaches below are in production somewhere, and their failure modes point in opposite directions.
The asymmetry is the whole lesson. Running too much wastes money and you find out immediately. Running too little skips a check and you find out in production.
A change touches four files in a repository with thirty services. What runs?
when Small or medium repos, or whenever you are unsure. The only option with no false negatives.
cost Compute and latency scale with repository size rather than change size. Becomes untenable at monorepo scale.
when A handful of clearly separated components with no shared code, and someone owning the filter list.
cost Silently wrong when a dependency is implicit — a shared library, a generated client, a runtime-loaded config. Failure mode is a green pipeline that tested nothing relevant.
when A build system already models per-target inputs and outputs (Bazel, Nx, Pants, Turborepo).
cost Requires that build system, and requires every target's inputs to be declared. Undeclared inputs make the derivation wrong in the same silent direction.
when Almost always, layered on top of one of the above. The pragmatic answer.
cost Trunk can break in ways the PR skipped, so it needs a fast revert path and someone watching the trunk pipeline (Continuous Integration).
How to do it properly
Most important first.
- Express dependencies, not order. In tools with stages, use the explicit dependency keyword to let a job start as soon as its real inputs exist.
- For each edge in your pipeline, be able to name the artefact that flows along it. An edge with no artefact is a habit, and habits are the ones to delete.
- Derive the affected set from the build graph where a build system can do it; treat path filters as a fallback with known false negatives (What a Build System Actually Is).
- When using path filters anyway, make them err wide: include shared directories, lockfiles, the pipeline definition itself, and the base image reference.
- Always run the full graph on trunk, whatever the PR ran. That bounds how long a wrong affected set can hide something.
- Fan-in matters: a job that asserts "everything that should have run, ran" is what turns a silent skip into a visible failure.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A wrong affected set lets an untested change reach trunk and then a deploy; containment is the full trunk run, staging, and the rollout strategy — not CI, which has already said yes.
What can go wrong
- A path filter missing an implicit dependency — a shared library, a generated client, a config file read at runtime — so the affected test never runs and the pipeline is green.
- Filters on the pipeline definition itself, so a change to the workflow file skips the very jobs it changed.
- A dependency edge that exists only because both jobs write to the same cache key, making a cache into an undeclared coupling (Caching in CI).
- A DAG so fine-grained that scheduling and setup overhead per job exceeds the work in the job.
- Diamond fan-in where two branches both rebuild the same artefact slightly differently, and the joining job gets whichever finished last.
- A cycle introduced by accident — job A waits on B, B waits on A — which most tools reject at parse time, but which some express as a job that simply never starts.
- "Stages make the pipeline easier to understand." They make it easier to read and harder to understand, because the barriers hide which relationships are real.
- "Path filters are just an optimisation." They change which checks run. That makes them part of the correctness story, not the performance story.
- "If it is green, the affected set was right." A green pipeline that skipped the relevant test is exactly the failure. Greenness cannot detect its own gaps.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- For every wait in the pipeline you can name the artefact being waited on.
- Adding a job that depends on nothing does not increase pipeline duration.
- A change to a shared module triggers the tests of every consumer — verified by deliberately breaking a shared function and confirming the downstream suite goes red.
- The trunk pipeline and the PR pipeline are reconciled: you know which checks the PR skipped and that trunk ran them.
- Reverting to a coarse graph — run everything, always — is always available and always correct, just slow. That is the right emergency move when you suspect the affected-set logic is wrong.
- When rolling back a graph change, re-run the full pipeline on trunk once before trusting it again, because the changes a wrong graph let through are still in the tree.
- Automate graph derivation from the build system. A generated affected set is checkable and stays correct as the code moves; a hand-written filter list does not.
- Automate a periodic full run — nightly or weekly — as the safety net that catches whatever the affected-set logic missed.
- Do not automate widening the graph in response to a miss without understanding it. "Add the path to the filter" fixes one instance of a class and leaves the class.
- A precise graph is faster and cheaper and is another artefact that has to be correct. The correctness burden moved; it did not disappear.
- Fine-grained jobs parallelise better and multiply per-job overhead: checkout, dependency restore, container pull.
- A build system that models the graph properly gives you affected-set computation and remote caching, and it also imposes its own build description language on every team in the repo (What a Build System Actually Is).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- TOOL-SPECIFICGitHub Actions has only a DAG —
needs— and no stages, so the barrier problem does not exist but neither does a default ordering. GitLab CI defaults to stage barriers and you opt out per job withneeds:. Jenkins declarative pipelines are sequential stages withparallelblocks inside them. Argo Workflows and Tekton model the DAG as first-class graph objects. The same design is four different files. - SCALE-SPECIFICAffected-set computation only pays off in a repository large enough that running everything is genuinely expensive. In a single-service repo, running everything is the correct design and adding a change-detection graph buys latency at the price of a new silent failure mode.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — deciding which tests a change requires, which is the correctness question this graph is trying to answer mechanically.