Designing the Pipeline
Order checks by signal per unit of cost, gate on the cheap ones, and be explicit about which checks run on a branch, on trunk, and nightly.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Given a set of checks, in what order should they run and which ones should block?
A pipeline assembled by appending each new check to the end reports the cheapest failures last, blocks on things that should not block, and grows monotonically until nobody can say what it covers.
Run everything on every push. Ordering is an optimisation; correctness first. If it gets slow we will add more runners.
Everything-on-every-push means the ten-second formatting failure is reported after the twenty-minute browser suite, for the same commit, at the same time.
- Everything-on-every-push means the ten-second formatting failure is reported after the twenty-minute browser suite, for the same commit, at the same time.
- Adding runners buys throughput, not latency. A single serial job does not get faster because you have more of them (Parallelising CI).
- Checks that cannot be acted on — an advisory scan reporting a hundred findings — accumulate at the same priority as checks that can, and drag the median attention down (Scanning, and Why a Finding Is Not a Risk).
- A pipeline with no notion of "which changes need this" runs the full browser suite on a README edit and the full deploy path on a comment fix.
- When it eventually gets slow, the change that happens under pressure is deleting checks, because that is the only lever nobody had to design in advance.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A check has three properties: what it costs to run, what class of defect it can detect, and how often it is right. Ordering follows from the ratio, not from the category the check belongs to.
- Cheap-first ordering works because failure is not uniformly distributed. Most failures are trivial — a type error, a lint rule, a missing import — and a trivial check catches them in seconds.
- Gating and reporting are separate decisions. A check can block a merge, or annotate without blocking, or run only on trunk and page someone. Conflating them is why pipelines end up with required checks nobody can fix.
- A pipeline is really several pipelines on different triggers: on branch push, on PR, at merge, on trunk, on a schedule, on release. The same check can appear in more than one with different consequences.
- The hard part is not the first design. It is having a rule for where the next check goes, so that the shape survives two years of additions.
Order by signal per unit of cost
The useful ranking is not "unit before integration before end-to-end". It is how much of the likely failure space a check eliminates per second it costs. Some end-to-end checks are cheap and some unit suites are not.
Read the last column carefully. A check's blind spot is what justifies the check after it — and if two checks have the same blind spot, the second one is buying less than it looks.
| Check | Relative cost | Catches | Cannot catch |
|---|---|---|---|
| Format / lint | Seconds | Style drift, obvious dead code, banned patterns | Anything about behaviour |
| Typecheck | Seconds to a minute | Signature mismatches across the whole repo at once | Logic that typechecks and is wrong |
| Build / compile | Minutes | Broken imports, missing files, toolchain mismatch | Runtime behaviour |
| Unit tests | Minutes, shards well | Logic regressions in isolated units | Wiring between units; anything mocked out |
| Integration tests | Minutes, needs services | Wiring, schema mismatches, real query behaviour | Real data volume and concurrency (Why Local Success Predicts So Little) |
| End-to-end / browser | Tens of minutes, flake-prone | User-visible paths across the whole system | Everything not on the scripted path |
| Dependency / secret scan | Seconds to minutes | Known-vulnerable versions, committed credentials | Whether the finding is reachable (Scanning, and Why a Finding Is Not a Risk) |
| Build reproducibility check | One extra build | Undeclared inputs leaking into the artefact | Whether the artefact is correct (Reproducible Builds) |
A gate, then a fan
needs, strategy.matrix and fail-fast are its spelling of the idea. GitLab expresses the same DAG by adding needs: to jobs that would otherwise wait for their whole stage, and its equivalent of matrix expansion is parallel:matrix. Do not copy the keys across.The shape that works for most repositories is one cheap gate followed by everything independent running at once. It is not clever; it is just the thing that happens if you express dependencies honestly instead of writing a sequence.
Note what the needs edges say: unit, integration and the container build all depend on the gate, and on nothing else. They are not sequenced relative to each other because they are not related.
1jobs:2 gate: # cheap, high signal, everything waits on this3 runs-on: ubuntu-latest4 steps:5 - uses: actions/checkout@v46 - uses: actions/setup-node@v47 with: { node-version: 20, cache: npm }8 - run: npm ci9 - run: npm run lint10 - run: npm run typecheck11 12 unit:13 needs: gate14 runs-on: ubuntu-latest15 strategy:16 fail-fast: false # one shard failing must not hide the others17 matrix:18 shard: [1, 2, 3, 4]19 steps:20 - uses: actions/checkout@v421 - run: npm ci22 - run: npm test -- --shard=${{ matrix.shard }}/423 24 integration:25 needs: gate # not needs: unit — they are independent26 runs-on: ubuntu-latest27 steps:28 - uses: actions/checkout@v429 - run: docker compose up -d postgres30 - run: npm run test:integration31 32 image:33 needs: gate34 runs-on: ubuntu-latest35 steps:36 - uses: actions/checkout@v437 - run: docker build -t app:${{ github.sha }} .fail-fast: false on the matrix is the non-obvious line. The default cancels sibling shards on the first failure, which means a run with three independent failures reports one and hides two — and the author pays three round trips instead of one.
The same check, four different triggers
Most arguments about "should this be in CI" are really arguments about which trigger it belongs on. A check that is unbearable as a merge blocker is often perfectly reasonable as a nightly job with an owner.
Write this table down for your own repository. The exercise of filling in the response column is what surfaces the checks that block merges and have no defined response when they fail.
This check is valuable. On which trigger should it run, and what happens when it fails?
when Seconds to run, almost always right, fixable by the author alone. Format, lint, typecheck.
cost Runs constantly, including on drafts. Cheap enough that nobody notices, but it does consume runner capacity on work in progress.
when Detects a defect class that must not reach trunk, is reliable, and finishes inside the attention window (CI Is a Feedback System).
cost Every false positive blocks a human. Every minute is paid by every change, including the one-line fix during an incident.
when Useful context that is not a hard rule — coverage deltas, bundle size, scan findings.
cost Drifts into being ignored. Needs periodic review or it is pure noise (Alert Fatigue).
when Too slow or too flaky to gate, but you want to know within the hour. Full browser matrix, cross-version builds.
cost Trunk can be broken in the ways this covers. Only works with a fast, rehearsed revert path.
when Detects slow drift rather than per-change regressions: dependency advisories, certificate expiry, long soak tests.
cost Failures have no obvious owner because they are not attached to a change. Needs explicit routing or it fails silently for months (The Ownership Record).
How to do it properly
Most important first.
- Put a fast gate first: format, lint, typecheck, and a smoke build. Let its failure short-circuit everything downstream (The CI Dependency Graph).
- Split checks explicitly into blocking and informational, and write down why each blocking one blocks. A required check with no owner is a future emergency bypass.
- Move checks that are slow and rarely fail — full browser matrices, long soak tests, deep scans — off the PR path and onto trunk or a schedule, with a defined response when they fail.
- Make the trigger explicit per check: what runs on a draft push differs from what runs at merge (Required Checks).
- Give every check a failure message that names the file, the expectation and the reproduction command. That is design work, not polish.
- Re-derive the shape periodically. Which checks have never failed? Which fail for reasons unrelated to the change? Both are candidates for removal or for repair.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A misordered pipeline wastes time; a mis-gated one lets a class of defect through to whatever stage catches it next, and the containment is that stage — review, trunk, or a canary.
What can go wrong
- A gate so cheap it catches nothing, giving fast green verdicts with no content.
- Blocking checks that depend on an external service, so a third-party outage stops all merges (How Networks Fail in Production).
- Path filters used to skip checks, applied to a path graph that does not match the real dependency graph — the check that should have run silently does not (The CI Dependency Graph).
- Informational checks that quietly become the only coverage for a class of defect, because everyone assumed something else was blocking.
- Ordering optimised so aggressively that a failure in stage one hides five other independent failures, turning one round-trip into five.
- "Fail fast means abort on the first error." It means report the cheapest signal first. Within a stage, running all independent checks and reporting them together saves the author round trips.
- "Everything important should be a required check." Required means a human is blocked when it is wrong. Some important checks are better as trunk alarms with a fast revert (Continuous Integration).
- "The pipeline is DevOps' concern." The pipeline defines what merging is allowed to mean in this codebase. That is an engineering decision the whole team lives inside.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- For a change with a trivial defect, the failure arrives from the first stage, not the last.
- Every required check maps to a named owner and a written reason for being required.
- The share of pipeline runs that reach the expensive stages and then fail is low — the gate is doing its job.
- You can state, per check, which triggers it runs on and what happens when it fails.
- Pipeline changes are configuration and revert like any other commit — which is exactly why the pipeline definition belongs in the repository next to the code it checks.
- When reverting a pipeline change, remember that the forge's required-check list is usually separate state. Reverting the workflow file while a now-nonexistent check is still required blocks every merge, and the fix is in a settings UI rather than in git (Protected Branches).
- Automate the ordering by expressing dependencies rather than by hand-sequencing steps — the tool can then run everything independent in parallel (The CI Dependency Graph).
- Automate detection of never-failing and always-flaky checks; both are decisions waiting to be made, and both are visible from run history.
- Keep the choice of what blocks a merge human. It is a risk decision about this codebase and this team (Guardrails, Not Gates).
- Cheap-first ordering gives faster failures for common defects and slower total time to full verdict, because stages that could have run concurrently are now sequenced behind a gate.
- Moving slow checks to trunk shortens the PR loop and accepts that trunk can break in the ways those checks cover.
- Fine-grained per-path triggers reduce wasted compute and add a second graph to keep correct — and the failure mode of that graph is silence.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- TOOL-SPECIFICGitHub Actions has no stages at all — ordering exists only as a
needsDAG between jobs. GitLab CI has orderedstagesthat act as barriers unless a job opts out withneeds. Jenkins declarative pipelines havestageblocks that are strictly sequential within a pipeline. The same logical design is expressed three incompatible ways. - GENERALThe ordering principle — cheapest high-signal check first, expensive checks behind a gate — holds regardless of tool, language or scale.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — choosing which layer of the test portfolio covers which risk, which is what determines where each check can sit.