Triaging a CI Failure
A red pipeline has four common causes with four different correct responses, and telling them apart quickly is a learnable procedure.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
The pipeline is red. Is that my change, the environment, a flake, or something already broken on trunk?
A failed check reports that something is wrong, not what kind of wrong. Without a procedure, everyone applies the same response — re-run it — which is correct for exactly one of the four causes.
Re-run the job. If it passes, it was a flake and we can move on. If it fails again, read the log and fix it.
Re-running is a legitimate *diagnostic* and a terrible *fix*. It distinguishes deterministic from intermittent failure and then tells you nothing about which intermittent cause you have.
- Re-running is a legitimate *diagnostic* and a terrible *fix*. It distinguishes deterministic from intermittent failure and then tells you nothing about which intermittent cause you have.
- A flake that passes on re-run is not resolved, it is deferred — and the deferral is invisible, so nobody counts how often it happens (Flaky Tests).
- If trunk is already broken, everyone independently investigates the same failure on their own branch, and the total cost is one defect multiplied by the number of open pull requests.
- Environment failures — a registry timeout, an expired token, a runner image change — look exactly like code failures in the log, and the response is completely different.
- Logs designed for machines make every cause look the same:
exit code 1after four thousand lines is not a signal, it is a homework assignment.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The four causes are distinguishable by cheap questions, in a specific order, because each question is cheaper than the one after it.
- The first question is not about your change at all: is trunk green? If it is not, your failure is probably not yours, and the correct action is to look at trunk's response rather than to debug your branch.
- The second is whether the failure is in code the change touched. That is a strong prior, not proof — but it points investigation at the right half of the diff.
- The third is determinism: does it fail identically on the same commit? A single re-run answers it, which is the one legitimate use of the button.
- The fourth is reproducibility outside CI. A failure you can reproduce in a container locally is an ordinary debugging problem; one you cannot is a statement about the environment (Build Environments).
- Underneath all of it: how much a failure costs to triage is decided when the check is written. The message, the artefacts it saves, and whether it prints a reproduction command are design choices, not log formatting.
Four causes, four responses
These are the failures that make up nearly all red pipelines. They share a symptom — a red check — and share almost nothing else.
The value of the table is in the last column. If the response is the same for every row, you are not triaging, you are guessing.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Your change altered behaviour a test asserts | Deterministic failure, in or near the code you touched | A genuine regression, or a test that encoded the old behaviour | Fix the code, or update the assertion and say why in the commit message |
| Nothing in particular; it fails sometimes | Same commit, different verdicts | Order dependence, timing, shared state, unseeded randomness | Classify, quarantine with an owner and a deadline, and fix the underlying race (Flaky Tests) |
| Registry, network, runner or token | Failure in a setup step, before any test ran | External dependency of the pipeline, not of the code | Retry the step with backoff, and count the retries so degradation is visible (How Networks Fail in Production) |
| A change that already landed | The same failure on every open pull request | Trunk is broken and every branch inherited it | Revert on trunk; do not debug it on your branch (Continuous Integration) |
| The runner image or a pinned action changed | Everything failing since a specific time, no code change | The environment moved under an unpinned input | Pin the input, then investigate; this is a dependency problem wearing a CI costume (Dependency Pinning) |
| A dependency published a new version | Failure in code nobody edited, starting overnight | An unpinned range resolved to something new | Pin, reproduce with the old version to confirm, then upgrade deliberately (Dependency Management) |
Triage in order of cheapness
The order below is chosen so that each step costs more than the one before it, and each can terminate the investigation. Most failures are resolved in the first two steps, which take seconds.
The third step is the one to be disciplined about: re-running is a measurement, and a measurement you do not record is not a measurement.
- 11. Is trunk green?
Separates "my change" from "everyone's problem" before any log is read.
fails by Skipped, because it needs a different tab. Put the link in the failure notification.
evidence Trunk pipeline status at the time your job started, not now.
- 22. Is the failure related to the diff?
Points investigation at part of the change instead of at the whole log.
fails by A false negative: unrelated-looking failures are often the real signal from an unintended consequence.
evidence The failing test's subject compared against the files touched.
- 33. Re-run once, and record it
Distinguishes deterministic from intermittent.
fails by Becoming the fix. Once the second run passes, the incentive to investigate disappears.
evidence A recorded pass-after-fail event on an unchanged commit — which is the definition of a flake (Flaky Tests).
- 44. Reproduce outside CI
Separates the code from the environment.
fails by Reproducing on a laptop that shares none of the runner's properties, so a pass proves nothing.
evidence Reproduction in the same container image digest the runner used (Build Environments).
- 55. Bisect
Names the commit responsible when the change is large or the history is unclear.
fails by Bisecting an intermittent failure — the result is noise unless each step runs enough times.
evidence A specific commit whose revert makes the failure disappear (Change Correlation).
Steps 1 and 2 cost seconds; step 5 costs an afternoon. The ordering is the entire method.
Failure output is a design decision
Two checks can detect the same defect and cost the team wildly different amounts, because triage cost is decided by what the check prints when it fails.
This is not about verbosity. The second block below is shorter than the log the first one buried its message in.
... 4127 lines of build output ... FAIL src/billing/invoice.test.ts Error: expect(received).toEqual(expected) at Object.<anonymous> (invoice.test.ts:214:22) Process completed with exit code 1
FAIL billing » invoice » applies tax before discount expected total: 1080 (tax 80, discount 0) received total: 1000 (tax 0, discount 0) seed: 4429871 image: ci-node20@sha256:1f0c... reproduce: npm test -- invoice --seed 4429871 artefacts: invoice-request.json, invoice-response.json
Both reports contain the same information about the defect. The second also contains everything needed to act: which behaviour is wrong, what the values were, and a command that reproduces it in the same environment. The seed and the image digest are what make the reproduction actually reproducible — without them the author is running a different experiment and drawing conclusions from it.
How to do it properly
Most important first.
- Check trunk health first, every time. Make it a link people can see from the failure, not something they have to think to do.
- Adopt a revert-first rule for trunk: a broken mainline is reverted, and the fix is re-landed afterwards. Debugging on a red trunk blocks everyone (Continuous Integration).
- Re-run exactly once, to classify. Record the outcome — a second run that passes is data about flakiness, and it should be counted somewhere.
- Make failures reproducible by construction: print the exact command, the seed, the container image digest, and save the artefacts the failure produced.
- Separate infrastructure failures from test failures in the reporting, so "the registry timed out" does not read as "your code is broken".
- When bisecting, bisect on the pipeline, not on your laptop. The environment is part of the input (Change Correlation).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Contained by review and by later stages — unless the response was to re-run until green, in which case the defect proceeds with a green checkmark and nothing contains it.
What can go wrong
- The re-run reflex: enough intermittent failures that re-running becomes automatic and nobody notices when a real defect starts failing intermittently.
- Triage owned by nobody, so a trunk failure sits for hours while everyone assumes someone else is on it (The Ownership Record).
- Failure output that requires opening the CI UI, so people merge without reading it.
- A broken trunk fixed forward with a second untested change, producing two problems.
- Infrastructure failures auto-retried invisibly, hiding a degrading dependency until it fails completely.
- Bisect run against a moving base, so the result names a commit that is not actually responsible.
- "It passed on re-run, so it was a flake." It was intermittent. A real race condition is intermittent too, and it will be intermittent in production as well (Heisenbugs: The Bug That Leaves When You Look at It).
- "The failure is in a test I did not touch, so it is not mine." Untouched tests are exactly what catches unintended consequences. That is what they are for.
- "Trunk is red, so I should wait." You should check whether your change is implicated, and otherwise carry on — but never merge onto a red trunk, because your verdict is then meaningless.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Median time from a red trunk to a green trunk is measured, and the response is usually a revert rather than a fix.
- Failures are classified — code, flake, infrastructure, pre-existing — and the proportions are visible.
- Someone unfamiliar with a check can act on its failure message without opening the pipeline UI. Test this by asking them.
- Re-run counts per job are recorded, so the flake rate is a number rather than a feeling.
- Reverting the change is the default response to a trunk failure and needs no diagnosis to be correct — it restores a state that was known good.
- If a revert is not clean, that is information about the change: it was coupled to something else, or it carried a migration that cannot be undone by a git operation (Why Migrations Are the Dangerous Change).
- Rolling back the *pipeline* is also on the table: a failure that started with a runner image update or an action version bump is reverted the same way.
- Automate classification where the signal is mechanical: same commit failing then passing is a flake; a non-zero exit from a network step is probably infrastructure.
- Automate the trunk-health signal so it reaches people rather than waiting to be checked.
- Automate revert tooling — one command that reverts and opens the follow-up — so the cheap correct action is also the easy one.
- Do not automate blanket retries on test steps. Retrying an infrastructure step with backoff is reasonable; retrying an assertion until it passes deletes the signal (The Automation Trap).
- Revert-first is fast and occasionally reverts a change that was fine, wasting the author's time. That trade is almost always worth it, and it is a real cost.
- Rich failure output costs artefact storage and makes logs longer; the answer is structured summaries, not more logging.
- Automatic infrastructure retries improve the felt reliability of CI and hide the degradation trend unless the retries are counted.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- ORG-SPECIFICRevert-first on a broken trunk is a team policy, not a technical fact. Teams with very long pipelines or heavy release ceremony sometimes prefer fix-forward with a designated owner; what matters is that the rule exists and everyone knows it, not which rule it is.
- TOOL-SPECIFICRe-running only the failed jobs is available on GitHub Actions and GitLab and preserves the rest of the run; Jenkins typically re-runs the whole pipeline unless stages are explicitly restartable. That difference changes what re-running costs, and therefore how casually it gets used.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — writing assertions whose failure messages carry the diagnosis rather than just the verdict.