Continuous Integration
Merging everyone's work into a shared mainline often enough that divergence stays small, and proving the merged result actually works.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What does continuous integration commit you to, beyond having a pipeline that runs on push?
Several people change the same codebase at once. Every hour their work stays apart, the chance grows that two changes are individually correct and jointly broken — and nothing has looked at the combination.
We have CI. A pipeline runs the test suite on every pull request and we do not merge red. That is continuous integration.
A green pipeline on a branch proves the branch is consistent with itself. It says nothing about the branch merged with the eleven commits that landed on trunk since it was cut.
- A green pipeline on a branch proves the branch is consistent with itself. It says nothing about the branch merged with the eleven commits that landed on trunk since it was cut.
- Textual merge conflicts are the easy half. The dangerous half is semantic: your branch renames a function, mine adds a caller. Git merges both cleanly, and the result does not compile — or worse, compiles and behaves differently.
- A two-week branch that was green every day integrates in one enormous event at the end, which is exactly the integration risk CI was invented to remove.
- Once the pipeline is the definition, teams optimise for the pipeline being green rather than for the mainline being releasable. Those diverge the moment anyone starts skipping checks to unblock a release.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The word doing the work is continuous, and it modifies *integration*, not *testing*. The practice is merging into a shared trunk frequently — at least daily per engineer — so that no two working copies are ever far apart.
- The pipeline is the proof mechanism, not the practice. It answers "is the integrated state good?" quickly enough that the answer is useful.
- Which state gets verified is the whole design question. Verifying the branch tests something nobody will ship. Verifying the merge result tests what will actually exist on trunk. The gap between those two is where "it was green on my PR" outages come from.
- Systems close that gap in three ways: test the merge commit rather than the branch head, serialise merges through a queue that re-verifies each candidate against the current trunk, or accept the gap and catch it with a post-merge run on trunk.
- Any of the three works. Not knowing which one you have is what does not work.
Integration is the merge, not the pipeline
Two teams can run identical pipelines and have completely different integration risk, because risk lives in how long work stays apart rather than in what runs when it comes back together.
The comparison below is not about discipline. It is about how many unverified combinations exist at any moment. In the first shape there are three; in the second there is one, briefly.
trunk ─────────────────────────────● feature-a ├── 9 days ──────────┘ feature-b ├── 6 days ───────┘ feature-c ├── 12 days ──────────┘ each branch green the whole time combined state: never built until merge day
trunk ─●─●─●─●─●─●─●─●─●─●─●─●─●─●─● each ● = someone's work integrated each ● = verified against the trunk it lands on combined state: rebuilt continuously divergence: hours, not weeks
The first shape has three pairwise combinations and one triple that no build has ever seen. The second never has more than one unverified combination, and it exists for minutes. The test suite is the same in both; the number of untested states is not.
What a green build actually claims
Every CI verdict is a claim about a specific commit in a specific environment with a specific set of checks. Reading it as a broader claim is how teams end up surprised by things CI never looked at.
The right habit is to be able to complete the sentence: "green means that *this* state passed *these* checks on *that* environment." Everything outside those three slots is unverified.
| A green build says | True | Not established |
|---|---|---|
| This commit compiled | On the CI runner's toolchain | On any other toolchain version |
| These tests passed | The ones that ran | The ones skipped, quarantined or not written |
| Nothing regressed | In behaviour the tests assert | In behaviour nothing asserts — performance, cost, layout |
| It integrates | With the base at the time the check ran | With whatever landed since (Required Checks) |
| It will deploy | Nothing about deployment was tested unless a deploy job ran | Config, migrations, capacity, dependency behaviour (Why Local Success Predicts So Little) |
| It is safe to ship | No — CI checks the artefact, not the rollout | Blast radius, coexistence, rollback (Version Coexistence: N and N+1, in Both Directions) |
Where the merge gets verified
There are three places to prove the integrated state is good, and they trade compute against the size of the window in which trunk can be broken. None is correct in general; the mistake is not knowing which one you are relying on.
Between "this change passed" and "trunk is good", where does the proof happen?
when Low merge rate, small team, cheap revert. Simplest to reason about and to configure.
cost The merge result is never built before it exists. Two independently green changes can break trunk, and the person who finds out is the next one to pull.
when You want the common case covered without serialising merges. This is the default shape on most forges.
cost The check tests the merge against the base *at check time*. If the base moves and the check is not re-run, the verdict is stale — a real and frequently misunderstood race.
when Merge rate is high enough that the base reliably moves during a pipeline run, and a broken trunk is expensive.
cost Serial re-verification, so throughput is capped by pipeline duration unless the tool speculates on batches. Speculation means a failure can invalidate several candidates at once and the queue has to unwind.
when As a backstop under any of the above, and as the only realistic option for checks too slow to gate a PR.
cost Trunk is broken between the merge and the failure. Only acceptable if the response — usually automatic revert — is fast and rehearsed.
How to do it properly
Most important first.
- Merge to trunk at least daily. If a change cannot be finished in a day, put the unfinished part behind a flag rather than behind a branch (Trunk-Based Development, Feature Flags: Deploy Is Not Release).
- Verify the merged state, not the branch head. Know which of the three mechanisms above your CI system uses, because the default differs by tool.
- Make the checks that define "integrated" required and enforced by the forge, not by convention (Required Checks, Protected Branches).
- Keep changes small enough that the verdict is unambiguous — a red build on a 40-line diff names its own cause (Change Size: Why Small Changes Are Safer, and When They Are Not).
- Treat a red trunk as an emergency with a defined response, because every other engineer is now blocked or, worse, building on top of it.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A broken trunk blocks or corrupts every engineer's work at once; containment is revert speed and the fact that it has not shipped yet.
What can go wrong
- Branch-only verification: green PRs that combine into a red trunk, discovered by whoever merges next.
- A merge queue that re-verifies serially and becomes the bottleneck at high merge rates, so people start bypassing it.
- Required checks that grew until nobody can land a one-line fix during an incident, and the emergency bypass becomes the normal path.
- A test suite that is the definition of integrated but does not actually exercise integration — every module mocked, so cross-module breakage passes.
- "We run tests on every PR, so we do CI." You have automated testing. CI is about merge frequency; the tests are how you survive it.
- "CI means the build is always green." It means breakage is small, visible and quickly reverted. A trunk that is never red is often a trunk nobody merges to.
- "Long-lived branches are fine if we rebase often." Rebasing pulls trunk into your branch. It does not push your branch into anyone else's working copy, so everyone else is still integrating blind (Long-Lived Branches).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Trunk is green almost all of the time, and when it is red you can say for how long and who is fixing it.
- The median branch age at merge is a day or less, measured rather than asserted.
- You can name which artefact CI verified: the branch head, the merge commit, or the post-merge trunk commit.
- The last three trunk breakages were each caused by one identifiable change, not by "something in this batch".
- Revert first, diagnose second. A revert restores a known-good trunk for everyone else in one commit; a forward fix under pressure is a second unverified change on top of a broken state.
- If the change cannot be reverted cleanly — because it carried a migration or another change built on it — that is itself the signal that it was too large to have been merged as one unit (Change Size: Why Small Changes Are Safer, and When They Are Not).
- Automate the merge-and-verify sequence completely. A human deciding when to rebase and re-run is a human introducing variance into the one process that must be identical every time.
- Automate trunk-health signalling: if trunk is red, everyone should know without asking.
- Do not automate the decision to bypass a required check. That is a judgement about risk, and it should leave a record naming the person who made it (The Audit Trail).
- Frequent integration means shipping incomplete work behind flags, which adds flag state to reason about and to clean up later.
- Verifying the merge result costs more compute than verifying the branch, and re-verifying in a queue costs more still — you pay linearly in runner time for a serialisation guarantee.
- Short-lived branches reduce integration risk and reduce the amount of review context available in one place. Reviewers see slices rather than a whole feature.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe practice — merge often, verify the merged state — is independent of language and CI system. What differs is which state your tool verifies by default.
- TOOL-SPECIFICGitHub pull request checks run against a merge commit of the PR into the base branch, which is recomputed when the base moves but is not necessarily re-run; GitLab offers merged-results pipelines and merge trains as opt-ins; Gerrit verifies each change against the current tip at submit time. The word "green" means a different thing in each.
- SCALE-SPECIFICBelow a few merges a day, verifying the branch head and letting trunk catch the rest is entirely adequate. Above roughly a merge every pipeline duration, a queue stops being optional because trunk moves faster than a pipeline can finish.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — what a test suite can establish about a merged state, and what it structurally cannot.