Invariant Leaks
The rule is enforced in the service. Then a background job, an admin tool or a migration writes straight to the table. This is the characteristic failure of the whole module, and it is an ownership failure before it is a technical one.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind survives until the requirement changes.
The rule is correctly implemented and the data is still wrong. Where did the guarantee go?
Nobody asked for this. It is what happens eighteen months after a careful team enforced "a subscription must have an active payment method" in SubscriptionService, and a support tool, a churn-recovery job and three migrations later, forty accounts are active with no payment method and nobody knows since when.
The service enforces it. Anything writing around the service is a mistake by whoever wrote it, and the fix is to tell them to use the service.
It is true and it does not work, which is the uncomfortable part. Each bypass was locally reasonable — the support tool predates the service, the migration has to write directly by nature, the growth team did not know the service existed — and telling people to be more careful does not survive turnover.
- It is true and it does not work, which is the uncomfortable part. Each bypass was locally reasonable — the support tool predates the service, the migration has to write directly by nature, the growth team did not know the service existed — and telling people to be more careful does not survive turnover.
- The rule is unenforceable against writers you do not control, and "we do not control them" is a fact about the organisation that the design has to absorb rather than object to (Code Ownership).
- Because the enforcement is invisible from the table, there is no signal at the moment of the bypass. The violation is discovered by a failed charge, a support ticket or an auditor — always downstream, always weeks later (Debuggability by Design).
- And the remediation is a data problem, not a code problem: forty accounts, no record of when each became invalid, and no way to tell an intentional exception from a bug. That asymmetry — cheap to prevent, expensive to remediate — is what makes leaks worth designing against specifically.
What limits the solution, and what must never stop being true
This domain leads with these two. A design that ignores its constraints is not a design, and an invariant nobody named is one nothing is protecting.
- The support tool is a low-code internal product with direct database credentials, used daily by people who are not engineers, and it cannot be made to call a service this quarter.
- Migrations run as a superuser by design, because they have to be able to alter schema.
- The churn-recovery job was written by a growth team in a different repository against the same database.
- An active subscription always has a usable payment method.
- If that is not enforceable on every path, then at minimum every violation is detected within a bounded time and attributed to the writer that caused it.
- The set of writers to a table is known, and adding to it is a decision somebody makes rather than a thing that happens.
Who owns what, and where the seams fall
Responsibilities decide boundaries; boundaries decide what an interface has to say.
- Someone owns the *table*, not just the service. Ownership of state means the list of writers is known and adding one requires their agreement, which is an organisational mechanism with a technical enforcement (State Ownership).
- The migration process owns re-asserting invariants after it runs, because migrations are the one writer that legitimately must be able to bypass everything.
- Whoever adds a writer owns bringing it through the owner, or owns arguing for the exception explicitly — including the reconciliation that will now be needed.
- The boundary that matters is the one around *write access to the state*, and it is enforced by database grants far more effectively than by code review. A boundary that only exists in the module diagram is not a boundary (Architecture Boundaries).
- Where a writer genuinely cannot be brought inside — the low-code tool — the boundary moves down to the constraint, which is the only line the tool also crosses (Where Invariants Live).
- Migrations sit outside every boundary by design. That is not a flaw to be fixed; it is a category that needs its own discipline (Data Migration).
The unit that believes it owns the invariant
Run a responsibility map on the service and the leak becomes obvious in one line, because the map asks what the unit *depends on* and what it *changes when* — and neither of those questions has an answer that mentions the three other writers.
The verdict is the point. This is not a badly written service; it is a well-written service holding a guarantee it is not in a position to make.
- — What makes a subscription valid
- — That an active subscription needs a usable payment method
- — How to transition between subscription states
- — Validates every transition it is asked to perform
- — Writes to the subscriptions table
- — Emits events for downstream consumers
- — Rejects activation without a payment method
- — The subscriptions table
- — The payment method repository
- — A clock
- — The subscription lifecycle changes
- — The definition of a usable payment method changes
Two reasons to change and a clean design — and the map is still hiding the problem, which is the lesson. Nothing in dependsOn or changesWhen mentions the support tool, the churn-recovery job or the migrations, because the service does not depend on them; *they* depend on the table. The responsibility map is drawn from the code, and the leak is a writer the code cannot see. The fix is to map the state rather than the service: for subscriptions, list every credential with write access, and the answer is four, of which one is this service.
How a leak opens, step by step
No step here is a mistake in isolation, and that is what makes the sequence worth reading. Each is a reasonable decision by someone with incomplete information, and the guarantee is gone by step four without anyone having decided to remove it.
- 1The service is built
Enforces the rule at every transition. Genuinely correct, well tested, reviewed.
fails by Nothing yet. The guarantee is real because the service is the only writer.
- 2The support tool is connected
A low-code tool gets database credentials so support can fix customer problems without an engineer.
fails by The tool predates nothing and knows nothing; it was connected by an ops ticket and never crossed an engineer's desk. The guarantee is now conditional and nobody has been told.
- 3A migration runs
Backfills a column across two million subscriptions, as superuser, with constraints deferred for speed.
fails by A subset is left in a state the service would have rejected. The migration succeeded, so nothing reports a problem (Data Migration).
- 4Another team writes a job
A churn-recovery job in another repository reactivates lapsed subscriptions to offer a discount.
fails by It reactivates accounts whose payment method has since been deleted. It is correct against its own requirements and it does not know yours (Dependency Cycles is the code-level version of this discovery problem).
- 5Someone notices
A failed charge, a support ticket, or an auditor asks a question.
fails by By now the data has been invalid for months, the cause is one of three writers, and there is no timestamp that distinguishes them. Remediation is archaeology.
The step to intervene at is the second, and the intervention is not technical: it is that connecting a new writer to a table with invariants is a decision somebody has to make. Every step after that is cheaper to prevent than to detect and much cheaper to detect than to remediate.
The smell that says an invariant has already leaked
You can usually find leaks before an incident does, because they leave a signature: code that fixes up state rather than producing it, written defensively by someone who has learned not to trust the data.
As always the smell is a question. Defensive code against genuinely untrusted input is correct; defensive code against your own system's state is a report that the guarantee is not being kept.
looks like Reads that repair: const method = sub.paymentMethod ?? await findAnyMethod(sub.customerId). Null checks on fields the schema says are required. A comment reading "shouldn't happen but does". A nightly job whose name contains fix, cleanup or sync. Filters like WHERE payment_method_id IS NOT NULL in reports that are supposed to cover all active subscriptions.
suggests Somebody has already met data that violates the invariant and has coped with it locally rather than reporting it. Each such site both hides the leak and makes it harder to close, because the invalid data now has code depending on being tolerated.
fix Do not delete the check — find out how the state got there. Add logging that records the violating row and its writer, let it run for a week, and you will have the leak. Then close it at a point all writers cross, remediate the existing rows deliberately, and only then remove the defensive code — in that order, because removing it first turns a silent violation into an outage (Where Invariants Live).
How to build it
Most important first.
- Enumerate writers from the database side, not from the code side: list grants, then list who holds each credential. This takes an hour and reliably finds writers nobody in the room knew about.
- Push the guarantee down to a point all of them cross, which for a shared table means a constraint (Enforcing Invariants).
- Revoke what you can. A reporting pipeline with a read-only role cannot leak an invariant, and downgrading it is usually a five-minute change that nobody has ever proposed (Least Privilege as a Design Decision).
- Make the remaining bypasses explicit and monitored: name them, and add a reconciliation that detects violations with an owner and an alert.
- Add an invariant re-assertion step to the migration process itself, so a migration that leaves the data invalid fails the deploy rather than completing (Designing the Migration).
What the next change costs
The field this whole domain exists for. A structure is only better if it makes the change after this one cheaper — and it is worth saying which changes it does not help.
- With the leak open: every new writer is a coin flip, and the cost of the next incident is a data investigation whose size depends on how long it went unnoticed. That cost is unbounded and unbudgetable, which is why it never appears in a plan.
- With the guarantee pushed to a constraint: a new writer costs one failed test run and a conversation. Changing the rule costs a migration on a large table plus a code change — call it a day and a maintenance window.
- With the leak named and reconciled instead of closed: a new writer costs nothing at write time and shows up in the reconciliation within its window, so the cost is bounded at "one alert plus a fix". That is a genuinely acceptable design where closing the leak is not available, and it should be chosen deliberately rather than defaulted into.
- What closing the leak costs going forward: the support team can no longer fix an urgent customer problem by editing a row, so every such fix now needs an engineer or a purpose-built tool. That is a real reduction in operational flexibility, it will be felt on a bad day, and it is the honest price of the guarantee.
- Closing leaks with grants and constraints reduces operational flexibility at exactly the moment flexibility is most wanted — during an incident — and teams that have not felt that yet tend to underrate it.
- Detection instead of prevention keeps the flexibility and accepts a window during which the invariant is false, which some domains and some regulators will not accept.
- Auditing writers is unglamorous, recurring work with no visible output when it finds nothing, which is precisely why it stops happening.
What can go wrong
- The constraint is added and a later migration disables it "temporarily" to load data, and it is never re-enabled. This is by a distance the most common way a fixed leak reopens.
- The reconciliation exists and its alert is routed to a channel nobody reads, so detection is nominally in place and functionally absent.
- The bypass is documented in a wiki rather than in the code, so the next engineer reading
SubscriptionServicestill believes the rule is guaranteed (Documentation Decay). - The team responds by forbidding direct database access entirely, the support team loses the ability to fix customer problems, and an unofficial workaround appears that is worse than what it replaced.
- The reconciliation itself becomes a writer — auto-correcting violations — and now silently destroys the evidence of the bug that caused them.
- A constraint-based guarantee depends on the migration process not disabling it, which is a dependency on a human procedure and should be automated into the pipeline.
- A reconciliation depends on being able to distinguish a violation from a legitimate exception, which usually requires a field recording *why* — a design cost the prevention approach does not have.
- Revoking write access creates a dependency of the tool team on your service's roadmap, and that is a real organisational cost that has to be paid rather than wished away (Dependency Direction).
- "This is a discipline problem." It is an access problem wearing discipline's clothes. Discipline does not survive turnover, deadlines or a support engineer with an angry customer on the phone (Tone, Disagreement and Receiving Review).
- "Lock the database down and the problem is solved." Partly, and it creates a new one: the paths people used to fix things are gone and the replacement usually does not exist yet. Closing a leak has an operational cost that has to be paid at the same time (Least Privilege as a Design Decision).
- "The migration is a special case, so it does not count." Migrations are the single most common cause of bulk invariant violation, precisely because they bypass everything and run against production data with no application code in the loop.
- "If we had used a proper domain model this would not happen." A domain model constrains code that uses it. The leak is by definition the writer that does not (The Anemic Domain Model and this lesson are arguing about the same case from opposite ends).
- shotgun-surgery
- god-object
Testing it, and how it ages
- Test the constraint by writing directly to the table with the same credentials the support tool uses. If that write succeeds, the guarantee does not exist for that writer (Where a Test Must Be Real).
- Test that the schema still has the constraint, as an assertion in CI, so a migration that drops it fails the build (Characterization Tests).
- Test the reconciliation on seeded violations, including the case where zero violations exist — a reconciliation that only works when it finds something is a common and undetectable bug.
- Add a test that enumerates database roles with write access to the table and fails when the list changes, so a new grant is a review conversation rather than a discovery (What to Automate Out of Review).
- Leaks accumulate with organisational growth, not with code age. Every new team, every acquired tool and every "quick script" is a candidate writer, and the rate is roughly proportional to how many people have credentials.
- They are also self-concealing: the longer a leak has been open, the more the invalid data looks like normal data, and the harder it becomes to argue for closing it because "we have always had accounts like that".
- The design that survives is the one where the guarantee is at the storage layer and the writer list is asserted in CI. Everything above that erodes on a timescale of about two years.
Where this applies
This domain's advice is contested more than most. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view rather than a caricature.
- GENERALAny state reachable by more than one writer has this failure available to it; what varies is how many writers there are, which is an organisational fact rather than a technical one.
- SCALE-SPECIFICWith one team and one deployable, code review really does catch new writers and the service-level guarantee is close to real. The failure appears at the point where a second team, an off-the-shelf tool or an outsourced integration acquires credentials — which is usually well before anyone thinks of themselves as a large organisation.
- CONTESTEDThe strongest opposing view: locking state behind a single owner is how you get a service that becomes a bottleneck for every adjacent team, and organisations that do this end up with shadow databases and CSV exports — a far worse outcome than a leak. On this account the right answer is to make the invariant cheap to satisfy and easy to discover rather than to make the state unreachable. That is a serious position and it is right about the failure mode of over-locking; the counter is that it works only where the invariant is genuinely soft, and "an active subscription has a payment method" is not soft — it is a promise to charge someone.
Where the depth lives
This domain teaches the codebase-level structure and hands the rest off.
- — Testing & Reliability Engineering — continuous verification of invariants against production data, which is the discipline that turns a leak from an eighteen-month secret into a fifteen-minute alert.