Failure-Aware Feature Design
Four questions that change the structure rather than adding a branch: what if the database is gone, the dependency is slow, the request arrives twice, and half the work already succeeded.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind survives until the requirement changes.
The happy path works. What does this feature do when the write fails, the dependency times out, the request repeats, or part of the work has already committed?
Pause shipped and produced three incidents in a month: a customer charged the night they paused, a customer paused for six months when they asked for three, and a pause that sent a confirmation email for a change that never committed.
Wrap the handler in try/catch, return 500 on error, and let the client retry. It handles every failure uniformly, which sounds like a virtue, and for a read it very nearly is.
A 500 tells the client nothing about whether the work happened. A timeout is indistinguishable from a slow success, so "retry on error" means "sometimes do it twice", and for a write that is a different feature (The Retry Is a Decision, Not a Reflex).
- A 500 tells the client nothing about whether the work happened. A timeout is indistinguishable from a slow success, so "retry on error" means "sometimes do it twice", and for a write that is a different feature (The Retry Is a Decision, Not a Reflex).
- Uniform handling erases the distinction that matters: a duplicate request should return the original result, an invalid one should be rejected forever, and a dependency failure should be retried. One catch block turns three responses into one (An Error Taxonomy That Survives Contact).
- The email is already sent when the transaction rolls back, because it went out before the commit and nothing can un-send it (The Dual Write Problem).
- As soon as one dependency is slow rather than down, the catch block never runs. The request sits holding a connection until the pool is empty, and the outage is now everywhere rather than in one endpoint (Timeouts).
- The retrofit is what costs. Making the command idempotent later means an id the client never sent, on requests already in flight, against rows already written (Designing the Happy Path Last).
What limits the solution, and what must never stop being true
This domain leads with these two. A design that ignores its constraints is not a design, and an invariant nobody named is one nothing is protecting.
- The nightly billing job cannot be made transactional with the pause command; they are different processes hours apart, and one of them serves four products.
- Email goes through a third-party API with a p99 of eleven seconds and no transactional guarantee of any kind.
- The client is a browser with a retry on network error, and a mobile app with a more aggressive one. Neither can be changed on your schedule.
- Adding a queue is available but is a new operational surface for a five-person team, and that cost is real (The Cost of Change).
- Three pause requests with the same intent produce one pause. Not approximately one — one (Idempotency by Design).
- No side effect visible to a customer happens for a state change that did not commit. An email about a pause that did not happen is worse than no email.
- If the system half-succeeds, it says so. The forbidden outcome is not partial failure; it is partial failure that reports success (Partial Failure).
- A paused subscription is never charged, including during the window where two processes disagree about what it is.
Who owns what, and where the seams fall
Responsibilities decide boundaries; boundaries decide what an interface has to say.
- The command owns idempotency: it decides what "the same request" means and stores enough to recognise the second one.
- The transaction boundary owns atomicity of state, and owns being drawn around state only — an HTTP call inside it is a decision to hold a lock across a network (External Calls Inside a Transaction).
- The outbox, or whatever plays its part, owns the fact that a side effect was promised, so the promise survives a crash between commit and send (The Transactional Outbox).
- The caller owns nothing about correctness. A design that requires clients to retry correctly has delegated an invariant to code you do not control.
- The idempotency boundary is the command, not the endpoint. The admin tool and the API both pause subscriptions, and if the key lives in the HTTP layer only one of them is protected (Error Boundaries).
- The transaction wraps state changes and nothing else. Email, webhooks and analytics live outside it and are triggered by the committed fact rather than by the intention.
- The distribution boundary is where in-process reasoning stops. The nightly job is a different process, so "the state is paused" is something it learns, not something it shares (What Changes at the Network Boundary).
The four questions
These four are not a checklist of things that can go wrong — there are hundreds of those. They are the four that change the *structure* of the code rather than adding a branch to it, which is why they belong before implementation and the other hundreds belong in a runbook.
The test for whether a failure question belongs here is simple: can it be answered by adding a branch to existing code, or does answering it require a new place for something to live? An id, a table, a boundary, a second deploy. If it needs a place, it has to be decided while there is still room to make one.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| The store is unavailable | The write fails after the email has already gone out, so the customer is told about a pause that does not exist. | The side effect was ordered before the commit, so the two can disagree in exactly one direction. | Commit state first; trigger effects from the committed fact. Accept at-least-once delivery as the price (The Transactional Outbox). |
| The dependency is slow, not down | Requests pile up holding connections; the pool empties; endpoints unrelated to pause start failing. | No deadline on the call, so "slow" is unbounded and the failure is shared across the whole service. | A timeout shorter than the caller's patience, plus a decided behaviour at the deadline (Timeouts: The Latency Contract Nobody Writes Down). |
| The request arrives twice | A three-month pause becomes six months, and nothing in the logs looks like an error. | The command is not idempotent, and the client retried a request whose outcome it never learned. | Client-supplied key, outcome stored against it before the work, repeat returns the stored outcome (Idempotency Keys: The Mechanism). |
| Half the work committed | State says paused; the entitlement service says active; support believes whichever one they checked first. | Two stores changed in one logical operation with no atomicity across them (The Dual Write Problem). | One writer plus an event, or an explicit saga with compensation. Never two writes and hope (Sagas: Trading Isolation for Availability). |
The states you find by asking
The failure questions produce states the happy path never needed. "Charging right now" is not a business concept anyone asked for; it exists because the nightly job and the pause command can overlap, and something has to be true during the overlap.
Making that state explicit is what converts a race into a guard. The alternative is not the absence of the state — it is the same state, unnamed, represented by whichever row the job happened to lock, and defended by nothing (Explicit State).
| From | On | To | Guard | Effect |
|---|---|---|---|---|
| active | nightly job selects | charging | renewal date is today | row locked for the duration of the charge |
| charging | charge settles | active | — | renewal date advanced |
| active | pause command | paused | pauses used this year < limit | PauseStarted event; audit row |
| charging | pause command | pause_pending | always — this is the overlap case | client gets 202 and an honest answer, not a silent loss |
| pause_pending | charge settles | paused | — | PauseStarted event emitted after the charge, not before |
| paused | resume, manual or scheduled | active | clock passed resume date, if one is set | — |
| paused | cancel | cancelled | — | — |
| active | cancel | cancelled | — | — |
- cancelled → paused — A cancelled subscription has no billing to suspend. Allowing it produces rows that entitlement reads as "has access, not billed", which is a free account created by a state transition nobody authorised.
- paused → charging — This transition existing at all is the original incident: it is what "the job selected its batch before the pause committed" looks like in the model. The guard belongs on selection, not on the charge.
- paused → paused — A self-transition here means a repeated request moved the resume date. Idempotency is what makes this transition unrepresentable rather than merely unwanted (Idempotency by Design).
- pause_pending → active — The customer asked to pause and the charge finishing is not a reason to forget that. The pending state exists precisely so the intent survives the charge.
Two of these five states came from the failure questions rather than from the requirement. That ratio is normal, and it is the reason failure analysis has to happen before the state model is fixed rather than after (Designing the Happy Path Last).
Retrofitted, and designed
These two do the same thing on a good day. The difference is where the decisions live: one has them scattered through a procedure, so each new failure adds a branch to a function that is already long; the other has them in a shape, so each new failure adds a case to a type (Result Types).
The important line in the second version is the one that stores the key before doing the work. Checking whether a pause already exists is the version most people write, and it races with itself: two concurrent copies both see nothing and both proceed.
async function pause(id: string, until?: Date) {
try {
const sub = await repo.find(id)
if (sub.paused) return // "idempotent"
await email.sendPauseConfirmation(sub)
sub.paused = true
sub.pauseUntil = until
await repo.save(sub)
} catch (e) {
logger.error(e)
throw new HttpError(500)
}
}type PauseResult =
| { ok: true; pausedUntil?: Date }
| { ok: false; reason: 'already_paused' | 'cancelled' | 'limit_reached' }
async function pause(cmd: PauseCommand, now: Clock): Promise<PauseResult> {
const prior = await idempotency.claim(cmd.key) // before the work
if (prior) return prior.result // same answer, always
const result = await tx(async (t) => {
const sub = await repo.find(cmd.id, t)
const r = sub.pause(cmd.until, now) // guards live here
if (r.ok) { await repo.save(sub, t); await outbox.add(t, PauseStarted(sub)) }
await idempotency.record(t, cmd.key, r)
return r
})
return result // email drains from outbox
}The first is not idempotent: the sub.paused check races with a concurrent copy of itself, and neither one is wrong at the moment it looks. It emails before committing, so a rollback leaves a customer informed of a change that did not happen. And it collapses "already paused", "cancelled" and "database is on fire" into one 500, so a client cannot tell a permanent rejection from a retryable one. The second stores the outcome against the key inside the same transaction as the work, which is what makes the repeat return the original answer rather than recomputing one, and it puts the email behind a committed fact so the only remaining failure is sending it twice.
How to build it
Most important first.
- Ask the four questions in order and write the answers down before the handler exists. What if the store is unavailable; what if the dependency is slow rather than down; what if this arrives twice; what if half of it committed.
- Make the command idempotent by construction. Take a client-supplied key, store the outcome against it, and return the stored outcome on a repeat — not "check if already paused", which races with itself (Idempotency by Design).
- Give every external call a timeout shorter than the caller's patience, and decide what happens at the deadline. An untimed call is an unbounded one, and unbounded is the failure mode that takes the whole service down (Timeouts).
- Move side effects outside the transaction and make them replayable from committed state. That converts "email sent twice" — survivable — into the only remaining failure, from "email sent for a pause that never happened" (Effect Boundaries).
- Choose, explicitly, what the system does when it cannot finish: fail closed and report, or complete forward with compensation. Both are valid; the failure is having neither, and discovering which one you built during an incident (Designing for Failure).
- Design the read path for disagreement. Two processes will see different states for some window; decide which one wins and make the other one's view harmless rather than assuming the window does not exist.
What the next change costs
The field this whole domain exists for. A structure is only better if it makes the change after this one cheaper — and it is worth saying which changes it does not help.
- Adding a fifth failure question later — "what if the customer is deleted mid-pause" — costs one row in the failure table and one guard, because the structure already has somewhere for it to go.
- Retrofitting idempotency onto a command that was not designed for it costs an API change, a client change you do not control, a new table, and a period where old and new clients coexist and only one is protected. That is roughly ten times the cost of deciding it on day one, and the multiplier is why this lesson sits before implementation rather than after.
- Adding a second side effect — a webhook alongside the email — is one more consumer of the committed event under this design, and one more thing inside the transaction under the naive one, where it also lengthens the lock.
- What does not get cheaper: the nightly job's window. Two processes disagreeing for a few seconds is a property of them being two processes, and no amount of structure inside your service removes it (The Network Changes Everything).
- Idempotency keys are a table, an expiry policy, a client contract and a class of subtle bugs of their own. For a genuinely idempotent operation — setting a value, not incrementing one — that machinery buys nothing and should not be built.
- Moving side effects outside the transaction converts exactly-once-looking behaviour into at-least-once, so the email can now arrive twice. That is a real regression in one dimension, chosen because the alternative failure is worse (Where You Put the Acknowledgement Decides Everything).
- Designing for these four failures makes the code visibly larger than the feature. On a low-traffic internal tool the failures are rare enough that manual repair is genuinely cheaper, and pretending otherwise is over-design (When Design Does Not Pay).
What can go wrong
- The idempotency key is stored after the work instead of before, so two concurrent copies of the same request both see no key and both proceed (Optimistic Concurrency: Versions and If-Match).
- The key has a TTL shorter than the client's retry schedule — twenty-four hours of retries against a one-hour key — so the guarantee lapses precisely when it is needed.
- Retries are added without backoff, and a dependency that was recovering is knocked over by its own clients (Retry Storms: The Load You Generated Yourself).
- Compensation is treated as rollback. Refunding a charge is a new event with its own failure modes, its own receipt and its own tax consequence — it does not restore the previous state, it adds to history (A Refund Is Not a Rollback).
- The mitigation fails too: an outbox is added, its drain job dies quietly on a Friday, and the system now loses side effects with total reliability while every state change looks perfect.
- The command depends on durable storage for its idempotency records, which must be as available as the command itself — an idempotency store in a cache that evicts under memory pressure provides exactly the guarantee it appears to remove (Idempotency Storage).
- It depends on the client sending a stable key across retries. That is a contract, it belongs in the API documentation, and it is the part most often assumed rather than specified.
- The email path depends on committed state and nothing else, which is the point of moving it: it can then be retried freely because it reads a fact rather than an intention.
- "So wrap everything in retries." Retries turn one failure into many and are only safe on operations that are idempotent — which is why idempotency comes first and retry second, not the other way round (Retries Are a Property of the Operation).
- "Use a transaction and the problem goes away." A transaction makes state changes atomic within one store. It does nothing about the email, the other service, or the client that never saw the response (Partial Failure).
- "This is distributed systems and we are a monolith." A browser retrying over the internet is already two parties and one unreliable channel. The failures arrive at a monolith unchanged (A Remote Call Is Not a Function Call).
- "Exactly-once delivery would solve this." What is achievable is at-least-once delivery plus idempotent processing, which produces exactly-once *effects*. The distinction is not pedantic — it decides where the work goes (Exactly-Once Is a Scope, Not a Guarantee).
- duplicate-knowledge
Testing it, and how it ages
- Send the same command twice with the same key, concurrently, and assert one pause and two identical responses. Sequentially is the easy case and it is not the one that breaks (Concurrency by Design).
- Fail the store after the side effect would have fired, and assert no email. This test is the reason the side effect moved, and it is the first one deleted when it turns flaky (Test Doubles, Precisely).
- Make the dependency slow rather than absent, and assert the timeout fires and the connection is released. Testing "down" and not "slow" is the most common gap in this area (Designing for Failure).
- Assert that a repeat after the idempotency record has expired is rejected rather than silently re-executed — the expiry policy is part of the contract and deserves an assertion.
- The first thing to change is the idempotency window, because it is a guess about client behaviour and clients get more aggressive over time, never less.
- Once three commands need this, it becomes shared machinery — a decorator, a middleware, a base command — and that is the right moment, not before, because you can see what actually varies (The Rule of Three).
- It stops being sufficient the moment a pause has to coordinate a change in another service. In-process atomicity no longer covers the operation, and the design moves to a saga with explicit compensation (Sagas: Trading Isolation for Availability).
Where this applies
This domain's advice is contested more than most. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view rather than a caricature.
- GENERALThat a timeout cannot distinguish a slow success from a failure is a property of message passing over an unreliable channel, so it holds for any language and any transport, including a local HTTP call to the same machine. What changes is how much the runtime hides it from you.
- SCALE-SPECIFICAt ten requests a day, duplicate pauses are a support ticket and manual repair is genuinely cheaper than the machinery. At ten thousand, the same rate of duplicates is a daily incident and the machinery is cheaper than the repair — the crossover is where manual correction stops fitting in someone's morning.
- CONTESTEDThe strongest opposing view is that most teams over-engineer this: real duplicate rates are low, most side effects are harmless when repeated, and the idempotency table plus outbox plus timeout policy is a permanent complexity tax paid against an event that would have cost an apology and a refund. That argument holds well for reversible, low-value operations, and fails badly for money, messages to third parties, and anything a regulator can ask about.
Where the depth lives
This domain teaches the codebase-level structure and hands the rest off.
- — Testing & Reliability Engineering — fault injection is how you find out whether any of this works, and running it is a discipline that domain owns.