ReliabilityGENERALDOMAIN-SPECIFICCONTESTED

Designing for Failure

A function call either returns or throws. A remote call has a third outcome — you do not know — and an interface designed without a name for it will be wrong in a way no amount of error handling fixes.

Constraints & invariants Who owns what What the next change costs

The requirement, the obvious build, and why it breaks

Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind survives until the requirement changes.

The question

What does an interface have to look like once the call can time out, be retried, or half-succeed?

The requirement

Checkout calls charge(order) and shows a confirmation. Once the payment provider moved behind a network, one customer in four hundred is charged twice and one in two thousand sees an error for a payment that actually succeeded.

The obvious build

Wrap the call in try/catch. On an exception, show an error and let the customer try again. It worked for years when the payment library was in-process, and the code is the same code.

Why it breaks

In-process, an exception means it did not happen. Over a network, an exception means you did not hear — and "did not happen" and "did not hear" require opposite responses. The catch block cannot distinguish them, so whichever it assumes is wrong some of the time (Partial Failure).

How it breaks as requirements change

In-process, an exception means it did not happen. Over a network, an exception means you did not hear — and "did not happen" and "did not hear" require opposite responses. The catch block cannot distinguish them, so whichever it assumes is wrong some of the time (Partial Failure).
Asking the customer to try again is a retry with a human in the loop, and it is the least safe retry available: no request id, no bound, and a person who will click three times.
A timeout is not a failure signal at all. The request may still be in flight; the provider may charge sixty seconds later, after your transaction rolled back and your logs say the payment failed (Retries Are a Property of the Operation).
The result type is the real problem. Receipt has no value meaning "unknown", so the code is forced to pick success or failure at exactly the moment neither is known, and everything downstream inherits the guess.

Requirement→Constraints→Invariants→Responsibilities→Boundaries→Interfaces→State→Dependencies→Failure→Implementation→Tests→Feedback→Evolution

What limits the solution, and what must never stop being true

This domain leads with these two. A design that ignores its constraints is not a design, and an invariant nobody named is one nothing is protecting.

Constraints

The provider is a third party; you cannot change its API and cannot make it faster.
The customer is waiting on a page, so the request has a human-scale deadline of a few seconds.
Double charges are a regulatory and reputational problem, not merely a bug.
The existing signature charge(order): Receipt is called from six places, so changing it is a real edit.

Invariants

A customer is charged at most once per checkout attempt, regardless of how many times any part of the system runs (Idempotency by Design).
The system's record of whether a payment happened must eventually agree with the provider's record. Disagreement is not a display bug; it is money.
No interface may report success for work that did not complete, and none may report failure for work that did.

Who owns what, and where the seams fall

Responsibilities decide boundaries; boundaries decide what an interface has to say.

Responsibilities

The caller owns supplying an identity for the attempt, because only the caller knows whether this is a new intent or a repeat of one (Idempotency by Design).
The boundary owns classifying outcomes into at least three — succeeded, definitely did not happen, unknown — and never collapsing the third into either of the others (An Error Taxonomy That Survives Contact).
Something owns resolving unknowns: a reconciliation job that asks the provider what actually happened, because unknown is a state you exit by asking, not by waiting.
The domain owns the invariant. "At most one charge per checkout" is a business rule, and it must be enforced by a stored constraint rather than by hopeful control flow (Enforcing Invariants).

Boundaries

The seam is the place where the failure model changes. On one side a call either returns or throws; on the other it has a third outcome, and the boundary's job is to stop that third outcome leaking untyped into domain code (Boundary Adapters).
The boundary is also where the deadline lives. A caller that cannot say how long it is prepared to wait has delegated that decision to a socket default, which is usually minutes (What Changes at the Network Boundary).
Draw it around the operation, not around the client library. PaymentGateway.charge is a boundary; HttpClient.post is a transport (Anti-Corruption Layer).

The third outcome

The whole lesson fits in one observation: an in-process call has two outcomes and a remote call has three. Local failure is informative — the exception means the work did not happen. Remote failure is not: a timeout tells you nothing about the server, only about your patience.

Every bug in this lesson comes from a codebase whose types can express two outcomes trying to represent three. The code is forced to guess, and it guesses consistently, so the errors are systematically biased in one direction — either double charges or phantom failures, depending on which branch the catch chose.

Unknown is not an error. It is a state with an owner and an exit, and an interface without a name for it will map it onto one of the other two (Explicit State).
The exit requires an attempt id the provider also knows, which is why the id belongs in the signature (Idempotency by Design).
If the provider offers no way to ask, Unknown is permanent and the design must tolerate it — usually by making the operation safe to repeat instead (Retries Are a Property of the Operation).

What a caller actually knows after a remote call

UserLLMAgentToolDataDecisionHumanGuardrail

Two signatures, priced

The argument for putting failure in the interface is not that it is more rigorous. It is that the alternative makes a specific, common change enormously more expensive — and that change is not "handle errors better", it is "add a second provider", which every payments team does eventually.

Add a second payment provider, and make the flow retry-safe

The change

Introduce a fallback provider for cards the primary declines, and make the whole flow safe to retry after a timeout.

`charge(order): Receipt`, throws on failure

CheckoutServicePaymentClientOrderRepositoryRefundServiceAdminRechargeRenewalJobWebhookHandler

testscheckout_testrefund_testrenewal_testwebhook_testadmin_test

7 modules · 5 test files

Failure semantics live in seven catch blocks that each interpret an SDK-specific exception hierarchy. Nothing has an attempt id, so every call site must be found, given one, and have that id threaded down from wherever the intent originates — which for renewals is a job that has already forgotten it. The double-charge risk moves rather than disappearing.

`charge(attemptId, order): Charged | Declined | Unknown`

PaymentGateway (one new adapter)Reconciler (one new query)

testspayment_gateway_contract_testreconciler_test

2 modules · 2 test files

The second provider implements an outcome type that already exists. No caller changes, because no caller ever handled provider-specific failure — they handled three cases, and there are still three. Retry safety was already there.

what it cost Every call site is longer: three branches instead of a try/catch, and the Unknown branch has to say something to a waiting customer, which is a product decision someone now has to make rather than avoid. The attempt id has to be threaded from wherever intent is formed, which means the renewal job needs a durable id it did not previously have — a real schema change. And the reconciler is a new operational component with its own alerting, its own on-call surface, and its own capacity to be wrong.

What each failure actually demands

Grouping failures as "errors" is the mistake. They differ in exactly the property that decides your response: whether the work may have happened. That single bit determines whether retrying is safe, whether compensation is needed, and whether the user should be told anything at all.

The failure modes of one remote call

Trigger	Symptom	Cause	Response
Connection refused before the request was sent	Immediate error	The work definitely did not happen — this is the only genuinely informative failure	Retry freely; it is safe even without idempotency (Retries Are a Property of the Operation)
Timeout waiting for a response	Nothing, for a configured duration	The work may have completed. The request, the processing and the response are three separate places it could have been lost	Do not guess. Record `Unknown`, retry with the same attempt id, reconcile
Provider returns 500	An error with a body	Ambiguous by default — some providers fail atomically, some do not, and the status code does not say	Treat as unknown unless the provider documents atomicity, and write down which you assumed (Architecture Decision Records)
Provider returns 200 but the connection drops mid-body	A parse error on a successful call	Success that looks like a transport failure — the most misleading case there is	Idempotent retry resolves it correctly; a non-idempotent one double-charges
Your process crashes after the call and before the write	No record of a charge that happened	The remote effect and the local record are not in one transaction and cannot be	Persist intent before the call so restart can find and resolve it (The Transactional Outbox in Backend)
The provider is slow, not down	Requests queue, threads block, everything degrades	A deadline that is longer than the caller's patience turns a dependency's latency into your outage	Bound the wait and shed load; slow is a failure mode (What Changes at the Network Boundary)

How to build it

Most important first.

Put the attempt id in the signature. charge(attemptId, order) makes retrying safe by construction; no discipline elsewhere can make charge(order) safe (Idempotency by Design).
Return a three-valued outcome. Charged | Declined | Unknown forces every caller to decide what to do when it does not know, at compile time, instead of discovering the case in production (Result Types).
Persist intent before acting. Write "attempting payment for checkout X with id Y" in your own store, then call. If you crash mid-call the record is the thing that lets you find out what happened (Explicit State).
Give every remote call an explicit deadline and derive it from the caller's remaining budget rather than a constant (What Changes at the Network Boundary).
Make the unknown state visible in the model. Payment status is PENDING | SUCCEEDED | FAILED | UNKNOWN, and UNKNOWN has an owner and an exit path (State Machines).
Design the failure path with the same care as the happy path, at the same time. Retrofitting it means retrofitting it into a structure with no room for it (Designing the Happy Path Last).

What the next change costs

The field this whole domain exists for. A structure is only better if it makes the change after this one cheaper — and it is worth saying which changes it does not help.

Cost of the next change

Under the naive design, "add a second payment provider" costs re-deriving the failure semantics from scratch for the new provider, because none of the old ones were written down — they lived in the shape of a try/catch. Expect a fortnight and a fresh class of double charges.
Under the designed one, the same change costs one new adapter implementing an outcome type that already exists, plus its reconciliation query. The failure model is stated once and reused (Strategy).
The next change to *retry policy* — different backoff, a budget, a circuit breaker — is contained at the boundary and touches no domain code, because retry safety is already a property of the operation (Retries Are a Property of the Operation).
What stays expensive: the attempt id must be threaded from wherever intent originates, so introducing it later means touching every call site and every stored record. That is the retrofit §170 warns about, and its cost is why the parameter belongs in the signature from the start.

What the recommended approach costs

Three-valued outcomes make every call site longer and noisier. For a call whose failure genuinely does not matter — a metrics ping — that ceremony is pure cost and a plain try/catch is right.
Persisting intent before acting adds a write to the hot path, which costs latency on every request to protect against a failure in a small fraction of them.
Reconciliation is a second system that must itself be correct, monitored and idempotent. You have traded an unhandled case for an operational component, which is a better trade and not a free one.

What can go wrong

Failure modes

The three-valued outcome is added and callers pattern-match Unknown to the failure branch, which is the original bug with more ceremony and a false sense of rigour.
The attempt id is generated inside the retry loop, so each retry has a new id and the provider sees three distinct payments. This is the single most common way idempotency is implemented incorrectly.
Reconciliation is written and never scheduled, so unknowns accumulate silently until a finance report finds them a quarter later.
Intent is persisted in the same transaction as other work, and the transaction rolls back after the remote call has already succeeded — so the money moved and the record did not (The Dual Write Problem in Backend).

Dependencies, and their direction

The domain depends on an outcome type, not on the HTTP client, the SDK or its exception hierarchy — that inversion is what keeps a provider swap from touching business rules (Dependency Inversion).
Reconciliation depends on the provider offering a way to ask "what happened to attempt Y". If it does not, that is a fact about your design space you must discover before shipping, not after (Volatile Dependencies).
Every caller now depends on being able to produce a stable attempt id, which pushes id generation up to wherever the intent is first formed (Stable Identifiers).

Misreads

"Retries fix this." Retries multiply the problem unless the operation is safe to repeat. A retry on a non-idempotent charge is a design decision to sometimes charge twice (Retries Are a Property of the Operation).
"Use a transaction." A local transaction cannot include a remote side effect. The database rolls back and the money does not (Partial Failure).
"Exactly-once delivery solves it." There is no exactly-once over an unreliable network; there is at-least-once delivery plus idempotent processing, which produces exactly-once *effects*. The distinction is the whole design (Idempotency by Design).
"This is over-engineering for a payment we make twice a day." Frequency is not the variable — consequence is. Twice a day for two years is a thousand chances to charge someone twice (When Design Does Not Pay).

Smells this explains

swallowed-errors

Testing it, and how it ages

What to test, and at which boundary

Test the unknown branch explicitly: inject a timeout, assert the payment lands in UNKNOWN and that reconciliation resolves it. If nothing exercises this path it does not work (Failure-Aware Feature Design).
Test that a repeated attempt id charges once, against the provider's sandbox rather than a double, because idempotency is a property of the provider you are trusting (Contract Tests).
Test the crash point: kill the process between persisting intent and the remote call, restart, assert exactly one charge (Where a Test Must Be Real).
Assert the invariant, never the calls: "one charge exists for this checkout" survives every refactor of how charging happens (Mocking).

How this design ages

The three-valued outcome spreads. Once one boundary has it, every other remote boundary looks obviously wrong without it, and that spread is the design paying off (An Error Taxonomy That Survives Contact).
Reconciliation grows into a general mechanism — one job resolving unknowns across several providers — and becomes one of the more valuable pieces of the system.
It ages badly if the provider adds asynchronous settlement, at which point "charged" itself becomes provisional and the state machine needs another state. That is a genuine model change, not a bug (Resources Have State Machines in API Design).

Where this applies

This domain's advice is contested more than most. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view rather than a caricature.

GENERALThat a call which may or may not have been received admits three outcomes rather than two follows from the medium, so it holds for HTTP, gRPC, queues, database calls over a socket and any RPC mechanism regardless of language.
DOMAIN-SPECIFICFor money, inventory and anything legally consequential, an unresolved unknown is unacceptable and reconciliation is mandatory. For an analytics event or a cache warm, the correct design is to drop it and move on — the same three outcomes exist, and two of them are genuinely fine to ignore, which changes the whole cost calculation.
CONTESTEDA strong opposing position holds that three-valued outcome types are ceremony that most teams will not sustain: callers pattern-match Unknown into the error branch anyway, so you have paid for a type and bought nothing, and a simpler design — retry aggressively, make the provider idempotent, reconcile nightly — gets the same result with far less code. That is right whenever the provider genuinely supports idempotency keys and the business tolerates a nightly window, which covers a great deal of ordinary commerce. It is wrong when the customer is looking at a screen and needs an answer now, because then someone has to decide what to show for "unknown" and the type is what forces that decision to be made deliberately.

Where the depth lives

This domain teaches the codebase-level structure and hands the rest off.

API Designidempotency retryability error-model

Backendidempotency-keys transactional-outbox timeouts dual-write-problem

Distributed Systemspartial-failure timeout-ambiguity the-network-changes-everything crash-vs-slow failure-models

Domains that do not exist yet

— Testing & Reliability Engineering — fault injection, chaos experiments, error budgets and how to run a reconciliation process in production are that domain's subject. This lesson only covers what the failure model does to a signature.
— System Design — whether this call should cross a network at all, and what redundancy sits behind it, is decided before any of this and changes which failures are even possible.