ResearchGENERALDOMAIN-SPECIFICILLUSTRATIVE

Guarantees and Failure Modes

Read a technology for what it promises and what happens when the promise cannot be kept. "Fast" and "reliable" are not guarantees; "a write acknowledged is durable across restart" is, and "the most recent writes may be lost on crash" is the failure mode that goes with it. Your design depends on which one you assumed.

The move Worked example Next questions

The situation, the reflex, and why it stalls

Every lesson starts where being stuck starts: someone has a problem, and the first move that comes to mind feels like progress.

The question

A technology says it is fast, reliable and scalable. What are the actual guarantees hiding behind those words, and what does your system do when each one fails?

The situation

The store has adopted a shared cache for product data and a queue for order emails. Both were described as reliable. On a Tuesday the cache restarts and every product page shows an old price for a minute; on Wednesday the queue restarts and a batch of confirmation emails is sent twice. Neither behaviour was in anyone's mental model, and both were in the documentation.

The reflex

Trust the adjectives. "Reliable" on the home page, "battle-tested" in the blog post, and a large user base feel like a guarantee, and reading the persistence page feels like an operational detail for later.

Why it stalls

The adjectives were true and said nothing. "Reliable" described the software's quality, not its contract; what it guaranteed after a restart was a configuration choice the store never made, so the default made it.

What the reflex produces — and fails to produce

The adjectives were true and said nothing. "Reliable" described the software's quality, not its contract; what it guaranteed after a restart was a configuration choice the store never made, so the default made it.
The design was built on an assumed guarantee — that a queued job runs once — that the queue never offered. The email handler was not written to be safe under redelivery, because nobody knew redelivery was the contract.
When the failure arrived, it was treated as a bug in the technology rather than as its documented behaviour, and the fix was a restart rather than a design change, so it happened again.
The guarantees that were needed were never written down, so there was nothing to check the technology against, and there is still nothing to check the next one against.

Problem→Understand→Requirements→Constraints→Unknowns→Decomposition→Smallest Step→Model→Experiment→Observe→Debug→Learn→Iterate

The move

Precisely enough to apply it to a problem you have never seen — not a slogan.

Separate what the technology guarantees from what it is good at. A guarantee is a statement of the form "if X, then Y, always" — a write acknowledged is on disk; a single-key operation is atomic; a message is delivered at least once. Everything else is a property, and properties are not something your design can depend on.
For each guarantee, find its failure mode: the case the guarantee explicitly does not cover, and what the technology does there. At-least-once delivery has the failure mode "duplicates"; configurable persistence has the failure mode "the most recent writes may be lost"; a cache has the failure mode "the copy may be stale". The failure mode is not a defect; it is the other half of the contract.
Write your system's needs in the same form before reading: "a confirmation email must be sent, and must not be sent twice"; "a product page may show a stale price for up to a stated window". Then match need to guarantee. Where the technology's guarantee is weaker than the need, the gap is yours to close in your code — an idempotency key, a version check — and where it is stronger, you are paying for something you do not use.
Confirm the guarantee you depend on with an experiment, because the documented contract describes the default configuration and a version, and yours may differ. Restart it; kill it mid-operation; fill it; partition it. Each is an hour and finds the failure mode before a customer does (Failure Injection).

Needs, guarantees, failure modes, gaps

The matrix is the whole method in one table: the store's needs in the form that can be matched, the guarantee each technology actually offers, the failure mode that is the other half of that guarantee, and the mechanism in the store that closes the gap where the need is stronger. A row with no gap is a need the technology meets; a row with a gap and no mechanism is a production incident with a date not yet chosen.

Need (store)	Guarantee (technology)	Failure mode	Gap closed by
Confirmation email must be sent	Queue: an acknowledged job is delivered at least once, retried on worker failure	A job can be lost if the broker loses its state before delivery	Persistence on; the order records "confirmation queued" so a lost job is detectable
Confirmation email must not be sent twice	Queue: none — at least once means duplicates are possible	Redelivery after a crash mid-job or a lost acknowledgement	Handler records "sent" for the order id transactionally before sending; skips if present
Product page may show a stale price for up to the accepted window	Cache: a key is served until expiry or delete	Stale until expiry; all keys gone on restart without persistence	Expiry set to the window; cache miss falls back to the database
Product page must never show a removed product	Cache: none — expiry is time-based, not event-based	A removed product is served until its key expires	Removal deletes the key in the same code path as the database write
Product page must stay up when the cache restarts	Cache: none — a restart is a miss storm	Every request goes to the database at once; the database may fall over	Fallback with a short per-key lock so one request refills while the rest wait

The failure modes, as a customer sees them

A failure mode written as "the cache restarts" is an operator's sentence. Written as what a customer sees, it becomes something the business can accept or reject, and something the store can be designed around. Every row below is the same failure, translated.

Contract failures on the store

Trigger	Symptom	Cause	Response
Worker crashes mid-job	A customer receives the confirmation email twice	At-least-once delivery redelivered the job; the handler was not idempotent.	Record "sent" before sending, keyed by order; skip on the second run.
Cache restarts with persistence off	Product pages are slow or erroring for a minute	Every request missed at once and the database took the full load.	Fallback with a per-key lock; consider persistence if the window is unacceptable.
Hot product's key expires	A brief slowdown on the most popular page, every expiry	Stampede: many concurrent misses refill the same key.	The same per-key lock, or refresh the key before expiry (Cache Stampede: Everyone Misses at Once).
Admin removes a product	The product is still visible and orderable until its key expires	Expiry is time-based; nothing told the cache about the removal.	Delete the key in the removal code path; make the order path re-check the database.
Broker loses state before delivery	A customer never receives a confirmation	Persistence was off; the job existed only in memory.	Persistence on; the order carries a "confirmation queued" flag so the gap is detectable and re-queued.

The experiment that confirms the contract

The documentation describes a default and a version. The experiment describes your deployment. The script below is the queue's at-least-once contract, checked in an afternoon: predict, run, observe. The prediction is written first so that a surprise is recognisable as one.

Checking at-least-once, and the gap-closing handler

1# Prediction: killing the worker mid-handler causes redelivery;
2# with the idempotency record, the second run is a no-op.
3 
4enqueue(send_confirmation, order_id = 42)
5start worker
6  handler(order_id):
7    in transaction:
8      if confirmation_sent(order_id): log "skip"; return
9      mark confirmation_sent(order_id)
10    send_email(order_id)          # <- kill the worker here, once
11    log "sent"
12 
13observe:
14  run 1: marked, killed before send      -> email not sent, record exists
15  run 2: record exists                   -> "skip"; email never sent!
16 
17# Surprise: the record was written before the send succeeded, so a crash
18# between mark and send loses the email. The guarantee needed is
19# "sent, not twice"; the mechanism must mark AFTER a successful send and
20# tolerate a duplicate send in the crash window — or send, then mark, and
21# accept a rare duplicate. Choose, and write down which.

The experiment found that the first gap-closing design traded duplicates for losses. That is the kind of thing only running it finds, and it is why the contract is confirmed by observation rather than by reading.

How to do it

Most important first.

Write each of your needs as "must", "must not" or "may, for up to". That form has a matching guarantee or it does not; adjectives do not (What Must Never Break).
Find the page in the documentation where the words persistence, durability, consistency, delivery, ordering or atomicity appear. That is the guarantees page, whatever it is called. Read it before the features.
For every guarantee, write its failure mode next to it in one sentence: "at least once → duplicates are possible". If the docs do not say what happens in the failure case, that is a finding, and an experiment.
Match needs to guarantees in a table. Where the need is stronger than the guarantee, write the mechanism in your code that closes the gap (Duplicate Requests).
Run the restart, the mid-operation kill and the full-memory experiments against the actual configuration you will deploy, and record what was observed rather than what was documented (Experiment Design).
Put the guarantees you depend on into a test or a monitor so that a configuration change that weakens them is noticed (Invariants as Tests).

Worked on a concrete problem

The move has to produce something. This is what it produced.

The queue. Need: an order confirmation email is sent, and not sent twice. Documented guarantee: a job acknowledged by the broker is delivered to a worker at least once; a job is retried after a worker fails to acknowledge within a timeout. Failure mode: the same job can run twice — after a worker crash mid-job, after a network blip on acknowledgement, after a broker restart with persistence on. The need is stronger than the guarantee. The gap is closed in the store: the email handler records the order id as "confirmation sent" in a transaction before sending, and skips if the record exists. Experiment: kill the worker mid-handler; the job ran again; the second run skipped. Now the Wednesday duplicates are impossible by design rather than absent by luck.
The cache. Need: the product page may show a stale price for up to a window the founder accepted, and must never show a price for a product that has been removed. Documented guarantee: a key is served until its expiry or until deleted; on restart with persistence off, all keys are gone and reads miss. Failure modes: a stale value until expiry; a stampede when a hot key expires and every request goes to the database at once; on restart, a minute of every request missing. The first is within the need; the second and third were not in anyone's model. Gap closed in the store: product removal deletes the key in the same code path that updates the database, and the cache read falls back to the database on a miss with a short lock so the stampede is bounded. The Tuesday stale minute was the accepted window working as designed; the outage-shaped miss storm on restart was the unconsidered failure mode, and it now has a fallback.

How you know it worked

What now exists that did not before, and what question you can now ask.

Every need your design has is written as must, must not, or may-for-up-to, and each has a matching guarantee or a mechanism in your code that closes the gap.
You can name the failure mode of every guarantee you depend on in one sentence.
You have restarted, killed and filled the technology in its deployed configuration and written down what was observed.
A configuration change that weakens a guarantee you rely on would be caught by a test or a monitor.

The questions you can now ask

The field this whole domain exists for. After this lesson, these are the questions to put to an unfamiliar problem.

Next questions

?What does my system need, stated as must, must not, or may-for-up-to?
?What does this technology guarantee in the form "if X then Y, always" — and where in its documentation does it say so?
?What is the failure mode of each guarantee — the case it explicitly does not cover, and what happens there?
?Where my need is stronger than the guarantee, what mechanism in my code closes the gap?
?What did I observe when I restarted it, killed it mid-operation and filled it, in the configuration I will actually deploy?

What can go wrong

How the move itself fails

Guarantee maximalism: needs written as "must" everywhere, so every component is required to be durable, ordered and exactly-once, and the design becomes a distributed transaction for an email. Most needs are "may, for up to", and writing them honestly is what makes the simple option available.
The guarantees page is read and the experiment is skipped, so the design depends on the documented default and the deployment uses a different one.
The failure mode is found and treated as a reason to reject the technology, when the gap is a few lines in the store. At-least-once with an idempotent handler is a fine design; the failure mode is closed, not avoided.
The needs are matched to guarantees once, at adoption, and never again; a later configuration change or version upgrade weakens the guarantee silently. Guarantees you depend on are invariants, and invariants need tests.

What the move costs

Writing needs precisely surfaces disagreements — is a stale price for a minute acceptable? — that adjectives would have left comfortable, and settling them takes a conversation with the business.
Closing a gap in your code means owning the mechanism: the idempotency record is yours to design, store and clean up.
The experiments cost an environment where you can restart and kill things safely, which a team without a staging copy of the technology does not have on day one.

Misreads

"A technology with weak guarantees is a bad technology." Weak guarantees are usually why it is fast. The question is whether the guarantee matches the need, and a cache that promised durability would be a database.
"If the docs say it is reliable, it is." Reliable is a property of the software's quality. The contract is the guarantees page, and the two are unrelated: reliable software with at-least-once delivery reliably delivers duplicates.
"Exactly-once would solve this." Exactly-once between two systems is a coordination problem, not a setting; what most systems that claim it provide is at-least-once plus deduplication, which is the mechanism this lesson asks you to write (Where You Put the Acknowledgement Decides Everything).

Where this applies

Problem-solving advice is stated as universal far more often than it is. These labels say what each method is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.

GENERALNeed as must/must-not/may, guarantee as if-then-always, failure mode as the uncovered case applies to any dependency — a database, a queue, a cache, a payment provider, a cloud storage service, a library's API contract.
DOMAIN-SPECIFICFor a payments system most needs are genuinely "must not" and the gap-closing mechanisms are the core of the product; for an analytics dashboard most needs are "may, for up to" and a lost event is a rounding error. The same method produces very different designs, and the difference is in the needs column, not the technology.
ILLUSTRATIVEThe Tuesday restart, the Wednesday duplicates and the observed experiment results are invented to show the shape of the argument; the guarantees described are the general shape of a cache and an at-least-once queue, not any product's specification.

Where the depth lives

This domain asks the question and hands the answer off by name.

Backendjob-idempotency cache-ttl cache-stampede

Databasecache-invalidation

Distributed Systemsdelivery-semantics durability-and-recovery consistency-models idempotent-operations

Further

— The manifesto at /manifesto/delegating: a queue delegates retry; deciding whether a duplicate or a loss is the acceptable failure stays yours, and the experiment above is how you find out you have not decided yet.