Failure Injection
Once the store works, make it fail on purpose: a payment that times out, a product that runs out during checkout, a checkout submitted twice, a database that is not there. Each injection turns a failure you reasoned about into one you observed.
The situation, the reflex, and why it stalls
Every lesson starts where being stuck starts: someone has a problem, and the first move that comes to mind feels like progress.
You have a failure model on paper. How do you find out whether the system actually does what the paper says?
I wrote the failure table for checkout and implemented the responses. The tests pass. But every test I have runs to the end — none of them makes the provider hang or kills the process between steps — so I do not actually know whether the pending-order recovery works. I know that I wrote code intended to make it work.
Trust the unit tests. Each handler has a test: the timeout branch is tested with a mocked exception, the duplicate branch with a repeated call. The code paths are covered, and coverage is a number that goes up.
The mocked exception tests the branch and not the failure. A real timeout means the provider may have charged the card; the mock raises and moves on, so the test proves the code handles an exception it was told to expect, not that the system reaches the state the failure model promised.
- The mocked exception tests the branch and not the failure. A real timeout means the provider may have charged the card; the mock raises and moves on, so the test proves the code handles an exception it was told to expect, not that the system reaches the state the failure model promised.
- Nothing is ever killed. The partial-failure recovery is triggered by a process dying between two lines, and no unit test does that; the recovery job has been run only against fixtures that were written by the same person who wrote the job, with the same assumptions.
- The failures are tested one at a time, in isolation, against the handler. The store fails as a system: the timeout, the retry, the notification and the job interact, and the unit tests have no place for the interaction to happen.
The move
Precisely enough to apply it to a problem you have never seen — not a slogan.
- For each row of the failure model, build a way to cause it in the running system rather than in a mock: a fake provider that hangs, declines or drops responses on command; a product whose stock can be set to zero mid-checkout; a switch that makes the database connection fail; a harness that sends the same request twice at once (Failure Modeling).
- Before each injection, write the prediction: which state the order will be in, what the customer will see, what the recovery job will do and when. The prediction is the failure model's claim; the injection tests it, and a wrong prediction is the most useful outcome (Prediction Before Execution).
- Inject against the working store, one failure at a time, and observe the whole system — database, logs, provider fake, browser — not just the return value. Then inject pairs: timeout followed by notification; duplicate submit during a hang. The pairs are where the model is usually wrong.
- Keep the injections. They are the failure model in executable form, and the store from here on is only as understood as the injections it still passes (Invariants as Tests).
The injection order, and why it is one order
The sequence below injects the cheapest and most informative failures first and the interactions last, because the pairs only make sense once the singles are understood. It is not the only order: a team whose biggest fear is the database would invert it.
The capstone's later stages ask for exactly this list — a timed-out payment, an out-of-stock product, a duplicated checkout, an unavailable database — injected against the store built in the earlier stages.
- 1Duplicate checkout submit, two at once
because Needs no fake at all — just two requests — and tests the highest-stakes invariant first.
- 2Provider declines
because The fake's simplest behaviour, and it checks the failed-payment state and the kept cart before anything subtle.
- 3Provider hangs, then reports paid on query
because Exercises the pending state, the reference and the recovery job — the whole partial-failure design.
- 4Out of stock between validate and reserve
because Tests the reservation transaction and the customer message with no provider involved.
- 5Database unavailable at mark-paid
because Tests the notification handler's retry and the provider's redelivery together.
- 6Pairs: duplicate during hang; notification during job
because The interactions are where the model is wrong; run them once every single passes.
A provider you can make fail
The fake below is the whole harness for the provider boundary. It is deliberately small: a mode, a charge that behaves according to the mode, a query that answers from the record, and a way to deliver a notification. Nothing about the real provider's API surface is reproduced beyond what the failure model needs.
1fake provider:2 mode = one of: succeed, decline, hang, succeed-drop-response3 charges = map from reference to state4 5 charge(amount, reference):6 if mode == decline: charges[reference] = declined; return declined7 if mode == hang: wait forever8 charges[reference] = paid9 if mode == succeed-drop-response: drop the connection # charged, caller never hears10 return paid11 12 query(reference): return charges[reference] or absent13 14 deliver notification(reference, times = 1):15 repeat times: POST to the store's handler with reference and charges[reference]The "succeed-drop-response" mode is the one most fakes lack and the one that models the real incident: the card is charged and the caller gets nothing. Without it, the timeout injection tests the wrong thing.
What each injection is allowed to prove
The matrix records, for each injection, the claim it tests and the claim it does not — which is the thing a passing injection most tempts you to forget. A hang that resolves to paid proves the recovery job; it proves nothing about a hang that resolves to declined, which is a separate row and a separate run.
| Injection | Proves | Does not prove |
|---|---|---|
| Duplicate submit | One charge, one order, matching responses under concurrency | That a duplicate during a hang is handled — that is the pair |
| Decline | payment-failed state, reason shown, cart kept | Anything about timeouts; a decline is a clean answer |
| Hang → paid on query | pending state, reference stored, job completes forward | Hang → absent: the cancel-and-release path is its own injection |
| Out of stock mid-checkout | Reservation fails atomically and the customer learns which item | Behaviour under two carts racing for the last unit — needs two concurrent checkouts |
| Database down at mark-paid | Handler retries; redelivery completes the order | Database down at reserve — different transaction, different outcome |
How to do it
Most important first.
- Build the fake provider first. It needs to succeed, decline, hang, succeed-but-drop-the-response, and deliver notifications on command including twice. Everything else in the module depends on it (Treating External Systems as What They Are).
- Make each injection a switch you can flip from a test or a command line, not a code change. An injection that needs a redeploy will not be run.
- Write the prediction as the order's expected state and the expected customer-visible outcome, then run, then compare. Record the misses; they are the model corrections.
- Run the injections against the smallest working store, before scale, caching or a second warehouse exist — the point is to understand the failure, and a small system is easier to observe.
- Promote each injection into the test suite once it passes, so that a later change that breaks the recovery is caught.
Worked on a concrete problem
The move has to produce something. This is what it produced.
- Injection: provider hangs. Prediction: order goes pending with reference; customer sees "we are confirming your payment"; the job queries the fake, gets "paid", marks paid, sends email. Observed: order pending, customer saw the message — and the job never ran, because it was scheduled for a window that was measured from the wrong timestamp. A prediction miss, a bug found, and no customer involved.
- Injection: last unit of stock bought by another checkout while this one is between validate and reserve. Prediction: reservation fails, order cancelled, customer told the product sold out. Observed as predicted, and the message told the customer which item, which had not been specified anywhere and got decided in the moment.
- Injection: duplicate submit while the provider is hanging. Prediction: the second submit finds the reserved key and waits for or returns the first result. Observed: the second submit returned "no stored result yet" as an error, because the stored-result path assumed the first attempt had finished. The pair found what the singles did not.
- Injection: database unavailable at "mark paid" after the fake reports success. Prediction: the notification handler retries and the provider's redelivery completes it later. Observed as predicted, which is the boring outcome — and the one that turns the paper claim into an observed behaviour.
How you know it worked
What now exists that did not before, and what question you can now ask.
- Every row of the failure model has an injection that causes it against the running store, and a recorded prediction next to a recorded observation.
- At least one prediction was wrong, and the model or the code changed because of it. If every prediction was right, either the model was excellent or the injections were too gentle.
- The injections run in the test suite, so the next person who changes checkout finds out what they broke before a customer does.
The questions you can now ask
The field this whole domain exists for. After this lesson, these are the questions to put to an unfamiliar problem.
- ?For this failure in the model, what would cause it in the running system — and can I flip that from a test?
- ?What do I predict the order state, the customer view and the recovery will be — before I run it?
- ?Which two failures could overlap in practice, and what happens when they do?
- ?Which injections are worth keeping as permanent tests, and which were one-time questions?
What can go wrong
- The injection harness becomes the project. A fake provider that supports every feature of the real one is a second provider; it needs exactly the behaviours the failure model lists and nothing else.
- Injections are run once, at the end, as a gate, and discarded. Their value is in being re-run after every change to the path they exercise.
- Injections are run without predictions, so every outcome looks like information and none of it corrects anything. The prediction is what makes an injection an experiment rather than a demonstration.
- The fake provider and the switches are code to write and maintain, and they only pay off if the failure rows they exercise ever happen. For a prototype, the prediction exercise alone may be enough.
- Injections against a running system are slower than unit tests and can be flaky if the recovery involves timers; keeping them in the suite costs build time.
- "Injection replaces the failure model." It tests the model. Without the rows and the predictions, injection is just breaking things and watching, which finds bugs by luck rather than by design.
- "This is chaos engineering." Chaos engineering injects failures into production at scale to find unknown weaknesses; this is injecting known failures into a small working store to check known claims. The spirit is shared; the stage and stakes are not.
- "Mocks are bad." A mock that raises is fine for testing a branch. It is not evidence about the system's state after a real timeout, and that is the claim the failure model makes.
Where this applies
Problem-solving advice is stated as universal far more often than it is. These labels say what each method is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- STAGE-SPECIFICOn a working store with real payments ahead, the injections are the cheapest insurance available. On a throwaway prototype, write the predictions and skip the harness; on an existing production system, the harness usually has to be built around fakes at the boundary because the real provider cannot be made to hang on request.
- TEAM-SPECIFICA solo learner gets the most from injections done by hand with a fake and a kill command, watching the database; a team ships them as automated tests because the person who injects is rarely the person who changes the code next.
- ILLUSTRATIVEThe four injections and their outcomes — including the job scheduled from the wrong timestamp — are invented to show what an injection finds; a real store's misses will be its own.
Where the depth lives
This domain asks the question and hands the answer off by name.
- — Testing & Reliability, when it exists, owns fault-injection frameworks and chaos engineering in production; this lesson is the small, deliberate version against a store you can hold in your head.