Essential vs Accidental Complexity
Payments can fail: that is essential, and every checkout must handle it. Five frameworks for one checkout: that is accidental, and it was chosen. The distinction tells you which complexity to design for and which to remove — and it is not always as clear as the slogan suggests.
The situation, the reflex, and why it stalls
Every lesson starts where being stuck starts: someone has a problem, and the first move that comes to mind feels like progress.
The system is complicated. How do you tell which of the complexity is the problem's and which is yours — and what do you do differently about each?
Checkout has a frontend framework, a state library, a form library, a backend framework, an ORM, a job queue, a payment SDK, and a handler for the case where the payment fails after the order was created. A new engineer asks why it is so complicated, and you realise you cannot say which parts had to be there.
Simplify by removing whatever is most annoying — usually the failure handling, because it is the ugliest code — and keep the frameworks, because they are familiar. It feels like cleanup.
The failure handling was the essential part. Removing it makes checkout shorter and wrong: the payment that fails after the order exists now leaves a paid-looking order with no payment.
- The failure handling was the essential part. Removing it makes checkout shorter and wrong: the payment that fails after the order exists now leaves a paid-looking order with no payment.
- The frameworks stay because each was chosen for a reason that was true at the time; nobody re-asks whether the reason still holds, so accidental complexity is permanent and essential complexity is the thing that gets cut.
- "Don't over-engineer" is applied to the wrong axis. It was meant to stop the fifth framework; it is used to stop the failure branch.
- The new engineer's question is never answered, so they learn the complexity as if all of it were necessary, and add a sixth framework for the next feature.
The move
Precisely enough to apply it to a problem you have never seen — not a slogan.
- For each piece of complexity, ask: is this in the problem, or in our solution? A payment that can fail after the order exists is in the problem — any implementation, in any language, by any team, has to handle it. A state library is in the solution — a different team would make a different choice or none. The first is essential; the second is accidental.
- Test the classification by imagining the simplest possible implementation. If the complexity is still there — the failed payment still has to be handled in a shell script — it is essential. If it disappears — the shell script needs no ORM — it is accidental.
- Treat the two differently. Essential complexity is designed for: it gets the clearest code, the best tests, the failure table. Accidental complexity is budgeted: each piece must earn its place against the problem it solves, and it is removed when it stops earning it (The Complexity Ledger).
- Be honest that the boundary moves. Idempotent webhook handling is essential given the provider you chose and accidental relative to a provider that never retries — so some accidental complexity is essential once a decision is made. Record which decision made it so, because that is the decision to revisit if the complexity becomes unaffordable.
Checkout, sorted
The feature's complexity in two columns, with the sentence that justifies each placement and the shell-script test applied. The last row is the boundary-moving case: essential given a decision, and the decision is named.
| Piece | In the problem or our solution? | Shell-script test | Treatment |
|---|---|---|---|
| Payment can fail after order exists | Problem — essential | A script would still face it | Design for it: state machine, test, failure row |
| Confirmation may arrive twice | Essential given this provider | A script would still face it with this provider | Handle; record "because provider retries" |
| Stock can run out mid-checkout | Problem — essential | A script would still face it | Design for it: enforcement point, test |
| State library for a three-field form | Solution — accidental | A script has no state library | Ledger: solves nothing here; remove |
| Form library | Solution — accidental | Same | Ledger: remove; revisit at many fields |
| ORM for two tables | Solution — accidental | Same | Ledger: stays for consistency with the rest of the store |
| Job queue, one job | Solution — accidental, earning its place | A script would still need "send later" | Ledger: simplest thing that meets the async requirement |
The essential branches, designed for
Essential complexity gets the failure table, because that is what "designed for" means. These are the rows a simplification must never delete; they are also the rows the reflex would have deleted first.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Payment fails after the order row exists | An order with no payment, visible to the customer as placed | Order creation and payment are two steps and the second can fail | Order state "pending payment"; a timeout that releases stock; a clear message — see What If Payment Fails?. |
| Confirmation arrives twice | Order marked paid twice; a duplicate email; a double stock decrement | The provider retries until acknowledged | Deduplicate on event id before any side effect. |
| Last unit sold between add-to-cart and payment | A paid order that cannot be fulfilled | Stock is checked at cart time and changed by others | Enforce at order creation, in the database; tell the customer before payment. |
| Browser closed after Pay, before confirmation | Customer unsure whether they paid; may pay again | The user's view and the system's state diverge | Confirmation from the backend's state, not the browser's; the same order is shown on return. |
The slogan, made precise
"Don't over-engineer" is repeated because it is sometimes right, and it is dangerous because it does not say which axis. The comparison makes it falsifiable: the precise form can be wrong in a specific way, which the slogan could not.
"Don't over-engineer checkout." Applied to whatever is longest — the failure handling — because it looks like the most engineering.
"Add no accidental complexity ahead of a problem it solves, and never remove essential complexity to look simpler. In checkout: the state library goes; the failed-payment branch stays and gets a test."
The precise form names the axis. It can be checked — is the removed thing accidental? is the kept thing essential? — and it can be wrong, which means it can be argued with. The slogan cannot be wrong, so it cannot be right either.
How to do it
Most important first.
- List the components and the branches of the feature. For each, write "in the problem" or "in our solution" and one sentence of justification.
- Run the shell-script test on anything marked essential: would the crudest implementation still need it? If not, reclassify.
- For everything accidental, write what problem it solves here and what would happen if it were removed. If the answer is "nothing", remove it; if "we would write it ourselves", keep it and say so.
- For everything essential, check that it has a test and a failure-table row. Essential complexity that is untested is the most dangerous kind, because "simplifying" will target it.
- Make the slogan precise: "don't over-engineer" means "do not add accidental complexity ahead of a problem", and it never means "skip the essential branch" (What Cannot Be Simplified).
Worked on a concrete problem
The move has to produce something. This is what it produced.
- Checkout, sorted. Essential: the payment can fail, succeed late, or be confirmed twice; stock can run out between add-to-cart and payment; the customer can close the browser mid-way. Each is in the problem — a shell script would face it. Accidental: the state library (the form has three fields), the form library (the same three fields), the ORM for two tables, the job queue with one job type. Each was a choice; a smaller choice exists for each.
- The removal. The state and form libraries went — three fields do not need them, and the accidental complexity was a second thing to learn for anyone touching checkout. The ORM stayed with a note: "for two tables it is not earning its place; it stays because the rest of the store uses it and a second data-access style would be worse". The job queue stayed because the email must be sent asynchronously — The Why Ladder had already established that — and it is the simplest thing that does so.
- The new engineer's question, answered: "Three branches are the problem's; nobody can remove them. Two libraries were ours and are gone. Two stay for reasons written in the ledger. If you want to add a library, add a row." The next feature was built with one framework fewer.
How you know it worked
What now exists that did not before, and what question you can now ask.
- Every piece of complexity has a label and a sentence; the essential ones have tests and the accidental ones have ledger rows.
- A "simplify" proposal can be evaluated: it names accidental complexity and what problem stops being solved, or it names essential complexity and is rejected.
- The shell-script test has moved at least one thing between categories.
- The boundary-moving cases are recorded with the decision that moved them.
The questions you can now ask
The field this whole domain exists for. After this lesson, these are the questions to put to an unfamiliar problem.
- ?Is this complexity in the problem or in our solution — would the crudest implementation still face it?
- ?For each accidental piece: what does it solve here, and what happens if it goes?
- ?For each essential piece: is it tested, and does it have a failure-table row?
- ?Which decision of ours turned this accidental complexity into essential complexity, and is it worth revisiting?
What can go wrong
- Everything is essential. The engineer who chose the frameworks defends each as required, and the ledger has no removable rows. The shell-script test is the antidote: if a cruder implementation would not need it, it was a choice.
- Everything is accidental. The failure branches are called over-engineering because they are long, and checkout is simplified into incorrectness. The test is the same in reverse.
- The classification is done once and the boundary is frozen. When the provider is swapped for one that never retries, the idempotency handling becomes accidental — and stays, because nobody re-ran the sort.
- The move is used to win an argument about a framework rather than to understand a system. "That is accidental" is a classification, not a verdict; the ledger row says whether it earns its place.
- The sort takes an afternoon and produces removals that someone chose. Removing a colleague's framework is a conversation, not a commit.
- Accidental complexity that is removed sometimes comes back when the problem grows: the form library was unnecessary for three fields and necessary at twenty. The ledger row should say at what point.
- Designing essential complexity properly is more work than the ugly branch that was there. It is also the work; the branch was the debt.
- "Frameworks are accidental complexity, so use none." Frameworks are accidental complexity that solve accidental problems — routing, rendering, data access — which every implementation has. Use the ones that earn their row; the failure is the fifth one, not the first.
- "Essential complexity cannot be reduced." It cannot be removed; it can be isolated, named and tested so that it is the clearest code in the system rather than the ugliest.
- "Accidental means bad." Accidental means chosen. Most of a working system is chosen, and the ledger is how choices stay accountable (The Complexity Ledger).
Where this applies
Problem-solving advice is stated as universal far more often than it is. These labels say what each method is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERAL"In the problem or in our solution?" applies to any system; the shell-script test works for a compiler, a pipeline or a UI.
- CONTESTEDThe strongest opposing view holds that the essential/accidental line is too unstable to be useful: what is essential depends on the requirements, which change, and on the platform, which was chosen — so "essential" is only "accidental complexity we have stopped questioning", and the sort gives false confidence that the remaining complexity is untouchable. The reply is that the line moves and is still worth drawing, provided the decision that moved it is recorded; the disagreement is about whether the record is kept in practice.
- ILLUSTRATIVEThe five frameworks, the three-field form and the new engineer are invented to show the sort; the classification of any real system has to be argued from its own requirements.
Where the depth lives
This domain asks the question and hands the answer off by name.