The Why Ladder
"We need Kafka" → why? → async events → why? → checkout should not wait for the email. The real requirement is "send the email asynchronously", and a job table meets it. Kafka may be overkill — and the ladder also says when it is not.
The situation, the reflex, and why it stalls
Every lesson starts where being stuck starts: someone has a problem, and the first move that comes to mind feels like progress.
Someone says "we need Kafka". How many times do you ask why, what do you do with the answer at the bottom, and how do you know when the original claim was right?
A colleague has proposed Kafka for the store's order events. They have used it before and are confident. You have read about it and are not. The meeting is in an hour and "I don't think we need it" is not an argument.
Argue about Kafka. Read up on its operational cost, find the blog post about a team that regretted it, and bring counter-examples. It feels like preparation, and it makes the meeting about the tool.
A debate about Kafka is won by whoever knows Kafka better, which is the colleague. The requirement is never discussed, so the outcome is decided by expertise in the solution rather than by the problem.
- A debate about Kafka is won by whoever knows Kafka better, which is the colleague. The requirement is never discussed, so the outcome is decided by expertise in the solution rather than by the problem.
- Counter-examples from other teams are about their requirements. Your colleague can reasonably reply that this store is different, and they are right — nobody has said how.
- If you win, the store has no queue and the email still blocks checkout; if you lose, it has Kafka and a consumer to run. Neither outcome was chosen for a reason that can be written down.
- The next proposal gets the same treatment, and the team's architecture becomes a record of who argued best.
The move
Precisely enough to apply it to a problem you have never seen — not a slogan.
- Ask why, of the claim, and write the answer. Then ask why of the answer. Each rung replaces a solution with the problem it was solving, and you stop when the answer is a property the system must have that no technology is named in — the real requirement. Typically three rungs; rarely more than five; below the requirement is the business, and you do not climb into that.
- At the bottom, write the real requirement as a sentence someone could test. "Checkout completes without waiting for the confirmation email" can be tested with a slow email server. "We need async events" cannot.
- Name the simplest thing that meets the requirement. Not the simplest thing you can imagine — the simplest thing that meets *that* sentence, with its failure modes handled. A job row written in the same transaction as the order, and a worker that sends the email and marks the row done, meets it.
- Then name the case where the claim was right. Kafka is the simplest thing that meets some requirements: several independent consumers of the same ordered event stream, replay from a point in time, throughput beyond what a table-backed queue handles. If any of those is in the real requirement, the ladder ends with Kafka — and now everyone can say why (Add Complexity Only When Required).
Kafka, climbed
The ladder from §29, with every required part. Read justifiedWhen as carefully as simpler: the device without it is "big tools are bad", which is a slogan, and slogans are what the module is replacing.
“We need Kafka for order events.”
- ↓Why Kafka? We need asynchronous order events.
- ↓Why asynchronous? Checkout should not wait for the confirmation email to be sent.
- ↓Why not? The email provider is slow and sometimes unavailable, and the customer must see confirmation and the order must be durable regardless.
the claim was right when Several independent consumers each need the ordered stream of order events — fulfilment, analytics, recommendations — one of them needs replay from a point in time, or the event volume exceeds what a polled table handles without contention. Then a log-based stream is the simplest thing that meets the requirement, and the outbox becomes its producer.
The simpler thing, drawn
The outbox is simple to describe and has more parts than the sketch: a transaction, a worker, a retry, an alert. The diagram shows all of them, because a "simpler thing" drawn without its failure handling is not comparable to the tool it replaces.
Choosing between the simpler thing and the claim
The ladder ends with two candidates and a condition. The decision is which to build now, and it is decided by whether justifiedWhen is true, expected, or neither — not by which is more interesting.
The requirement is written. Which meets it now?
when One consumer (email); no replay; volume a polled table handles. justifiedWhen is neither true nor expected.
cost A table, a worker, a retry policy, a stuck-row alert. If consumers multiply later, the outbox becomes a producer — a migration, not a rewrite.
when justifiedWhen is expected within the horizon you plan for — the analytics team has asked.
cost Everything above plus a stream to operate; design the event shape now so the table can feed it later.
when justifiedWhen is true today: multiple consumers, replay, or volume — with evidence.
cost A new system to run, monitor and understand; the ladder says so, and now the team can say why it is worth it.
How to do it
Most important first.
- Do the ladder in writing, with the person who made the claim, before the meeting if possible. It is a shared tool, not a rebuttal.
- Stop climbing at the first rung that names a testable property of the system. Write it as a sentence with no technology in it.
- Write the simplest thing that meets that sentence, including how it fails — a job table needs a retry policy and a stuck-job check, and those are part of "simplest".
- Write
justifiedWhenhonestly: what would have to be true for the original claim to be the simplest thing? Then check whether it is true, or expected to become true, with evidence. - Record the outcome — requirement, simpler thing, justified-when — so the next time the claim arrives, the ladder is read rather than re-climbed (The Decision Journal).
Worked on a concrete problem
The move has to produce something. This is what it produced.
- "We need Kafka." Why? "For asynchronous order events." Why asynchronous? "Checkout should not wait for the confirmation email to send." Why not? "Because the email provider is slow and sometimes down, and the customer should see confirmation regardless." Real requirement: checkout completes and the order is durable regardless of the email provider; the email is sent eventually, at least once. Simpler: an outbox row in the order's transaction and a worker. Justified when: other systems — analytics, fulfilment, a recommendation service — each need the full ordered stream of order events independently, and one of them needs to replay last month.
- The meeting, run on the ladder instead of on Kafka: the colleague agreed with every rung, added that the analytics team had asked for order events last quarter, and the team discovered that "justified when" was closer than either side thought. The decision was the outbox now, with the event shape designed so that a stream could consume the same table later (Reversible vs Irreversible Decisions).
- "We need a search engine for products." Why? "Search by name is slow." Why? "It scans the whole product table." Real requirement: name search returns within tolerance on the current catalog. Simpler: an index on the name column, or the database's built-in text search. Justified when: fuzzy matching, ranking across many fields, and a catalog too large for the database's text search — none of which V1 has.
How you know it worked
What now exists that did not before, and what question you can now ask.
- The bottom rung is a sentence with no technology in it that could be tested with a fault — a slow email server, a large table.
- The simplest thing is written with its failure handling, not as a sketch.
justifiedWhenis written and has been checked against evidence, not against intuition — and sometimes it comes out true.- The person who made the claim agrees with the ladder, because it was built with them.
The questions you can now ask
The field this whole domain exists for. After this lesson, these are the questions to put to an unfamiliar problem.
- ?Why this — and why that — until the answer is a property of the system with no technology in it?
- ?What is the simplest thing that meets that property, failure handling included?
- ?What would have to be true for the original claim to be the simplest thing, and is it true now?
- ?What evidence would tell us that "justified when" has arrived?
What can go wrong
- The ladder is climbed to prove the claim wrong. Rungs are chosen to lead away from Kafka;
justifiedWhenis written as a straw man. The device becomes an argument with extra steps and the colleague stops participating. - Climbing past the requirement. "Why should the customer see confirmation immediately?" — "so they trust the store" — "why?" The business is not the system; stop at the system property.
- The simpler thing is sketched without its failure modes, so it looks simpler than it is. A job table with no retry and no stuck-job check is not simpler than Kafka; it is unfinished.
- The ladder is run once and the conclusion is frozen. "Justified when" was false in March and true in September when the analytics team arrived, and nobody re-read it.
- The ladder takes a conversation the colleague may not want to have; done badly it reads as distrust. Done with them, it is the fastest way to agree.
- The simpler thing is sometimes a migration later. An outbox table that later has to become a stream is a migration that choosing the stream at the start would have avoided — but only if the start had needed the stream, which is what the ladder is there to find out.
- Writing
justifiedWhencommits you to revisiting. That is a maintenance cost on the decision, and most teams do not pay it.
- "The ladder always ends in the simpler thing." It ends in the requirement; the simpler thing and the justified case are both written, and evidence picks. The worked example ended closer to Kafka than the questioner expected.
- "Five whys." The count is not the point; stopping at the testable system property is. Three is common; a claim that survives one why was already a requirement.
- "This is about big tools." "We need a React state library" and "we need a microservice for this" climb the same ladder. The device is about claims, not about size.
Where this applies
Problem-solving advice is stated as universal far more often than it is. These labels say what each method is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALAny claim that names a solution — a tool, a pattern, a team structure — can be climbed. The rung count varies; the stopping rule is the same.
- CONTESTEDThe strongest opposing view: on a team that already runs Kafka, the "simpler thing" is Kafka, because a table-backed outbox is a second queue mechanism to operate, monitor and explain, and consistency across the platform is itself a requirement the ladder rarely writes down. On that team the outbox is the added complexity. The reply is that the ladder should include "what we already operate" in the cost of each option; the disagreement is about whether platform consistency belongs on the ladder or above it.
- ILLUSTRATIVEThe colleague, the analytics team's request and the meeting are invented to show the ladder in use; the Kafka trade-offs are real but the store is not.
Where the depth lives
This domain asks the question and hands the answer off by name.