Evaluating a Technology
A technology evaluation has six parts: the problem it solves, its mental model, its guarantees, its costs, its failure modes and its alternatives. Filled in honestly for two or three options — including the one you already have — it decides; filled in for one option, it advocates.
The situation, the reflex, and why it stalls
Every lesson starts where being stuck starts: someone has a problem, and the first move that comes to mind feels like progress.
You have narrowed a need to a few candidate technologies. What does an evaluation have to contain to actually decide between them, rather than confirm the one you already liked?
The store has a measured problem now: product pages read the same handful of rows thousands of times an hour, and the database is the bottleneck on the product page. Three options are on the table — an in-process cache, a shared cache service, and read replicas — and each has an advocate. The meeting has gone round twice.
Compare features. Make a table with the three options as columns and every capability anyone mentions as rows, tick the boxes, and count. The option with the most ticks wins. It feels rigorous because it is a table.
The table has twenty rows and the store needs three of them. Counting ticks weights "supports clustering" the same as "survives a restart", and the winner is the option with the longest feature list, which is rarely the option with the right guarantees.
- The table has twenty rows and the store needs three of them. Counting ticks weights "supports clustering" the same as "survives a restart", and the winner is the option with the longest feature list, which is rarely the option with the right guarantees.
- The mental model of each option never appears in the table, so the team adopts a shared cache without anyone having said aloud that a cache is a second copy of the truth that can be wrong — and the first stale price on a product page is a surprise.
- The option the store already has — the database, with an index and a query fix — is not a column, because nobody advocated for it, so the evaluation cannot conclude "none of the above".
- The failure modes are a row labelled "reliability" with three ticks, which says nothing about what a customer sees when each option breaks.
The move
Precisely enough to apply it to a problem you have never seen — not a slogan.
- Give the evaluation six fixed parts and fill them in per option: the problem it solves (in one sentence, matched against the measured problem), its mental model (the one idea you must hold to use it without surprise), its guarantees (what it promises and does not), its costs (build, operate, learn), its failure modes (what the user sees when it is down, stale, full or restarted), and its alternatives (including the current system, improved).
- Fill in the current system as an option, always. If "the database with a better index" meets the measured problem, the evaluation must be able to say so, and it cannot if that option is not on the page.
- Score the axes that matter to this decision — simplicity, performance, reliability, cost, time, maintainability — with coarse ratings and a stated caveat, and refuse to sum them. The scores are for seeing where options differ, not for producing a winner; the winner comes from the guarantee the measured problem needs and the failure mode the store can live with.
- Make "premature optimisation" falsifiable: an optimisation is premature when it is adopted before the bottleneck it addresses has been measured. This store has measured its bottleneck; the evaluation is therefore not premature, and the phrase should not be used to block it.
The template, filled in
The matrix is the six parts across the store's four options. It is deliberately terse: each cell is a sentence, and the value of the template is that the same six questions are asked of every column, including the column nobody advocated for.
Read the failure-mode row first. It is the row that decides, and it is the row a feature table never has.
| Current system, improved | In-process cache | Shared cache service | Read replicas | |
|---|---|---|---|---|
| Problem it solves | A slow query, if the query is unindexed | Repeated reads of small hot data on one server | Repeated reads of hot data across servers | Read capacity in general |
| Mental model | None new | Each server holds its own copy; copies disagree briefly after a write | One shared copy that must be invalidated on write | The database copies itself with lag |
| Guarantees | The database's own | None across servers; per-server expiry | Single-key atomicity; persistence configurable | Eventual on replicas |
| Costs | An hour | An afternoon; nothing to operate | A service to run; an invalidation path | Infrastructure; a read/write routing change |
| Failure modes | None new; may do nothing | Stale price up to expiry on some servers; unbounded memory | Down: fallback or outage; stale until invalidated; stampede on hot-key expiry | Read-your-own-write returns the old value |
| Alternatives | Every other column | B when servers multiply | A when there is one server | Only if reads are not cacheable |
Scored, with the caveat that matters
The scores below are coarse on purpose and the caveat is part of the device. They show where the options differ — simplicity and cost separate them sharply, performance barely does — and they must not be summed, because the decision turns on one row of the matrix above, not on an average of six axes.
| Option | Simplicity | Performance | Reliability | Cost | Time | Maintainability | Note |
|---|---|---|---|---|---|---|---|
| Current system, improved | Best on everything except that it may not solve the problem; measure after. | ||||||
| In-process cache | Solves the measured problem with a stale window the founder has accepted. | ||||||
| Shared cache service | Right when servers multiply; a new component to run and a new way to be down. | ||||||
| Read replicas | Solves general read capacity, which is not the measured problem. |
caveat The performance column cannot distinguish "fast enough" from "faster", and fast enough is all the store needs. The reliability column hides that each option fails differently — stale, down, lagging — and only the matrix above says what a customer sees. Do not sum the rows; the decision is made on the guarantee needed and the failure mode accepted.
How the decision is actually made
With the matrix and the scores in hand, the decision is a short set of criteria. The technology decision tool at /thinking/decide asks the same questions for the common components — cache, queue, search, container orchestration — and every leaf names the simpler thing and the failure mode, because an evaluation that cannot say "not this" is not one.
What does the measured problem need, and what failure can the store accept?
when The hot query is unindexed or does redundant work; always first, because it is an hour and may end the evaluation.
cost May do nothing, in which case the hour bought a cleaner query and a second measurement.
when The reads are of a small set of hot rows on few servers and a brief stale value is acceptable to the business.
cost A stale window per server and a memory bound to set; the moment servers multiply, the windows disagree.
when Many servers must agree on the cached value, or the stale window across servers has become a support problem.
cost A component to operate, an invalidation path that is a source of bugs, and a stampede to design for (Cache Stampede: Everyone Misses at Once).
when Reads are not cacheable — every read is different — and the database's read capacity, not a hot set, is what is exhausted.
cost Replication lag as a user-visible failure mode, and routing logic that must know which reads may be stale.
How to do it
Most important first.
- Write the measured problem at the top of the page — the operation, the reading, the budget it misses. Every option is judged against that sentence and nothing else (What Problem Does It Solve?).
- For each option, including "current system, improved", write the six parts in one or two sentences each. Where you cannot, that is a research task, not a blank (Researching an Unknown Technology).
- Write the mental model as the sentence a new team member would need: "a cache is a second copy that can be wrong; you decide how wrong and for how long". If the sentence contains a surprise, that surprise is a failure mode.
- Write each failure mode as what a customer or admin sees, not as "downtime". "Product page shows yesterday's price for up to a minute" is a failure mode a founder can approve or reject (Guarantees and Failure Modes).
- Score the axes coarsely, state the caveat, and then decide on guarantees and failure modes — never on the total (The Trade-Off Matrix, Without Fake Precision).
- Record the decision with the alternatives considered and the observation that would reopen it (The Decision Journal).
Worked on a concrete problem
The move has to produce something. This is what it produced.
- Measured problem: the product page's hot rows are read far more often than they change, and the database is the product page's bottleneck under current traffic. Option A, in-process cache: solves repeated reads of small hot data; model: each server holds its own copy, so copies disagree briefly after a price change; guarantees: none across servers; costs: an afternoon, nothing to operate; failure modes: a stale price for up to the expiry on some servers, memory growth if unbounded. Option B, shared cache service: same problem; model: one copy shared, invalidated on write; guarantees: single-key atomicity, persistence configurable; costs: a service to run, an invalidation path to write; failure modes: cache down means fallback or outage, stale until invalidated, stampede when a hot key expires. Option C, read replicas: solves read capacity generally; model: the database copies itself with lag; guarantees: eventual on replicas; costs: infrastructure and a routing change; failure modes: reading your own write returns the old value. Option D, current system improved: an index on the product query and removing a redundant join; model: none new; costs: an hour; failure modes: none new; limit: if the reads are already indexed, it does nothing.
- The decision: D first, measured again. If the bottleneck remains, A, because the measured problem is a handful of hot rows on one page and a brief stale price is a failure mode the founder accepted in writing. B is the answer if the store grows to many servers and the stale window across them becomes a support problem — that observation is recorded as the reopening condition. C solves a problem the store does not have yet. The six parts made "none of the above, then the smallest" sayable, which the feature table could not.
How you know it worked
What now exists that did not before, and what question you can now ask.
- The current system, improved, is on the page as an option with the same six parts as the others.
- Every failure mode is written as something a user sees, and the founder has seen the list.
- The scores were not summed, and the decision cites a guarantee and a failure mode rather than a total.
- The decision record names the observation that would reopen it.
The questions you can now ask
The field this whole domain exists for. After this lesson, these are the questions to put to an unfamiliar problem.
- ?What is the measured problem, in one sentence, that every option is judged against?
- ?What is the one idea I would have to hold to use each option without being surprised?
- ?What does each option guarantee, and which of those guarantees does the measured problem actually need?
- ?What does a customer see when each option fails — down, stale, full, restarted?
- ?What would the current system, improved, do — and what observation would reopen this decision?
What can go wrong
- Six parts for six options: the evaluation becomes a document nobody reads and the meeting goes round a third time. Two or three options, including the current one, is the right width; more means the need is not yet narrow enough.
- The scores are summed anyway, because a number is comforting, and the option that scored best on axes the decision did not depend on wins.
- The evaluation is written after the decision, as justification. The tell is that the current system is missing or scored implausibly low.
- Applied to a decision that is cheap to reverse — a logging library — where the right evaluation is to try one for a week. Six parts are for decisions that are expensive to undo (Reversible vs Irreversible Decisions).
- Six parts per option is a page or two of writing before any code, and on a reversible decision that is more than the decision is worth.
- Insisting on a measured problem means the evaluation cannot start until measurement exists, and building the measurement is itself work.
- Refusing to sum the scores leaves the decision to judgment, which is harder to defend in a meeting than a number — and is the honest position.
- "The six-part template is the evaluation." The template is the structure; the evaluation is the honesty of what goes in it, especially the failure modes and the current-system column. A template filled in for one option is advocacy with headings.
- "Scores across seven axes are false precision, so skip scoring." Coarse scores are useful for seeing where options differ. What is false is summing them; the caveat exists to say so.
- "Measured means we cannot evaluate until production traffic exists." Measured means you have a reading of the problem you are solving; a load test on a staging copy is a reading. The point is to evaluate against a number, not against a fear.
Where this applies
Problem-solving advice is stated as universal far more often than it is. These labels say what each method is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALProblem, model, guarantees, costs, failure modes, alternatives apply to any component decision — a database, a queue, a framework, a cloud service, a build-versus-buy — because the six are the questions any such decision turns on.
- SCALE-SPECIFICOn a small system the "current system, improved" column wins most evaluations and the six parts are a way of proving it quickly. As traffic and team grow, the guarantees and failure-mode rows dominate, because the stale-price window that one server makes invisible becomes a support queue across twenty.
- ILLUSTRATIVEThe product-page bottleneck, the four options and the founder's acceptance of a stale window are invented to show the shape of the evaluation; the guarantees described for each option are their general shape, not any product's specification.
Where the depth lives
This domain asks the question and hands the answer off by name.
- — The manifesto at /manifesto/delegating: a cache client delegates storage and expiry; deciding how stale a price may be stays yours, and it is the row of the matrix that decides.