Reading an Experiment Honestly
Sample size decided in advance, a duration that covers the weekly cycle, a stop rule nobody changes after peeking, novelty accounted for — and "+3%" reported with its range and its conditions, not as a fact about the future.
The ask, the obvious response, and how it goes wrong
Every lesson starts where the work starts: someone asked for something, and the first response that comes to mind has a problem.
The dashboard says the new checkout is up 3% — what would you need to know before you believe it?
Day four of the one-page-checkout test. The PM posts in the channel: "Variant B is up 3% on completion and the tool says it's significant. Let's ship it to everyone."
The tool computed significance, the number is positive, and waiting costs orders. Ship the winner.
The test was planned for three weeks. Checking daily and stopping the first time the result looks significant makes a false positive far more likely than the nominal rate the tool reports, because each look is another chance for noise to cross the line.
- The test was planned for three weeks. Checking daily and stopping the first time the result looks significant makes a false positive far more likely than the nominal rate the tool reports, because each look is another chance for noise to cross the line.
- Four days is not a week. The variant ran Tuesday to Friday; weekend shoppers, who behave differently, never saw it. The result describes weekday shoppers.
- Returning customers see a new layout and click around it because it is new. Early lifts in UI changes often shrink as the novelty wears off; four days cannot tell you which part of the 3% will last.
- Six weeks after shipping, completion is flat against the old baseline. The team concludes "the experiment was wrong" and stops trusting experiments, when the experiment was simply read too early.
What is really going on
- An experiment result is an estimate with a range. "+3%" is the middle of an interval; the honest statement is "somewhere between a small loss and a modest gain, most likely around +3%" — and the width of that range is what sample size buys.
- Sample size is decided before the test from three things: the current rate, the smallest change worth detecting, and how much error you will accept. A test too small for the effect you care about will produce either nothing or an exaggerated lucky result.
- Who decides when to stop is a statistical question, not only a political one. A fixed horizon with a single read, or a method designed for repeated looks (sequential testing), both work. Stopping whenever the number looks good does not.
- Duration has to cover the cycles that change behaviour — at least whole weeks for a store — even if the sample size is reached sooner, or the result describes only the days it ran.
- Novelty and learning effects push in both directions: a new layout gets curious clicks early, or a confusing one loses early and recovers as people learn it. New visitors are the cleaner read, because they have nothing to compare against.
- Before reading the metric, check the split. If the tool assigned 50/50 and the groups are noticeably unequal, something in the assignment or tracking is broken, and the result cannot be trusted whatever it says.
What "+3%" can and cannot mean
A result like "+3%" is the centre of a range, measured on specific people during specific days. It can mean "B probably improved completion, by an amount that might be quite small". It cannot mean "B will add 3% to revenue", "B is better for every segment" or "B will still be 3% better in three months".
Also check which 3% it is. A completion rate going from 60% to 63% is a relative lift of 5% and an absolute lift of 3 points; a rate going from 60% to 61.8% is a relative lift of 3%. People say "3%" for both. The readout should say which, every time.
Variant B: +3% conversion, significant. Recommend ship.
Variant B: checkout completion 61.9% vs 60.1% (about +3% relative), 21 days, planned sample reached, split checked. The range runs from roughly flat to around +5% relative. New visitors show a similar lift to returning ones, so novelty is not the whole story. Guardrails: refunds and payment errors unchanged. Recommend ship, with a 5% holdout for four weeks.
The readout lets someone who did not run the test decide how much to trust it. The headline asks them to trust whoever wrote it.
Who decided when to stop
The single most common way an honest team fools itself is by looking every day and stopping on a good day. Each look is another draw; with enough draws, noise will eventually cross the significance line. Deciding the stop rule before starting — and writing down who owns it — is what makes the reported confidence mean what it says.
The PM posts in the team channel on day four of a planned three-week test that B is significantly ahead and asks to roll it out today.
"We can't, it's not statistically valid yet."
"It's looking good — and it's four weekdays of a test sized for three weeks, with no weekend in it yet. Stopping the first day it looks significant is how teams ship changes that come back flat. The read date is the 21st; if it holds, we ship that day. I've checked the split and the events and both look healthy, so the test itself is fine."
Write the plan before the first user sees it
The plan is short and it is the thing that makes the result believable. It fits in the one-pager, and the readout should quote it.
1## One-page checkout — experiment plan2 3- Question: does removing the separate review step raise completion?4- Primary metric: checkout completion (definition: metrics/checkout_completion.sql)5- Guardrails: refund rate within 30 days, payment error rate, support contacts per order6- Smallest effect worth shipping: +2% relative on completion7- Sample: from the calculator, at the current 60% rate -> reached in about 2.5 weeks8- Duration: 3 full weeks (Tue 1 Sep -> Mon 21 Sep), whole weeks regardless of sample9- Read: once, on 21 Sep. Daily views are for breakage only.10- Stop early only if: a guardrail breaks its threshold, or the split is off11- Segments named in advance: new vs returning, mobile vs desktop12- Decision owner: checkout PM; readout by: checkout engineerThe line that matters most is "Read: once". The sample estimate comes from a calculator with the real baseline, not from a rule of thumb — and the duration is rounded up to whole weeks even though the sample would be reached sooner.
After the read: ship, extend or stop
The experiment tells you how confident to be. The decision still weighs cost, reversibility and the guardrails.
The read date has arrived. What now?
when The primary moved in the predicted direction, the range excludes meaningful harm, guardrails are clean.
cost A small group keeps the old flow for weeks; two code paths to maintain until the holdout ends.
when The result is promising but the range is wide because traffic was lower than planned — and you decide the new end date now.
cost More days of split traffic; the temptation to extend again until it looks good, which is peeking by another name.
when The range is centred near zero and B adds complexity, or a guardrail degraded.
cost The team's work does not ship; the finding still needs writing up so nobody reruns the same test next year.
How to do it
Most important first.
- Write the plan before starting: primary metric, guardrails, minimum effect worth detecting, sample size from a calculator, duration in whole weeks, and the single date the result will be read (Decision Records).
- Decide the stop rule in advance and write down who owns it. If people will look daily — they will — use a tool or method built for repeated looks, or agree that daily views are for spotting breakage, not for deciding.
- Check sample ratio and event reconciliation on day one. A broken split found on day one costs a day; found on day twenty-one it costs the test.
- Read the result as a range, and segment only the cuts you named in advance — new versus returning, mobile versus desktop. Every extra cut is another chance to find a coincidence.
- After shipping, keep a small holdout or at least compare against the baseline for a few weeks, so the novelty question gets an answer (Feature Flags as Product Tools).
How to explain the decision
The sentences, the order, and what to lead with — for someone who did not make the call.
- Lead with what the number is, then what it is not: "B is ahead by about 3% on completion so far. That is four weekdays of data, and the test was sized for three weeks."
- Name the specific risk in plain words: "Checking every day and stopping the first time it looks good is how you ship things that turn out to be flat. That is the most likely way this 3% is wrong."
- Give the date and the rule, not a vague "wait": "The read is on the 21st. If it still holds across three full weeks and the guardrails are clean, we ship it to everyone that day."
- Separate reading from deciding: "Even if it is real, it is a range — likely somewhere from roughly nothing to a bit more than 3%. Given B is cheap to maintain, a small real gain is worth shipping; we don't need it to be big."
- "Every day we wait we lose orders." If B is really better, yes — for half the traffic, for a bounded time. If it is not, shipping early costs every user from now on. The first cost is known and capped; the second is not.
- "Other teams ship on day four." Then their results will be harder to reproduce. We can be the team whose wins hold up after launch; that is worth more in the next budget conversation than one faster ship.
- "We don't have enough traffic for a proper test." Then say so up front and choose a different kind of evidence — a bigger change, a before/after with its caveats written down, or conversations — rather than running a test that cannot answer.
What can go wrong
- Peeking and stopping early on a positive result, then being surprised when the effect disappears in production.
- Declaring "no effect" from an underpowered test, and killing a change that might have helped because the test could not have seen it.
- Slicing the result twenty ways after the fact and shipping to "mobile users in the evening" because that slice was significant.
- Treating the experiment tool's verdict as the decision. It answers "is this difference likely to be noise?", not "should we ship?" — guardrails, cost and reversibility still decide that.
- Over-correcting: refusing to ship anything without a perfectly powered test, on a change that is cheap to reverse and obviously fine.
- "Significant means big." It means unlikely to be noise under the test's assumptions. A significant +0.2% can be real and not worth the complexity.
- "Not significant means no effect." It often means the test could not have seen an effect of the size that exists. Absence of evidence from a small test is not evidence of absence.
- "+3% on completion means +3% revenue." It means +3% on that metric in that population during those weeks. Whether it reaches revenue depends on basket size, returns and everything upstream.
- "The experiment tool handles all of this." Most tools report correctly for the design they assume. If you stop when you like or slice after the fact, the numbers they show are no longer the numbers they mean.
Knowing whether it worked
- Every experiment has a written plan dated before its start, and the readout refers back to it.
- The team has, at least once, waited out a promising early result and reported honestly when it faded.
- Launch wins hold up: a few weeks after full rollout, the metric against baseline is roughly where the experiment said it would be.
- Nobody asks "is it significant yet?" in the channel, because everyone knows the read date.
- At low traffic, a store may not have enough checkouts to detect a small change in any reasonable time. Then the honest options are testing bigger changes, accepting before/after evidence with its weaknesses, or leaning on qualitative signals.
- At 10x traffic, small effects become detectable, and the danger shifts: many teams running many tests will find some winners by chance. A shared experiment log and pre-registered plans matter more, not less.
- At 10x team size, experiments collide — two teams changing checkout at once. Someone has to own which tests can overlap on the same users.
- Waiting costs something real. If B is genuinely better, every day of the full test is a day of lost orders for half the traffic — which is the price of not shipping something that is not better.
- Pre-registering a plan removes flexibility. Sometimes the interesting finding is in a segment you did not name, and you have to run a second test to confirm it rather than claim it.
- Proper experiments need traffic, tooling and someone who can read them. For small products, the machinery can cost more than the decisions it informs.
Where this applies
Product advice is context-sensitive. These labels say what each claim is specific to, and where a different stage, team or product would differ.
- GENERALDeciding sample size, duration and stop rule before the test applies to any randomised comparison. It does not apply to a staged rollout whose only purpose is catching breakage, where stopping early on errors is correct.
- PRODUCT-SPECIFICA store with daily purchase cycles needs whole weeks. A subscription product whose outcome is next month's renewal needs a duration measured in billing cycles, and may not be able to wait for the true outcome at all.
- SIMPLIFIEDThis lesson describes a fixed-horizon test read once. Sequential and Bayesian methods allow principled repeated looks; if your tool uses them, its stop rule replaces the fixed date — the rule of deciding in advance does not change.
Where the depth lives
This domain teaches the product-side judgement and hands the mechanism off.