Picking a Metric That Moves When the Product Gets Better
One number, close to the job the user came to do, that goes up when the product genuinely improves and is hard to push up any other way — written down precisely enough that two people compute the same value.
The ask, the obvious response, and how it goes wrong
Every lesson starts where the work starts: someone asked for something, and the first response that comes to mind has a problem.
Which single number would go up if this part of the product genuinely got better, and would be hard to push up for any other reason?
Kickoff for the checkout redesign. The PM says: "Let's track engagement." Finance says: "Revenue is the only number that matters." The designer asks for time on page. Nobody writes anything down.
Track all of them, put them on a dashboard, and see which ones go up after launch. More numbers means more information, and revenue is what the business cares about anyway.
With four numbers and no agreement on which one counts, the post-launch meeting becomes a search for the number that went up. Something always did. The redesign is declared a success on whichever metric was kindest, and nobody learns anything.
- With four numbers and no agreement on which one counts, the post-launch meeting becomes a search for the number that went up. Something always did. The redesign is declared a success on whichever metric was kindest, and nobody learns anything.
- Revenue moves for a hundred reasons the checkout team does not control: a marketing campaign, a price change, a holiday, a competitor's outage. A good checkout change can be invisible in it for months, and a bad one can hide behind a sale.
- "Engagement" and "time on page" point the wrong way for checkout. A person who spends longer on the payment step is usually confused, not delighted. Optimising it rewards friction.
- Two analysts compute "conversion" differently — one per visitor, one per session, one counting test orders — and the launch review turns into a fight about the definition instead of the result.
What is really going on
- A metric is a proxy for a user outcome. The outcome for checkout is "people who decided to buy manage to buy". Every candidate number is some distance from that; the closer it is, the fewer other things can move it.
- Distance creates noise and lag. Revenue sits downstream of traffic, pricing, stock and marketing, so it is noisy and slow. Checkout completion — of the sessions that reached checkout, how many placed an order — sits right next to the thing the team changes.
- Every metric is gameable in proportion to how easy it is to change the denominator or the definition. Completion rate goes up if you make fewer people reach checkout. That is why a primary metric always travels with guardrails (Guardrail Metrics).
- A metric only exists once it has a written definition: numerator, denominator, time window, exclusions, owner. Until then it is a word people agree on while meaning different things.
- One metric is not a belief that one number captures everything. It is a commitment made in advance about what would count as success, so the result can surprise you instead of being negotiated after the fact.
Distance from the job
Every candidate metric sits somewhere between the change you made and the outcome the business cares about. The further from the change, the more other things move it and the longer you wait to see anything. The closer to the change, the easier it is to move without helping anyone.
The good metric is the closest one that is still about the user's job rather than about your feature. "Clicks on the new express-pay button" is about the feature. "Checkout completion" is about the job — people paying — and the express-pay button is one way it might improve.
The definition is the metric
A metric that lives only as a word will be computed three ways by three people. Write it as a query, check it into the same place as the code, and put the decisions in comments — especially the ones someone will want to change later.
1-- Checkout completion: of the sessions that reached checkout,2-- the share that placed a real order within 24 hours.3-- Owner: checkout team. Changing this definition needs an entry in the decision log.4with sessions as (5 select6 s.id,7 date_trunc('day', s.started_at) as day,8 exists (9 select 110 from orders o11 where o.checkout_session_id = s.id12 and o.is_test = false13 and o.placed_at < s.started_at + interval '24 hours'14 ) as converted15 from checkout_sessions s16 where s.is_internal = false -- staff and QA accounts17)18select19 day,20 count(*) as checkout_sessions,21 count(*) filter (where converted) as converted_sessions,22 round(avg(converted::int), 4) as completion_rate23from sessions24group by day25order by day;The denominator is sessions that reached checkout, not all visitors, so a traffic campaign does not dilute it. That same choice is how it can be gamed: fewer, keener people reaching checkout raises the rate. The 24-hour window and the exclusions are decisions, not details — each one is a line someone can point at.
Choosing between candidates
Most arguments about metrics are really arguments about which question is being asked. Put the candidates side by side with what each would tell you and what it costs to rely on it.
Which number decides whether the redesign worked?
when The change is about pricing or basket size and you can run for long enough to see through seasonal noise.
cost Moves with campaigns, prices and holidays; a real checkout improvement can be invisible in it for months.
when The change is to the steps between "go to checkout" and "order placed".
cost Gameable through the denominator; needs orders per visitor as a guardrail.
when Almost never as a success metric; useful as a diagnostic when completion drops.
cost Points the wrong way — longer usually means confused, not engaged.
when Checking the element works and is found, during rollout.
cost Trivially true once the element exists; says nothing about the job.
Saying it before the result
The metric is only useful if it was chosen before anyone saw the result. The single most effective habit is a two-line note in the ticket or one-pager: which number, which direction, what you will conclude if it does not move (The One-Pager).
Kickoff for the checkout redesign. The PM has said "engagement", finance has said "revenue", and the meeting is about to move on without a decision.
"Let's track all of them and we'll see after launch what moved."
"Can we pick one now so we can be wrong about it? I'd propose checkout completion — of sessions that reach checkout, how many order within a day — because it is closest to people managing to pay. Revenue we'll watch but not decide on, because campaigns move it. And orders per visitor as a guardrail, so we can't win by shrinking who reaches checkout. I'll put the query in the one-pager today."
How to do it
Most important first.
- Start from the user's job, not from what is on the dashboard: "someone who has decided to buy manages to pay". Then ask which number moves first when that gets easier (Users and the Jobs They Hire Features For).
- Pick one primary metric per change, and write down its definition as a query before the work starts — numerator, denominator, window, exclusions, and who owns changing it.
- Prefer rates over counts for anything traffic can inflate, and per-user or per-session over per-pageview for anything a retry or a refresh can inflate.
- Ask how you would cheat it. If a one-line change could move the number without helping anyone, name the guardrail that would catch the cheat, and put it next to the primary.
- Say what direction and rough size you expect before shipping, and what you will conclude if it does not move. That turns the metric into a prediction you can be wrong about (Outcome vs Output).
How to explain the decision
The sentences, the order, and what to lead with — for someone who did not make the call.
- Lead with the user outcome, then the number: "What we want is that people who decide to buy manage to pay. The number closest to that is checkout completion — of sessions that reach checkout, the share that places an order within a day."
- Say why not the obvious one, in one sentence: "Revenue is what matters, but it moves with campaigns and prices we do not control, so a real checkout improvement could be invisible in it for a quarter."
- Say how it could be cheated and what stops it: "You could raise completion by making fewer people reach checkout, so we are watching orders per visitor alongside it."
- State the prediction: "We expect completion to go up by a small amount. If it does not move after three full weeks, the redesign did not fix what we thought was broken, and we will say that."
- "Leadership only cares about revenue." Then show the link once: when completion went up last time, did revenue per checkout session follow? If it did, completion is the fast, readable version of the same thing.
- "Picking one number is reductive." It is. That is the point — the alternative is picking the flattering number afterwards. We keep the others as guardrails and context.
- "We don't have an analyst to write the SQL." Then the definition is a paragraph and the query is twenty lines; the engineer who built the checkout is the best-placed person to write both.
What can go wrong
- The metric is chosen after launch, from the numbers that went up. This is the most common failure and it feels like analysis while it happens.
- The metric is right but the definition drifts: someone adds a filter for bot traffic in March, the rate jumps, and the jump is credited to whatever shipped in March.
- The metric is so close to the change that it is trivially true. "Clicks on the new button" goes up when you add a button. It measures that the button exists.
- Over-correcting into one number for everything: forcing the order page, the cart and support tooling onto a single company metric that none of them can move on its own.
- "One metric means we ignore everything else." No — it means one number decides whether the change worked; guardrails and qualitative signals still decide whether it was safe and why.
- "The metric should be the company's north star." The company metric is usually too far from any single team's change to read. Your metric should be the closest one to your change that still links to it.
- "Once we pick it, it is fixed forever." The definition should be stable within an experiment and revisable between them — in writing, with a date, so nobody credits a definition change to a feature.
Knowing whether it worked
- Before launch, the PM, the analyst and you can each say the metric's definition and get the same number from the same data.
- The launch review is about what the number did, not about which number to look at.
- The metric has disagreed with the team's gut at least once, and the team believed the metric — or found a real flaw in its definition and fixed it in writing.
- Finance stops asking "but what did it do to revenue?" for every checkout change, because the link from completion to revenue has been shown once and is trusted.
- At small traffic the closest metric may be too noisy to read within a month; you may have to accept a slightly more distant but higher-volume proxy, or rely on qualitative signals (Qualitative Signals).
- At 10x team size, metric definitions need an owner and a single source — a shared metrics layer rather than a query in each analyst's notebook — or every team will have its own "conversion".
- At 10x revenue, the metric tree matters: the company metric at the top, team metrics beneath it, and an explicit, checked argument for why moving the team metric moves the one above.
- One number throws information away on purpose. Sometimes the thing that mattered shows up in a number you chose not to watch, and you only find it later.
- Writing the definition first takes a real hour of arguing about denominators before anything is built, and it will feel like bureaucracy on a small change.
- A metric close to the change is easy to move and therefore easy to over-credit. The closer it is, the more it needs a guardrail and a link upwards.
Where this applies
Product advice is context-sensitive. These labels say what each claim is specific to, and where a different stage, team or product would differ.
- GENERALChoosing the success number before the result is known applies to any change you intend to evaluate. It does not apply to pure maintenance work, where "nothing broke" is the outcome.
- PRODUCT-SPECIFICFor a B2C store with daily traffic a rate like checkout completion is readable within weeks. For a B2B product with a few dozen accounts, rates are noise; a count of accounts that completed the job, plus conversations, works better.
- STAGE-SPECIFICPre-product-market-fit the metric is usually retention of a small cohort, because the question is whether anyone comes back. In a mature product the question is narrower, and so is the metric.
Where the depth lives
This domain teaches the product-side judgement and hands the mechanism off.