The Readiness Scorecard
Status per dimension with evidence — READY, PARTIAL, MISSING, UNKNOWN — and deliberately never a number, because a number invites optimising the number.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
How do you summarise a service's operational readiness without producing a figure that people will then manage?
Readiness reviews produce a lot of detail, and detail does not travel. Someone will ask for a summary. The obvious summary is a score, and a score is the one form of summary that reliably makes things worse.
Score each dimension out of five, add them up, and report a readiness percentage. Now readiness is comparable across services and trackable over time.
A number becomes a target the moment it is reported. Teams optimise the number, and the cheapest points are always the ones that matter least — a dashboard that exists satisfies "dashboards" as fully as a good one does.
- A number becomes a target the moment it is reported. Teams optimise the number, and the cheapest points are always the ones that matter least — a dashboard that exists satisfies "dashboards" as fully as a good one does.
- Weighting is unjustifiable and decisive. Is a tested restore worth the same as a runbook? Whatever you choose, the ratio is arbitrary and it silently sets everyone's priorities.
- Averaging hides the fatal gaps. A service with twelve strong dimensions and no rollback path scores 92% and cannot be recovered when it breaks. The one number that mattered has been diluted by twelve that did not.
- Comparison across services becomes possible and is meaningless: a 78% internal tool and a 78% payment service have nothing in common, and now they are on the same dashboard.
- "We do not know" is unrepresentable. It gets coded as a low score, which looks like a known weakness — but a known weakness and an unknown state need completely different responses.
- The score replaces the evidence. Nobody reads the underlying detail once there is a number, and the detail was the entire product of the review.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A score compresses several incommensurable judgements into one ordinal value, and compression is lossy in a specific direction: it discards exactly the information needed to act. "3 out of 5 on backups" does not tell you what to do; "PARTIAL: backups run nightly, restore never tested" does.
- Four statuses carry the useful distinctions. READY — evidence exists and someone has checked it. PARTIAL — something exists with a named gap. MISSING — it does not exist, and we know. UNKNOWN — nobody has checked, which is a different and often more urgent state than MISSING.
- UNKNOWN doing real work is the strongest argument for this format. A score has no way to say "we did not look", so the unexamined dimension gets a plausible number and stops being a question. Unknown risk is not small risk; it is unmeasured risk.
- Every status must be backed by evidence: a link, a date, an artifact. "READY" with nothing behind it is an opinion, and the format is designed so that an unbacked claim is visibly unbacked.
- The output is a worklist, not a rating. Every PARTIAL and MISSING is a named gap with an owner, and every UNKNOWN is a question with someone assigned to answer it. That is what a review is for (Production Readiness Review).
- The one legitimate aggregate is a blocking rule, and it is categorical rather than numeric: some dimensions may not be MISSING for a service at this tier. That is a gate with a reason, not a threshold with a number.
What a scorecard looks like
A real scorecard for a service about to carry payment traffic. Read it as a worklist: four gaps, each with a specific next action, and one genuine unknown that somebody has to go and answer.
Now imagine this as a percentage. Nine READY out of thirteen is 69%, which sounds like a passing service that needs a bit of polish. What it actually is: a payment service that cannot be rolled back and has never had a restore tested. Those two rows are the whole story, and averaging is the operation that destroys them.
| Dimension | Status | Evidence |
|---|---|---|
| Ownership | READY | catalogue/checkout-api.yaml, team payments, rotation has 6 members |
| Runbook | PARTIAL | 4 entries; no entry for provider rejection — the failure mode that caused the last incident. Owner C, due 3 Jul |
| Dashboards | READY | grafana/d/checkout-overview; used in the last two incidents |
| Alerts | READY | 3 symptom-based rules routed to payments-oncall; test alert delivered 12 Jun |
| Capacity | PARTIAL | Bottleneck is the database connection pool at 200. Headroom for failover not modelled. Owner D, due 10 Jul |
| Timeouts | READY | All outbound calls bounded; provider gateway at 3s, verified in code review |
| Retries | READY | Bounded with jitter; handlers idempotent by order id |
| Backups | MISSING | Nightly snapshots run. No restore has ever been performed. Owner D, drill scheduled 8 Jul |
| Rollback | MISSING | Last two releases included forward-only migrations. No tested path back. Owner A, blocking |
| Health checks | READY | Readiness and liveness distinct; liveness does not test the provider |
| Security | READY | Workload identity scoped to one bucket and one secret; no unused permissions |
| Dependencies | PARTIAL | Three dependencies listed; degradation behaviour stated for two. Owner B, due 3 Jul |
| SLO | UNKNOWN | No objective defined and nobody has checked whether one exists elsewhere. Owner B to determine by 1 Jul |
Why the number is worse
This is the same assessment rendered two ways. The left is what a scoring rubric produces; the right is what the format above produces. They contain the same underlying judgements.
The specific harm is not that the score is inaccurate. It is that the score is actionable in the wrong direction: the cheapest way to raise 69% is to close the two PARTIALs that are nearly done, which changes nothing about whether this service can be recovered when it breaks.
Checkout API readiness: 69% (9/13 dimensions passing). Trend: up from 54% last quarter. Target: 80% before launch.
Checkout API readiness. BLOCKING for tier 1: rollback MISSING — last two releases had forward-only migrations, no tested path back. Owner A. MISSING: backups — snapshots run, restore never performed. Drill scheduled 8 Jul, owner D. UNKNOWN: SLO — nobody has checked whether an objective exists. Owner B, by 1 Jul. PARTIAL: runbook, capacity, dependencies — gaps named, owners assigned, all due within three weeks. Everything else READY with evidence linked.
The left version can be improved by three days of work on the wrong things, and "up from 54%" reads as progress. The right version cannot be satisfied without doing the two things that actually decide whether this service survives its first bad release. It is also shorter to act on, which is the opposite of what people expect from removing the number.
Four statuses, and the one people avoid
The four are not a scale. They are four different situations that call for four different responses, and collapsing them into an ordering is the first step back towards a score.
- UNKNOWN is not a worse MISSING. MISSING is a known gap you can plan around; UNKNOWN is an unmeasured risk you cannot.
- A dimension that has never been assessed defaults to UNKNOWN, never to blank and never to a neutral middle value.
- A status with no evidence is UNKNOWN however confident the team sounds.
- A status expires when the system changes materially. An old READY is a claim about a service that no longer exists (Environment Drift).
- PARTIAL without a named gap is a way of avoiding MISSING. Require the gap in writing.
| Status | What it asserts | What it requires | The response it calls for |
|---|---|---|---|
| READY | Exists, and someone has checked it | A link, an artifact or a date — never an assertion | Nothing now. Re-assess when the system changes |
| PARTIAL | Exists with a specific named gap | The gap stated concretely, plus an owner and a date | Close the gap, or accept it explicitly and record who accepted it |
| MISSING | Does not exist, and we know that | Confirmation that it was looked for | Build it, or decide at what tier it is not required |
| UNKNOWN | Nobody has checked | Honesty, which is the expensive part | Assign someone to find out; resolve to one of the other three |
How to do it properly
Most important first.
- Report status per dimension. Never a total, never a percentage, never a letter grade, never a traffic-light average.
- Require evidence next to every status — a link, a date, an artifact name. If the evidence is an assertion, the status is UNKNOWN.
- Use UNKNOWN honestly and often. It is the most valuable status in the set and the one teams are most reluctant to write down.
- Attach a named owner and a date to every PARTIAL and MISSING. A gap with no owner is a note.
- Define blocking rules categorically by tier: a tier-1 service may not launch with MISSING ownership, MISSING alerting or MISSING rollback. State the rule as a rule, not as a score threshold.
- Re-assess on change rather than on a calendar. A status is a claim about a system, and it expires when the system changes (Environment Drift).
- When someone asks for a single number — and they will — give them the count of MISSING and UNKNOWN dimensions on tier-1 services instead. It is a fact rather than a construction, and it does not average away the fatal ones.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Nothing contains a measurement that has become a target. The damage is organisation-wide, slow, and disguised as improvement: numbers rise while the fatal gaps stay open.
What can go wrong
- A score reintroduced downstream: someone counts READYs and puts a percentage in a slide, and within a quarter that percentage is the thing being managed.
- Statuses assigned without evidence, which is a score with more words.
- UNKNOWN never used, because it looks like an admission. Every dimension gets a confident status and the scorecard becomes fiction.
- PARTIAL as a comfortable resting place — technically not MISSING, no gap named, no owner, indefinitely.
- Colour-coded status turned into an average by a dashboard tool, which is the score arriving through the back door.
- The scorecard used as a stick between teams rather than as a worklist within one, after which every status is negotiated rather than assessed.
- "Statuses are just a score with words." A score is ordinal and averages; statuses are categorical and do not. You cannot mean-average four MISSINGs and nine READYs into anything, and that is exactly the point.
- "UNKNOWN is a cop-out." It is the most actionable status in the set: it names a specific thing nobody has checked. MISSING says we have a gap; UNKNOWN says we do not know whether we have one.
- "We need a number for management." You need a summary. "Three tier-1 services are MISSING a tested rollback path" is a better executive sentence than "readiness is at 84%", and it is one someone can act on.
- "READY means reliable." READY means the operational surface exists and has been checked. Reliability is measured on the running system over time (SLOs: A Target, a Window, and a Reason).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Every dimension has a link or an artifact next to it, not an adjective.
- No document, dashboard or slide anywhere in the organisation converts these statuses into a number.
- UNKNOWN appears on real scorecards and gets resolved into READY, PARTIAL or MISSING within a defined period.
- Gaps close: the count of MISSING dimensions on tier-1 services falls, and you can point at the changes that closed them.
- A launch has actually been blocked by a categorical rule, and the rule was cited rather than argued.
- If the format is producing negotiation rather than assessment, the fix is to narrow the audience rather than to soften the statuses. A scorecard used within a team is a worklist; the same scorecard on an executive dashboard becomes a performance measure and will be gamed.
- If someone has already built a numeric roll-up, remove it rather than adjusting its weights. Reweighting concedes the premise, and the premise is the problem.
- Automate the checkable dimensions and let the tooling produce the status directly: does an ownership record resolve, does an alert route, is there a dashboard, when was the last restore (Restore Drills).
- Automate UNKNOWN as the default. A dimension nobody has assessed should report UNKNOWN rather than absent, so the gap is visible instead of blank.
- Automate expiry: a status older than its re-assessment window reverts to UNKNOWN on its own.
- Do not automate the readiness judgement. Whether PARTIAL alerting is acceptable for this service, at this tier, this quarter, is a risk decision with owners (Guardrails, Not Gates).
- Do not let a tool aggregate statuses. If your dashboard can render these as a percentage, someone will make it, and the reasoning above stops applying.
- Statuses are harder to communicate upward than a number. That difficulty is the feature, and it costs real time in every conversation with someone who wanted the number.
- You cannot rank services by readiness, which occasionally is genuinely useful for prioritising effort. The substitute — count of MISSING on tier-1 dimensions — is coarser and much harder to game.
- Honest UNKNOWNs make a team look less prepared than a team that guessed confidently. Without leadership that reads UNKNOWN as diligence, the format degrades into optimism.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe objection to scoring is about what happens to any measure that becomes a management target, which is not specific to readiness or to software. It applies equally to DORA metrics used as goals rather than signals (§69).
- ORG-SPECIFICTiers, blocking rules and re-assessment windows are conventions. Some regulated environments mandate a numeric maturity rating; where that is unavoidable, keep the status-and-evidence assessment as the working artifact and derive the mandated number from it, never the reverse.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — coverage percentages as the same failure: a number that is easy to raise and only loosely related to what it claims to measure.