Dashboards Built Around Questions
A dashboard is not a place to put metrics — it is a tool for answering a specific question under time pressure. Two hundred charts is not thoroughness, it is an unindexed archive, and at 03:00 the difference between eight charts in the right order and two hundred in no order is the difference between five minutes and fifty.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Six questions, in order
A service dashboard exists to answer a short, ordered list of questions, and the order is the design. Is the service healthy? — the SLI and remaining budget, top-left, where the eye lands first. Is traffic normal? — because half of all "sudden" problems are a traffic change, and an unusual drop is as informative as a spike. Are users slow? — latency as a distribution, not an average. Are errors rising? — rate and breakdown by kind. Which dependency is failing? — per-dependency latency and error rate, because most incidents are somebody else's. Is capacity saturated? — the resource signals, deliberately last, because they explain rather than detect.
Ordering matters more than completeness. The responder works top to bottom and each question narrows the search; a dashboard that opens with CPU graphs invites the responder to start at the layer least likely to be responsible. Putting the SLI first also means the very first thing seen is whether users are actually affected, which prevents a good deal of wasted urgency.
A different discipline applies to different dashboard purposes, and conflating them is the usual root cause of the wall of charts. An overview dashboard answers "is it broken and roughly where" in under a minute. A drill-down dashboard answers "why is this specific component behaving this way" and can be dense, because whoever opens it already knows what they are looking for. A reporting dashboard answers "how did last month go" and is read by people who are not on call. One page trying to be all three serves none.
| # | Question | Panel | Why here |
|---|---|---|---|
| 1 | Is the service healthy? | SLI vs objective; error budget remaining | The only panel that says whether users are affected at all |
| 2 | Is traffic normal? | Request rate vs the same time last week | Sudden drops and spikes explain a large share of incidents |
| 3 | Are users slow? | Latency distribution or p50/p95/p99 together | A single line hides the tail that the budget is spent on |
| 4 | Are errors rising? | Error rate broken down by status class and route | Distinguishes a broken dependency from a broken client |
| 5 | Which dependency is failing? | Per-dependency latency and error rate | Most incidents originate below you, not in your code |
| 6 | Is capacity saturated? | CPU, memory, pool utilisation, queue depth | Explains the symptom; deliberately last because it rarely detects it |
Charts that answer, and charts that decorate
A few concrete choices separate a usable panel from a decorative one. Show latency as a distribution or as several percentiles together, never as a single average line — an average moves too little to be an alarm and too much to be ignored (The Average Was Fine and Users Were Not). Put a threshold or objective line on any chart with a target, so "is this bad?" is answered by looking rather than by remembering. Use consistent time ranges across the page, because comparing a 1-hour panel against a 24-hour panel beside it produces confident wrong conclusions.
Give every panel a comparison baseline. A request-rate chart showing this week over the same period last week makes anomalies visible instantly; the same chart without a baseline requires the viewer to remember what normal was, and under pressure nobody does. Overlay deploy markers everywhere, with the standing caveat that "latency rose three minutes after the deploy" is a lead and not a root cause (Correlation Is Not the Root Cause).
Then remove things. If a panel has never been the one that resolved an incident, it is costing attention. The honest test is whether someone who has never seen the dashboard can find the answer to question one within thirty seconds — and the way to run that test is to actually try it with a new team member, not to imagine it.
1checkout-service dashboard (214 panels, 7 rows, no ordering)2 3Row 1: cpu_user, cpu_system, cpu_iowait, cpu_steal, load1,4 load5, load15, ctx_switches, ... (28 panels)5Row 2: heap_used, heap_max, gc_count, gc_pause_total, ...6Row 3: [47 panels of per-pod memory, one per pod]7...8Row 7: avg_response_time (single line, no target)9 10# The SLI appears nowhere.11# Latency appears once, as an average, at the bottom.12# Nobody can answer "are users affected?" from this page.1checkout-service overview (12 panels)2 31 SLI vs objective (28d) | error budget remaining42 request rate + last week | rate by route (top 5)53 latency p50/p95/p99 | latency distribution64 error rate by status class | error rate by route75 dependency latency (db, payments, auth)86 saturation: cpu | pool in-use/max | queue depth9 10 deploy markers overlaid on all panels11 every panel: same 6h range, objective line where one exists12 -> drill-down dashboards linked, not inlinedThe right-hand page answers "are users affected, and roughly where is it" in the order a responder actually asks. The left-hand page contains strictly more information and cannot answer either question, because 214 undifferentiated panels is an archive, not an instrument.
Dashboards are for after the alert
The division of labour is worth stating explicitly: alerts detect, dashboards explain. An alert fires on a symptom because symptoms are what users feel (Alerts Worth Waking Someone For); the dashboard then narrows which layer is responsible. This is why resource panels belong on the dashboard and not in the paging rules — high CPU is a poor detector and an excellent explanation.
That framing also settles the recurring "should we alert on this?" argument. If a signal helps explain a problem you already know about, it belongs on a dashboard. If it reliably indicates users are being harmed right now and there is something a human can do, it can be an alert. Most signals are firmly in the first category, and treating them as the second is how alert sets decay (Alert Fatigue: The Page Nobody Reads).
The last piece is the link between them. Every paging alert should link to the dashboard that helps investigate it, and every dashboard should say which alerts reference it. Without that, the responder at 03:00 opens a dashboard list and starts guessing — which is the exact failure the whole apparatus was built to prevent.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| 1. SLI vs objective | 97.1% vs 99.0% | Users are affected — this is real | smoking gun |
| 2. Request rate vs last week | +3% | Not a traffic event; rules out a load spike | normal |
| 3. Latency p50 / p99 | 160ms / 4.2s | p50 flat, tail exploded — a subset of requests, not all | smoking gun |
| 4. Errors by status | 504s only, 2.9% | Gateway timeouts: something downstream is slow, not erroring | suspect |
| 5. Dependency latency | payments p99 3.9s | Found it — one dependency owns the tail | smoking gun |
| 6. Saturation (CPU, pool) | 31%, 8/50 | Confirms it is not us; we are idle, waiting | normal |
Key points
- A dashboard answers an ordered list of questions; the ordering is the design, and SLI-first prevents wasted urgency.
- Separate overview, drill-down and reporting dashboards — one page trying to serve all three serves none.
- Latency as a distribution or several percentiles, never a single average; objective lines and last-week baselines on every panel that has them.
- Alerts detect, dashboards explain: resource signals are poor detectors and excellent explanations, which is why they belong here and not in paging rules.
- A panel that has never resolved an incident is costing attention — test the page on someone who has never seen it.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Team → dashboard: adds a panel after each incident; nothing is ever removed.
- 2Panels → page: 214 charts in no particular order, with resource metrics first because they were added first.
- 3Incident → responder: opens the page, cannot find the SLI, scrolls; time-to-first-useful-signal is measured in minutes.
- 4Responder → queries: abandons the dashboard and types ad-hoc queries from memory, which only the experienced can do.
- 5Rotation → outcome: incident duration now depends on who is on call, which is a dashboard defect presenting as a staffing problem.
- • Treating chart count as coverage; it measures effort spent building, not questions answered.
- • Reading an average-latency line as representative when the tail is where the budget is spent.
- • Comparing panels with different time ranges side by side and inferring a relationship.
- • Concluding causation from a deploy marker lining up with a rise (Correlation Is Not the Root Cause).
- • Assuming a dashboard is working because its authors can use it — they know where everything is, and they are not the test.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Time how long it takes a responder unfamiliar with the service to answer "are users affected, and which layer" from the dashboard alone.
- • Record which panel resolved each incident; panels that never appear are candidates for removal.
- • Audit every paging alert for a dashboard link, and every overview dashboard for the alerts that reference it.
- • Check panels for missing objective lines, missing baselines and inconsistent time ranges as a standing review item.
- • Rebuild the overview around the six questions in order, with the SLI and budget top-left.
- • Move dense component detail to linked drill-down dashboards, and reporting views to a separate page for a separate audience.
- • Add objective lines, last-week baselines and deploy markers to every panel that supports them; standardise the time range.
- • Delete panels that have never resolved an incident, as a reviewed change rather than one at a time.
- • Link each paging alert to the dashboard that investigates it, and name the referencing alerts on the dashboard ([[alerting]]).
- • Re-run the unfamiliar-responder test and confirm question one is answerable in under thirty seconds.
- • Check during the next incident which panels were actually used, and whether the responder stayed on the dashboard rather than falling back to ad-hoc queries.
- • Confirm incident duration no longer varies sharply with who is on call — that variance is the clearest signal of a dashboard problem.
- • A minimal overview means genuinely useful detail lives one click away, which costs a few seconds during incidents that need it.
- • Standardised layouts constrain teams with unusual services, and forcing a shape that does not fit produces panels nobody reads.
- • Deleting panels occasionally removes the one that would have explained a rare failure; the compensation is that everything remaining is actually read.
- • Review the overview quarterly: add nothing without removing something, and re-test with someone unfamiliar.
- • Add "which panel would have shown this?" to the incident review; a missing answer is a dashboard gap, not an alert gap.
- • Keep drill-down dashboards out of the overview permanently — the accretion pressure is constant and only a standing rule resists it.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEPanel counts, the 214-chart example and the responder walkthrough are teaching constructions showing the shape of the problem, not measurements of a real dashboard.
- ENVIRONMENT-SPECIFICThe right overview depends on architecture: a service with one dependency needs no dependency row, and a batch pipeline needs freshness and lag panels a request-driven layout has no place for.