The question this answers
Which signals tell me the platform underneath the application is healthy, and which one lies?
Checkout must stay available. When it degrades at 02:00, the on-call engineer has to establish within minutes whether the fault is in the code, in the platform running the code, or in a provider service neither of them controls.
A view of the layer the application cannot see about itself: how much of its resource envelope is left, whether the platform is currently replacing it, whether traffic is actually being routed to it, and whether the provider is having a bad day.
The layer the application dashboard cannot see
An application dashboard answers application questions: request rate, error rate, latency, queue depth. Those are the right questions, and the method for reading them — baselines, percentiles, saturation, the golden signals, tracing a slow request across services — belongs to the Observability & Performance domain and is not repeated here. This lesson is about the *other* set of questions, the ones an application literally cannot answer about itself: how many of the instances that were supposed to be serving are actually serving, why the count changed at 02:14, and whether the box the process is running inside is running out of something.
The distinction is not academic. A process being CPU-throttled by its cgroup and a process being slow because a database query got worse look identical from inside the application: latency rose. Only the platform layer knows that the container was allocated 500 millicores, wanted 900, and spent 40% of every scheduling period in enforced sleep. Infrastructure observability is the set of signals that make that difference visible, and each layer of the stack fails upward in a way that is unrecognizable by the time it reaches the request log.
So the practical rule is: instrument the application for *what is slow*, and instrument the infrastructure for *what is running out and what is being replaced*. When those two views disagree — the application says healthy and the platform says four restarts in ten minutes — the platform view is usually the one telling the truth about what the user experienced.
The infrastructure signal set
The list below is deliberately short. Infrastructure observability rots when it becomes an exhaustive scrape of every counter a provider exports; a thousand panels nobody reads is worse than eight that someone can hold in their head at 02:00. Each row answers a question an on-call engineer actually asks in an incident, in roughly the order they ask it.
Note the last row. Cost is an infrastructure health signal, not a finance report. A bill that doubles overnight is usually telling you something operational — a retry storm hammering a metered NAT path, an autoscaler that scaled out and never scaled back in, a log pipeline that started ingesting debug output in production. See Cost Engineering; the cost view and the health view read the same underlying facts.
| Signal | The question it answers | What a bad value looks like |
|---|---|---|
| CPU and memory against the *limit*, not the host | Is the workload inside the envelope it was given? | Throttled milliseconds climbing, or working-set creeping toward the memory limit before an OOM kill |
| Disk usage and inode count on nodes | Will the node be able to accept a new workload, or write a log line? | A node that quietly stops scheduling and starts evicting healthy pods |
| Instance / pod count against desired count | Is the platform actually running what I asked for? | Desired 12, ready 7, and no application metric mentions the missing 5 |
| Restart and eviction counts | Is the platform repeatedly repairing something? | Self-healing masking a crash loop — see Self-Healing, and What It Does Not Heal and OOM Kills and CPU Throttling |
| Load-balancer target health | Is traffic being sent to this instance at all? | Healthy process, failing target check, zero requests — a readiness problem, not an application problem |
| Autoscaling events and cooldowns | Did capacity follow demand, and how late? | Scale-out triggered four minutes after saturation, which is the whole incident — see Autoscaling Signals |
| Provider service error rates and quota usage | Is the fault outside my code entirely? | Throttling responses from a managed API you did not know had a quota |
| Spend rate by tag, daily | Is something running that nobody meant to run? | A flat baseline that steps up and never comes back down |
The signal that lies
Every infrastructure component has a signal that stays green while the system is broken, and knowing which one it is per component is most of the diagnostic value in this module. The pattern is always the same: the check does not traverse the path the user traverses. A liveness probe that hits /healthz on localhost proves the process is alive and proves nothing about whether it can reach the database. A load-balancer target check that terminates at the reverse proxy proves the proxy is up. A NAT gateway that has exhausted its source ports breaks every outbound call while the health check, which never leaves the virtual network, reports perfect health.
The dump below is the shape of that lie: an application whose own dashboards are clean while the platform is visibly struggling. Reading it in order — desired versus ready, restart counts, throttling, then the probe results — gets you to the answer in under a minute. Reading only the first two lines gets you to "looks fine".
What this lesson deliberately does not do is teach the diagnostic method itself. How to form a hypothesis from a metric, how to compare against a baseline, how to follow a slow request across service boundaries — that is a full domain of its own, and this module hands off to it rather than paraphrasing it badly.
workload: checkout-api desired: 12 ready: 7 updated: 12 restarts (10m): 19 evictions (10m): 3 container cpu (limit 500m) usage p50 470m usage p99 500m throttled: 38% of periods container memory (limit 512Mi) working set p99 501Mi oom_kills (10m): 3 load balancer target group healthy 7 / 12 draining 0 requests to unhealthy targets: 0 node pool node-a disk 91% node-b disk 94% node-c disk 62% application dashboard error rate 0.4% p99 latency 310ms -> "no incident"
Key points
- Infrastructure observability answers what is running out and what is being replaced; the application dashboard answers what is slow. Neither substitutes for the other.
- Measure CPU and memory against the workload's *limit*, not the host's capacity — a throttled container on an idle node is invisible in host metrics.
- Desired count versus ready count is the single highest-value infrastructure signal, and almost no application metric reflects it.
- Every component has a check that stays green while it is broken, and it is always the check that does not traverse the user's path.
- Cost rate is a health signal: a step change in spend usually describes an operational event before finance ever sees the invoice.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • Agents on each node scrape kernel and container-runtime counters — cgroup CPU throttling, memory working set, disk and inode usage — and ship them with node and workload labels attached.
- • The orchestrator or scaling group publishes its own state: desired count, ready count, restart and eviction counts, and the events explaining each change.
- • The load balancer publishes target health independently, because "the process is alive" and "the balancer will send it traffic" are two different facts.
- • Provider services publish error, latency and quota metrics for the parts you do not run, which is the only visibility you get on the other side of the shared-responsibility line — see Shared Responsibility.
- • All of it lands in a time-series store with retention and cardinality limits, which is why label design is an infrastructure decision with a bill attached.
- • Decide the label set deliberately: environment, service, workload, node, zone. Cardinality is what makes a metrics bill explode, and per-request-id labels are the classic way to do it.
- • Keep an inventory of which alerts have a corresponding runbook. An alert with no documented action is a pager that trains people to ignore pagers.
- • Re-check dashboards after every topology change — a new zone, a new node pool or a renamed workload silently empties panels that filter on a label.
- • Own retention: high-resolution metrics for days, downsampled for months. Nobody needs per-second CPU from last quarter, and you are paying to store it.
- • Blind spot: a component nobody instrumented, discovered during the incident it caused. NAT gateways and DNS resolvers are the usual candidates.
- • Cardinality explosion: a well-meaning label containing a user id or request id, and a metrics backend that starts dropping writes for everything.
- • Alert fatigue: dozens of node-level alerts that never mapped to user impact, so the one that mattered was muted three weeks ago.
- • Monitoring hosted inside the thing it monitors — when the cluster fails, so does the dashboard that would have explained it.
- • Signals that agree with each other and disagree with users, because all of them terminate before the failing hop.
- • Metric volume grows with instances × workloads × label cardinality, and cardinality is the term that grows fastest and costs most.
- • Log volume grows with request rate, so a traffic spike raises the observability bill at exactly the moment you most need the data — see Infrastructure Logs.
- • Scrape intervals become a real load on the control plane at a few thousand targets; this is where managed collection stops being obviously cheaper than running your own.
- • The dimension that runs out first is human: past roughly a dozen dashboards per service, nobody has a mental model of what normal looks like.
- • Infrastructure metrics and logs describe your topology — node names, workload names, internal addresses, service dependencies. That is reconnaissance material; treat the observability stack as a production system with real access control.
- • The collection agent typically runs privileged on every node with a broad read scope. It is a high-value target and a genuine lateral-movement path; scope its identity as tightly as it will tolerate.
- • Never let application logs or metric labels carry secrets or personal data — see Infrastructure Logs and Audit Trails for the retention and access rules that follow from that.
- • Three meters: ingestion volume, retention duration and query load. Ingestion is usually the one that surprises.
- • Cardinality multiplies ingestion: one extra high-cardinality label can multiply the series count by thousands without any change in traffic.
- • Cross-region or internet shipping of telemetry is billed as egress on top of the observability bill — see Egress: Moving Data Costs Money, Not Just Storing It.
- • Observability commonly lands between 5% and 15% of total infrastructure spend. That is a design decision worth making explicitly rather than discovering.
- • Desired versus ready count per workload — the first thing to look at, and the one most often missing.
- • Throttled CPU periods and memory working set against the configured limit, not against the node.
- • Restart, eviction and scheduling-failure counts, with the platform events that explain them.
- • Load-balancer target health split by zone, which catches a zone problem before any aggregate metric moves.
- • The signal that lies: a liveness probe answering on localhost. It proves the process is alive and nothing else, and it is green through database outages, credential expiry and egress failure alike.
- • For a single VM running one process, the provider's built-in instance metrics plus an uptime check from outside your network is genuinely enough. A metrics stack is not the starting point.
- • An external synthetic check that exercises the real user path from outside is the cheapest thing that catches the failures internal checks miss — do this before adding node exporters.
- • Managed collection instead of a self-hosted metrics cluster: you trade per-series cost for not being paged about your monitoring at 03:00, which is usually the correct trade below a few hundred nodes.
- • If a signal has never once changed a decision, delete it. Fewer, well-understood signals beat completeness.
- • Buys the ability to distinguish "our code is slow" from "the platform is taking capacity away"; costs an ingestion bill that scales with traffic and a cardinality problem that scales with carelessness.
- • Self-hosting the stack removes per-series pricing and adds a distributed database you now operate, on call, during the incidents you built it for.
- • More retention means better post-incident analysis and a larger, permanent storage line item for data that is read approximately never.
What people believe, and what is true
We have application monitoring, so we have observability.
Application metrics cannot report that seven of twelve instances are missing, that the container is being CPU-throttled, or that the load balancer stopped sending it traffic.
Host CPU is the metric to watch.
For containers the enforced limit is what matters. A throttled container on a 20%-idle node is a real, user-visible outage that host CPU shows as healthy.
More dashboards means better coverage.
Past the point where a person can hold normal in their head, extra panels reduce coverage by diluting attention. Coverage is measured in answered questions, not panels.