Logs, Metrics and Traces

Logs answer *what happened*: a discrete event with a timestamp and context. Make them structured — JSON with level, service, request_id, user_id, order_id, duration_ms — not prose, so they can be filtered and aggregated. Include a correlation id (the trace id, or a request id the gateway minted) in every line, or you cannot join the gateway’s "received" to the order service’s "failed". Logs are the highest-cardinality signal: a line per event with any fields you like. That is their power and their bill; a service doing 5,000 requests per second writing three lines each is 1.3 billion lines a day.

Metrics answer *how much, how often*: numbers aggregated over time, cheap to store because they are pre-aggregated. The two standard sets: RED for request-serving services — Rate (requests/s), Errors (failures/s), Duration (a latency histogram) — and USE for resources — Utilisation, Saturation, Errors — for CPU, connection pools, queues, disks. Duration must be a histogram, not an average: an average of 80 ms hides that 1% of users wait 4 s. Report p50, p95, p99; alert on p99. The constraint is cardinality: every distinct label combination is a time series, so route and status are fine labels and user_id is not — a million users times ten routes is ten million series and a dead metrics database.

Traces answer *where did this request spend its time*: a tree of spans, each with a start, a duration, a service name and a parent, sharing one trace id. A trace is the only signal that shows *structure* — that the order service called the database 38 times in series, or that two calls that could run in parallel ran one after the other. They are sampled (1–10% of requests, plus 100% of errors) because a full trace per request costs more than the request.

The matrix is the answer to "why not just log everything?" and to "why not just add more metrics?". Each signal has a question it answers well, a cardinality it tolerates, a cost curve, and a blind spot that only another signal fills. The metric tells you p99 is 412 ms; the log tells you which order and that it made 38 database calls; the trace shows those 38 calls happened one after another and that the caller could not have known — the N+1 in the challenge the-140ms-database-span.

The instinct is to alert on everything measurable: CPU over 80%, a queue over 1,000 messages, a pod restarting. Those are *causes*, and most of them do not hurt anyone — CPU at 85% with p99 at 120 ms is a healthy, well-utilised service. Paging on causes trains the on-call to ignore pages. Alert instead on symptoms the user feels, expressed as SLOs (Availability, SLOs and Error Budgets): "99.9% of checkout requests succeed within 500 ms over 30 days". Page when the error budget is burning fast — a burn rate that would exhaust the month’s 43 minutes of allowed unavailability in a few hours — and put the cause-level signals on a dashboard the responder opens *after* the page.

This is where the three signals compose into a workflow. The metric fires the page (p99 or error rate over the SLO threshold). The trace for a slow sampled request shows which hop absorbed the time. The logs for that trace id show the input that triggered it. A team that has only logs cannot page reliably; a team that has only metrics knows something is wrong and not where; a team that has only traces sees 1% of requests and has no history.