Logs
Discrete events with enough context to reconstruct a failure: structured fields over prose, levels that mean something, correlation ids that survive every hop, and the secrets that must never reach log storage.
Every lesson below starts from an observable symptom and ends with the measurement that proves the fix worked. Numbers carry a label saying whether they were measured, estimated, simulated or invented to show a shape.
A log line is either a sentence a human greps or a record a program queries. The difference decides whether "how many payment timeouts hit provider X in the last hour" takes ten seconds or an afternoon of regex archaeology.
Nothing in any specification says what `warn` means. What it means is whatever your team decided, written down or not — and when it was never written down, everything becomes `info`, the error rate becomes unmeasurable, and the level field stops carrying information.
Without a shared identifier, logs from five services are five unrelated piles sorted by time. With one id propagated through every hop — and stored in a dedicated field — they become one request's story, and the log line becomes a doorway into the trace.
Log storage has a wider read audience, a longer retention and weaker access controls than the database the data came from. A token logged once is a token in a search index, in backups, and in whatever third-party service ships your logs — and no rotation policy knows it is there.
Log cost scales with traffic while its debugging value does not — the ten-thousandth identical success line teaches nothing. Sampling is how you keep the value and drop the volume, and the rule that makes it safe is simple: never sample what you would need during an incident.