What a Backend Should Actually Log
Six questions every log line should help answer, and the one category of data that must never appear in one.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
What do I log, at what level, so that a production question can be answered without redeploying?
Something went wrong for one customer on one tenant an hour ago. Nobody can reproduce it. The logs need to be enough on their own.
Log liberally with console.log. Print the entrypoint of each function, the objects being processed, and a message when things fail. More logging is more information.
The volume makes search useless. A high-traffic route emitting five lines per request drowns the one line that mattered, and log storage becomes a serious line item (The Log Bill and What It Is Buying).
- The volume makes search useless. A high-traffic route emitting five lines per request drowns the one line that mattered, and log storage becomes a serious line item (The Log Bill and What It Is Buying).
- The lines are unqueryable:
console.log('saving order', order)produces free text with an object stringified into it, so you cannot filter by tenant, status or amount (Structured Logging). - Whole request objects get logged, which means
Authorizationheaders, session cookies and card tokens are now in a log index that far more people can read than can read the database (Secrets in Logs). - The lines that would have answered the question were never written, because "log everything" produces breadth without the specific fields — which tenant, which dependency, which error code — that a real question needs.
- Log calls end up on the hot path. Synchronous writes under load add latency exactly when the system is already stressed.
What is actually happening
- A log line is a durable, queryable record of one event. Its value is entirely in whether it can be *found* and whether, once found, it *answers* something.
- There are six questions a request-scoped log line should collectively answer, and the useful discipline is checking each line against them: what happened, where (service, route, host, version), for which request (correlation id), for which user or tenant, which dependency was involved, and what error (category and code).
- Levels encode who should act, not how dramatic the event feels.
errormeans a human should look;warnmeans something absorbed a failure;infomeans a business-significant event happened;debugmeans it is for the engineer working on this code path (Log Levels Are a Convention, Not a Standard). - Logs, metrics and traces answer different questions. Logs answer "what happened to *this one* request", which is exactly the question metrics cannot answer because they are aggregates (The Metrics a Backend Must Emit).
- Log writing is I/O. Most production loggers buffer and write asynchronously, which means a crash can lose the last lines — precisely the lines describing the crash.
- Everything logged is retained, indexed, replicated and often shipped to a third party. The blast radius of a logged secret is much wider than the blast radius of a stored one.
The six questions
The reason most logs disappoint is not that there are too few lines. It is that the lines lack the fields needed to narrow from "millions of requests" to "this one". Each of the six questions corresponds to a field, and a line missing one of them fails a specific class of investigation.
The test is concrete: take a real support ticket and try to answer it. Every failed attempt names the missing field.
| Question | Field | Investigation it unblocks | What it costs to omit |
|---|---|---|---|
| What happened? | msg, event | Reading the line at all | A line that says "error" and nothing else |
| Where? | service, route, host, version | Narrowing to one deploy or one bad instance | "It only fails sometimes" — one unhealthy host, invisible |
| For which request? | correlationId | Joining across services and workers | Five greps and a time window (Correlation Ids That Survive Every Hop) |
| For which user or tenant? | userId, tenantId | Answering "is this customer affected" | Cannot scope an incident to a blast radius (Multi-Tenancy) |
| Which dependency? | dependency, operation, durationMs | Attributing latency or failure to a provider | Every dependency outage looks like your bug |
| What error? | error.category, error.code, cause chain | Grouping failures by real cause | One undifferentiated pile of 500s (An Error Taxonomy That Maps Cause to Response) |
One good line beats five mediocre ones
The single most valuable log in a backend is the request-completion line. It exists once per request, it is cheap, and it is the basis of almost every ad-hoc production question — error rates by tenant, slow routes, traffic shifts after a deploy, whether a specific customer's call ever arrived.
Contrast it with the trace-through-the-function style. The second pattern below emits more lines, costs more, and answers less, because none of the lines carry the identifying fields and none of them can be aggregated.
console.log('createOrder called')
console.log('validating', req.body) // <- whole body: card token, address
console.log('user is', req.user) // <- session id, email
console.log('inserting order')
console.log('order created', order.id)
// 5 lines/request, free text, no ids, PII and secrets included,
// and no way to ask "what is the p99 for this route on tenant 42?"logger.info({
event: 'http_request',
route: '/orders', // pattern, not the interpolated path
method: 'POST',
status: 201,
durationMs: 42,
correlationId,
tenantId, userId,
orderId: order.id, // chosen fields only
version: BUILD_SHA,
}, 'request completed')
// plus one business event, and one error line if it failed:
logger.info({ event: 'order_placed', orderId: order.id, tenantId, amountCents }, 'order placed')The second is queryable. "p99 duration for POST /orders on tenant 42 since the last deploy" is one filter over one field set. The first cannot answer that question at any price, and it puts a card token in the log index.
Levels are about who acts
Level inflation is a slow failure. Someone logs a client validation failure at error because it felt bad; a dashboard counts error lines; the count is always high; nobody looks at it again. The recovery is expensive because it requires re-auditing every call site.
Anchor the levels to action rather than to sentiment, and the mapping stays stable across teams and years.
Who needs to do something as a result of this line, and when?
when A human should investigate: an internal bug, a dependency failure that was not absorbed, a broken invariant.
cost Cheap, until it is over-used — then it is the reason nobody trusts the error graph.
when Something failed and was absorbed: a retry that eventually worked, a fallback that was used, a deprecated path that was hit.
cost Warns accumulate as unowned background noise unless someone graphs their rate.
when A business-significant event, or one request-completion line. Things you will be asked about later.
cost This is the bulk of the volume and therefore most of the bill.
when Detail for someone actively working on this code path. Off in production by default.
cost The level where whole objects get logged — and where secrets leak when it is switched on to diagnose an incident.
when The process cannot continue: config invalid at boot, a required dependency unreachable at startup.
cost Must be flushed synchronously before exit or the line describing the death is lost (Validate at Startup, Fail Loudly).
How to build it
Most important first.
- One line per request at the boundary, with method, route (the *pattern*, not the interpolated path), status, duration, correlation id, tenant and user. This single line answers most questions on its own.
- One line per business-significant event: order placed, payment captured, subscription cancelled. These are the events you will be asked about a year from now.
- One line per outbound dependency call that failed or was slow, naming the dependency, the operation, the duration and the outcome.
- One line per error, at the outermost boundary only, with the category, code and cause chain (Error Boundaries: Three Translations, Not One).
- Never log request or response bodies wholesale. Log specific, chosen fields — an allowlist. A denylist of "sensitive" keys fails the first time a new field is added (Secrets in Logs).
- Route the log through one configured logger, never
console.log, so level, destination, format and redaction are decided in one place. - Include the deploy version or commit sha on every line. "Which build was this?" is the second question in most incidents ("What Changed?" — Deploy Markers and the Invisible Deploys).
- Log the route pattern, not the raw path:
/orders/:idrather than/orders/8fa2..., or every line becomes unique and aggregation is impossible.
What can go wrong
- Logging inside a loop over a collection — one request produces ten thousand lines and a log-ingestion bill, or gets rate-limited and drops the lines that mattered.
- A logger configured to write synchronously to stdout on a blocked pipe, which turns a full log buffer into a stalled process (Blocking the Event Loop).
- Logging at
errorfor things nobody can act on — a client sending a malformed body is not an error condition for you, and treating it as one trains everyone to ignore errors (Alert Fatigue: The Page Nobody Reads). - Losing the last lines before a crash because the async logger never flushed. Flush on shutdown signals (Graceful Shutdown).
- The mitigation failing: a redaction function that scrubs
passwordbut notpwd,secretbut notapiKey, and nothing at all inside a nested object or a serialized JWT. - Logging an object with a circular reference or a huge nested structure, which either throws inside the logger or serializes megabytes per line.
- Two concurrent requests in one process write to the same logger. Without request-scoped context, fields from one request can be attached to another's line — usually via a module-level "current user" variable (Request Context Propagation).
- An async logger's buffer and a shutdown signal race: log lines written during shutdown are lost unless flush is awaited before exit.
- Never log credentials, tokens, session ids, API keys, card numbers or full authorization headers. Logs are typically readable by more people than production data, retained longer, and copied to vendors (Security-Safe Logging).
- Redact by allowlist. Choose the fields that go in; anything not chosen does not appear. A denylist is a promise about every field anyone will ever add.
- Personal data in logs is subject to the same retention and deletion obligations as data in your database, and log stores are rarely built for per-subject deletion — think before logging an email address (Sensitive Data Classification).
- Untrusted input in a log line is log injection: a newline in a user-supplied field forges a log entry, and a value rendered in a log UI can carry an escape sequence or markup. Structured logging removes most of this by construction.
- Authorization decisions — especially denials — should be logged as security events with the principal, the object and the rule. Those are the lines an incident responder needs (Audit Logs for Privileged Actions).
- "More logs are safer." More unstructured logs are strictly worse: cost rises, signal falls, and the chance of a leaked secret rises with every field logged.
- "Logs are for errors." The request-completion line for successful requests is the most-used log in most systems, because it is the one that establishes what normal looks like.
- "We have metrics, so logs are less important." Metrics tell you the error rate rose. They cannot tell you what happened to the specific customer on the phone.
- "Redaction handles secrets." Redaction handles the secrets you predicted. The discipline that works is choosing fields to include.
- "Debug logs are harmless in production." They are the ones that log whole objects, and whole objects are where credentials live.
Operating it
- Measure log volume per service and per route. A route whose lines-per-request rises after a deploy is usually an accidental loop or a debug line that shipped.
- Track the ratio of error-level lines to 5xx responses. Far more errors than 5xx means you are logging things at the wrong level.
- Sample a real customer complaint end to end once a quarter: can you answer it from logs alone, without adding a line and redeploying? If not, you know which field is missing.
- Run an automated scan of the log index for token-shaped and key-shaped strings. Finding one is normal; finding one and not knowing is the problem.
- At 10x, per-request logging is usually still affordable and debug logging is not. The first thing to go is the per-step line, not the per-request one.
- At 100x, sampling becomes necessary — but sample *successful* requests. Errors and slow requests should always be kept, because those are the ones anyone will ask about (The Log Bill and What It Is Buying).
- Cardinality moves from a search concern to a cost concern: high-cardinality *fields* in logs are fine and useful, unlike in metrics where they multiply series (Cardinality: The Label That Took Down Monitoring).
- At many services, a shared log schema matters more than any individual field. Different field names for tenant across five services makes cross-service queries impossible.
- Logging is cost — ingestion, indexing, storage, and often per-GB pricing. A team that logs everything discovers this on an invoice.
- Log calls are code. Each one is a maintenance burden, a potential leak, and a line that can become wrong when the surrounding code changes.
- Discipline about what to log means occasionally not having a line you wanted. The alternative — having every line — means not being able to find it.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe six questions are stack-independent. What differs is the library and whether async logging is the default.
- RUNTIME-SPECIFICOn Node, a synchronous write to a blocked stdout pipe stalls the single loop thread and therefore every in-flight request; a threaded runtime blocks only the writing thread. This is why Node logging libraries emphasise async transports far more than JVM ones do.
- CLOUD-SPECIFICIn container platforms the convention is to write JSON to stdout and let the platform collect it; on a VM the convention may be a file plus a shipping agent, with rotation you own. What "configure your log destination" means differs completely between the two, and the file-based option has a disk-full failure mode the stdout one does not.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.