Logging at Boundaries
Log state transitions and external interactions. A log line is an interface with a future reader, and most debug logging is a message the author sent to themselves an hour ago.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind survives until the requirement changes.
Which events deserve a log line, and which ones cost money, hide the useful lines, and tell a future reader nothing?
The logging bill is now a noticeable line item, the team has started sampling aggressively to control it, and the last two incidents were slower because the lines that mattered had been sampled away.
Log liberally. Add a line whenever something is unclear, remove them if they become noisy. Storage is cheap and the alternative is being blind — and that instinct is correct about being blind, which is why the pattern is so durable.
The lines never get removed, because removing one requires proving nobody depends on it and nobody knows who does.
- The lines never get removed, because removing one requires proving nobody depends on it and nobody knows who does.
- Volume grows superlinearly with traffic while usefulness stays flat, so the ratio gets worse every quarter until someone imposes sampling on everything uniformly — which drops rare events, and rare events are what incidents are made of (Sampling Without Throwing Away the Evidence).
- The lines are written for the author debugging that afternoon:
console.log("here 3", x). Six months later the reader has no idea what "here 3" meant, and the variable is a truncated object. - Personal data leaks in through generic dumps — logging the whole request body is one line to write and a compliance incident to unwind (Secrets in Logs).
- Because the useful lines are the same shape as the noise, an incident becomes a filtering exercise, and the filtering is done by the one engineer who remembers which prefixes matter (Bus Factor).
What limits the solution, and what must never stop being true
This domain leads with these two. A design that ignores its constraints is not a design, and an invariant nobody named is one nothing is protecting.
- The log vendor charges by ingested volume, so every line is a recurring cost and not a one-off (The Log Bill and What It Is Buying).
- The billing job touches forty thousand rows a night; per-row logging is not affordable and per-run logging is not useful.
- Some inputs are personal — pause reasons contain "hospital" and "divorce" — and the log vendor is outside the deletion path (What You Just Wrote Into a Log Half the Company Can Read).
- Existing log lines are unstructured strings that three dashboards already parse, so changing their format breaks things nobody owns.
- Every customer-visible state transition produces exactly one record. Not zero, and not one per layer that observed it.
- Every call leaving the process is recorded with its outcome and its duration, including the ones that succeed (Calling Something You Do Not Control).
- No log line ever contains a credential, a token, or personal data that the deletion path does not reach.
- A log line's fields are stable enough to query. A message whose shape changes with the code is not a signal, it is prose (Structured Logging: Fields a Program Can Read).
Who owns what, and where the seams fall
Responsibilities decide boundaries; boundaries decide what an interface has to say.
- The domain owns naming transitions. It does not own writing them to a file — a rule that takes a logger has acquired a reason to change that has nothing to do with the rule (Functional Core, Imperative Shell).
- The boundary layer owns recording: one line as a request enters, one as an external call returns, one per state transition committed.
- Types own preventing leaks. A
Sensitive<string>that has no serialisation is worth more than a review checklist, because it is enforced by the compiler on every future line (Sensitive State). - The Observability domain owns what happens next: levels, retention, sampling, cost. This lesson stops at what deserves a line.
- Two boundaries deserve logs: the process edge — a request arriving, a call leaving — and the state transition. Everything between them is internal control flow, and internal control flow is what a debugger and a test are for.
- The line between a log and a metric is volume and question shape: "how often" is a counter, "what happened to this one" is a log. Emitting a log to answer a "how often" question is the most common source of cost (Four Metric Types, Four Questions).
- The line between a debug log and an audit record is retention and obligation. They look identical and are governed by completely different rules, so deciding which one you are writing is part of the design (The Audit Trail).
Two kinds of line
The difference between these two blocks is not style. The left one records the path the code took, which the reader could have got from the source; the right one records what the system decided, which is nowhere in the source because it depended on data.
Note the volume, too. The left version emits four lines per subscription and forty thousand subscriptions a night; the right emits one line for the interesting case and a single summary. The right version is both cheaper and more useful, which is unusual enough to be worth pointing at.
for (const sub of subs) {
logger.debug('processing subscription', sub.id)
logger.debug('loaded', { sub }) // whole object: PII, tokens
if (sub.state === 'paused') {
logger.debug('skipping')
continue
}
logger.debug('charging')
await charge(sub)
logger.debug('charged ok')
}
logger.info('billing run complete')const summary = { charged: 0, skipped: 0, failed: 0 }
for (const sub of subs) {
const d = decide(sub, now) // pure; returns a reason
if (d.action === 'skip') {
summary.skipped++
if (d.reason !== 'not_due') log.info('billing.skip', {
subscription_id: sub.id, state: sub.state,
reason: d.reason, correlation_id: run.id,
})
continue
}
...
}
log.info('billing.run', { ...summary, correlation_id: run.id, build: BUILD })The left block cannot answer "why was 8842 not charged" — it says "skipping" with no reason and no id on the same line, so the reason has to be inferred from the previous line in a file that is interleaved across workers. It also logs the whole object, so the day someone adds a payment token to the subscription record, the token is in the log vendor. The right block logs a decision with its reason and the entity it is about, skips the uninteresting not_due case entirely, and ends with one queryable summary. It is roughly one line per thousand of the left version's volume and answers strictly more questions.
A line is an interface
Once a line exists, someone builds a dashboard on it, an alert on it, or a habit around it. At that moment its field names are a contract, and renaming sub_id to subscription_id is a breaking change that fails silently — the dashboard goes to zero and reads as "no problems" (API Stability).
That is the practical reason to treat log fields with the same care as an API: not because logs are precious, but because a broken log contract fails quiet, and quiet failures in a debugging system are discovered during the incident that needed it.
1type Transition = {2 event: 'subscription.transition'3 subscription_id: string // stable, support-supplied4 correlation_id: string // the request that caused it5 from: State6 to: State7 actor: { kind: 'customer' | 'support' | 'system'; id: string }8 reason_code: ReasonCode // enum, not free text - queryable9 at: string // ISO, from the injected clock10 build: string // which code decided this11}12 13// written by the boundary, after commit, exactly once:14await tx(async (t) => { await repo.save(sub, t); await outbox.add(t, transition) })15 16// NOT written: the customer's typed reason. reason_code is an enum;17// the free text is sensitive and lives only in the database.Three decisions are load-bearing here. reason_code is an enum so the line is queryable rather than merely readable. The write is inside the transaction and emitted after commit, so a rollback cannot leave a record of a change that did not happen. And the customer's free-text reason is deliberately absent, because the log vendor is not in the deletion path (Sensitive State).
The smell, and when it is not one
Breadcrumb logging is not a moral failing; it is a debugger written by hand for a situation where a debugger was not available, which is a completely reasonable thing to do. The problem is what it becomes when it is committed rather than deleted.
The fine case matters more than usual here. There are real environments where tracing the path is the only available instrument, and in those a breadcrumb is not a smell — it is the tool.
looks like Lines that name positions in the code rather than facts about the data: entering pause(), here 3, about to save, saved. Frequently at DEBUG, frequently with an object dumped whole, frequently with no entity id on the line.
suggests The author was debugging without a debugger and committed the scaffolding. The lines answer "which branch ran", which the source already answers, and not "what did it decide about this record", which nothing answers. Volume scales with traffic and usefulness does not.
fix Ask what question the line answers and who asks it. If the answer is "me, this afternoon", delete it before merging. If it is a real question a future reader will have, rewrite it as one structured line naming the entity, the decision and the reason, and move it to the boundary where the decision commits.
How to build it
Most important first.
- Log the decision, not the traversal. One line saying
subscription=8842 state=paused decision=skipanswers the question; twelve lines tracing the function calls that reached it do not. - One line per transition, written where the transition commits, carrying from-state, to-state, actor, reason and the correlation id (Stable Identifiers).
- One line per external interaction, on both success and failure, with target, outcome and duration. The successful ones are what let you say "the dependency was fine" during an incident (Calling Something You Do Not Control).
- Make it structured. Fields are queryable, string interpolation is not, and the difference decides whether an incident is a query or a grep (Structured Logging: Fields a Program Can Read).
- Write for the reader who has none of your context: name the entity, the decision and the reason. If the line does not say which record it is about, it is not a log line (Naming).
- For high-volume loops, log the exceptions and count the rest. Forty thousand rows should produce a handful of lines about the ones that were interesting and one line with a count (Counters: The Slope Is the Signal).
What the next change costs
The field this whole domain exists for. A structure is only better if it makes the change after this one cheaper — and it is worth saying which changes it does not help.
- Adding a field to a transition line costs one edit and is backward compatible if the format is structured. Under interpolated strings the same change costs three dashboards and a parser nobody owns.
- Adding a new transition type costs one line, because the place where transitions are recorded already exists. Under ad-hoc logging it costs a decision about where to put it, which is made differently each time.
- Removing a noisy line under boundary discipline is safe, because the set of lines is small and their purpose is stated. Under liberal logging it is unsafe forever, which is why nobody does it and why the volume only rises.
- What stays expensive: changing the log schema after external consumers exist. At that point it is a public contract with a deprecation cycle, whether or not anyone called it one (Deprecation).
- Boundary discipline means genuinely fewer lines, and there will be incidents where the line you needed was one you decided not to write. That is a real loss and it is the cost of the volume being manageable at all.
- Structured logging is more verbose at the call site than a formatted string, and on a small system that verbosity buys nothing you cannot get from grep.
- Recording transitions as first-class records is a second write on the hot path — latency and storage — for a benefit that only materialises during incidents (Designing for Cost).
What can go wrong
- Transitions get logged in two places — the service and the repository — so every change appears twice and counts are wrong in a way nobody notices until a dashboard is built on them.
- The line is written before the commit, so a rolled-back transaction leaves a log entry claiming a change that never happened. This one is common and is the logging version of The Dual Write Problem.
- A generic serializer logs a whole object and picks up a token field added by someone else six months later. The leak arrives via a change to a different file (Secrets in Logs).
- Sampling is applied uniformly to control cost, and the one-in-ten-thousand transition that explains the incident is dropped with the same probability as the routine ones (The Log Bill and What It Is Buying).
- The mitigation fails too: a team adopts strict boundary logging and then adds an exception "just for this investigation", which stays, and within a year the noise is back with a rationale attached.
- Logging at boundaries depends on the boundaries existing. In code where the transition is a column assignment in the middle of a handler, there is no place to put the line, and that is a design problem rather than a logging one (Effect Boundaries).
- It depends on correlation ids being present, otherwise a line names an entity but not the request that produced it (Correlation IDs: Turning Lines Into a Story).
- It creates a dependency from the boundary layer onto the log schema, which is why the schema is a contract: three dashboards already parse it, and changing a field name is a breaking change (Backward Compatibility as a Constraint).
- "So use log levels instead." Levels control what is emitted; they do not make a line meaningful. A DEBUG line that says "here 3" is noise at every level (Log Levels Are a Convention, Not a Standard).
- "Never log in the domain." Do not *write* in the domain — the domain should name transitions and return them, and the boundary should write them. That is a different statement from "the domain must be silent".
- "Structured logging solves this." Structure makes bad lines queryable. A structured line for every function entry is the same noise with better tooling and a larger bill.
- "Log everything and sample." Uniform sampling is exactly wrong for debugging, because it drops rare events at the same rate as common ones and rare events are the ones being investigated (Sampling Without Throwing Away the Evidence).
- duplicate-knowledge
Testing it, and how it ages
- Assert the transition record as part of the behaviour test: pausing produces one record with from-state active. That prevents the record and the behaviour from drifting apart (Where a Test Must Be Real).
- Assert that a sensitive field cannot be serialised — ideally by making it a compile error, and otherwise with a test that a log line for a pause reason does not contain the reason.
- Assert the count in the high-volume path: forty thousand rows produce a bounded number of lines. Cost regressions are behavioural regressions and can be caught the same way.
- Do not assert log lines for internal function calls. Those tests pin the implementation and are deleted at the first refactor, correctly (Mocking).
- Log lines become a schema, then a dependency, then a contract. That progression happens whether or not it is acknowledged, and acknowledging it early is what makes field renames possible (API Stability).
- As volume grows, the useful lines get promoted to metrics or traces and the log becomes the detail layer beneath them. Designing lines around decisions rather than traversal is what makes that promotion possible (RED: Rate, Errors, Duration).
- The discipline breaks down when the system spans services, because a single-process boundary log no longer describes the operation. Trace context takes over and log lines become spans' attachments (Distributed Tracing).
Where this applies
This domain's advice is contested more than most. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view rather than a caricature.
- GENERALThat a log line is read by someone without the author's context is a property of time passing, so it holds for any language or platform. What differs is the default: a framework that logs every request and query for free changes what you need to add, but not what a transition record has to contain.
- SCALE-SPECIFICAt low volume, liberal logging genuinely works — you can read it all, cost is nil, and the discipline is overhead. It stops working somewhere around the point where nobody reads a full day of logs, after which every additional line makes the useful ones harder to find rather than easier.
- CONTESTEDThe strongest opposing view is that verbose logging is the cheapest debugging tool ever invented and that discipline about it is false economy: storage and ingest keep getting cheaper, an engineer-hour does not, and the line you did not write is the one you needed. This argument is strong for low-traffic systems and for the first weeks after a launch — it is weakest where volume makes the signal-to-noise ratio itself the problem, which no amount of storage fixes.
Where the depth lives
This domain teaches the codebase-level structure and hands the rest off.
- — Testing & Reliability Engineering — whether an alert fires from a log line or a metric is a detection design question, and it decides how much a line is worth.