SecurityGENERALFRAMEWORK-SPECIFIC

Secrets in Logs

Logging a request object, an auth header or a webhook payload copies a credential into every system your logs reach.

The requirement, the obvious build, and why it breaks

Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.

The question

How do credentials end up in log storage, and why is deleting them not the fix?

The requirement

An integration is failing intermittently. To debug it, someone adds a log line with the outbound request and the provider's response, ships it, and the bug is found within a day.

The obvious build

logger.info('calling provider', { request, response }). Logging the whole object is the fastest way to see what is actually being sent, and structured logging makes it searchable.

Why it breaks

The request object contains headers. The headers contain Authorization, the API key, and the session cookie. All three are now in your log store, searchable, for the retention period.

How it breaks in production

The request object contains headers. The headers contain Authorization, the API key, and the session cookie. All three are now in your log store, searchable, for the retention period.
The response object of most HTTP clients embeds the request configuration — including headers — so logging an *error* leaks the credential even when logging the request does not.
Logs fan out. They are shipped to an aggregator, indexed, replicated, backed up, exported to a warehouse and read by everyone with dashboard access, including contractors and support. One log line becomes a secret in six systems with six different access-control models.
The leak is silent and permanent. There is no error, no alert and no way to know afterwards who read it, so the only honest response is rotation — which costs far more than the log line ever saved.

Requirement→API Contract→Application Logic→Data Access→External Deps→Concurrency→Failure→Security→Observability→Deployment→Scale

What is actually happening

Logging an object serializes whatever it currently holds, including fields added later by a library upgrade. An allow-list of fields is stable under that change; an object dump is not.
Secrets enter through a small number of recurring routes: an entire request or response object, an Authorization or Cookie header, a webhook body, a query string containing a token, an error object carrying request configuration, a connection string in a startup log, or a stack trace whose frames include configuration values.
Log pipelines are replication machines by design. Retention, indexing, backup and third-party shipping all mean the value exists in more places than the process that wrote it, with weaker access control than your secret store.
Redaction at the aggregator is a filter applied after the value has left the process and travelled over the network. It is a useful second layer and cannot be the first, because it fails on the shape it did not anticipate.
Personal data behaves the same way and is often the larger regulatory problem: an "anonymised" log with a customer email in a nested field is not anonymised, and log retention is rarely covered by the deletion process (Sensitive Data Classification in Security Engineering).

The object is not yours to log

The debugging instinct is right — you want to see what was actually sent. The mistake is reaching for the object, because the object is a container whose contents you did not choose and will not control after the next dependency upgrade.

Naming fields costs a few seconds and produces a log line that is smaller, cheaper to index, stable across library versions, and safe to hand to anyone.

Logging a failed provider call

Log the objects

try {
  const res = await http.post(url, body, { headers })
} catch (err) {
  logger.error('provider call failed', { err, request: { url, body, headers } })
  // err.config.headers holds the Authorization header
  // headers holds it too; body may hold a card token or an email
}

Log named fields

try {
  const res = await http.post(url, body, { headers })
} catch (err) {
  logger.error('provider call failed', {
    provider: 'payments',
    endpoint: '/v1/charges',       // the route, not the URL with its query string
    status: err.response?.status,
    providerCode: err.response?.data?.code,
    attempt,
    correlationId,                  // ties this to the request and the trace
  })
}

Every field on the right answers a debugging question and none of them is a credential. The line is also two orders of magnitude smaller, which matters directly for log cost and index cardinality, and it stays correct when the HTTP client changes what its error object carries.

The routes in

Teams fix the obvious call and leak through one of the others. Every row here is something that has happened repeatedly in real systems, and the response column is the structural fix rather than "remember not to".

The fourth row is worth special attention because it defeats otherwise careful services: your logger can be perfect while a dependency writes headers to stdout with its own configuration.

How credentials reach log storage

Trigger	Symptom	Cause	Response
Whole request object logged in middleware	Every request logs `Authorization` and `Cookie`	Object dump instead of named fields	Allow-list of request fields in the logging middleware; header allow-list, never a header blocklist
HTTP client error logged	Outbound credentials in error logs only, so it survives review of the happy path	Client error objects embed request config including headers	Sanitise errors at the boundary: keep status, code, message; drop config, headers and body
API key passed in a query string	Credential in access logs, proxy logs and browser history	The credential is in the URL, which everything logs	Credentials go in headers; never accept one from a query parameter (API Keys)
SDK or client debug logging enabled	Headers on stdout, not through your logger	A second logging system you do not configure	Audit dependency log settings at startup; fail startup if a known debug flag is on in production (Validate at Startup, Fail Loudly)
Webhook payload logged to debug signature failures	Payment identifiers, personal data and sometimes tokens stored	The raw body was needed for verification and got logged with it	Log a payload hash and the verification outcome, never the body (Webhook Signature Verification)
Exception tracker capturing request context	A second pipeline leaking what the log pipeline redacts	Automatic context capture with its own configuration	Configure the tracker's scrubbing explicitly and test it with a canary, exactly like the logger
Connection string logged at startup	Database password in the first line of every pod's log	Configuration echoed for debugging	Log configuration keys and their source, never values; use a non-printable secret type

Blast radius, and why rotation is the response

GENERALThe fan-out shape holds for any centralised logging stack, self-hosted or managed. Only the number of hops changes; the conclusion that deletion cannot reach them all does not.

Follow one line to understand why "delete the entries" is not a fix. The value is written to a file, read by an agent, sent over the network, indexed, replicated, retained under a policy measured in months, exported to a warehouse and included in a backup. Every hop has its own access model and its own copy.

By the time anyone notices, the exposure has already happened in an unknown number of places to an unknown set of readers. Rotation invalidates the value, which is the only action that acts on all copies at once. Clean-up is still worth doing — it stops future readers — but it is second.

UserLLMAgentToolDataDecisionHumanGuardrail

How to build it

Most important first.

Log fields you name, never objects you received. { provider: 'stripe', endpoint: '/charges', status: 402, requestId } answers the debugging question without carrying anything sensitive (Structured Logging).
Give secrets a type that cannot be printed: a wrapper whose toString/__repr__/Display returns a placeholder, so an accidental interpolation produces [redacted] rather than the value. This is the control that survives a tired engineer at 2am (Secrets Are Not Configuration).
Keep credentials out of URLs. Query strings are logged by proxies, load balancers, browsers and your own access logs, and no application-level redaction reaches them (API Keys).
Sanitise error objects at the boundary where they are logged: extract status, code and a message, and drop the config, headers and body.
Run redaction at the log pipeline as a second layer for known key names, accepting that it is a net with holes rather than a wall.
Never log request or response bodies for authentication, payment or webhook endpoints. If you need them to debug, log a hash or a shape summary, or capture them behind an explicit, time-boxed, audited debug flag (Feature Flags: Rollout, Kill Switches and Debt).
Test it: a unit test that runs a handler with a known canary credential and asserts the emitted log lines never contain it. This is the only mechanism that fails a build when someone adds an object dump.
Write the incident response down in advance: rotate first, then clean up. Deleting log entries reduces future exposure and does nothing about the exposure that already happened (The Secret Lifecycle in Security Engineering).

What can go wrong

Failure modes

Redaction keyed on password while the field is called pwd, secret, token, api_key or is nested three levels down in a serialized object.
A debug log level enabled in production during an incident, dumping full payloads for hours before anyone remembers to turn it off.
Third-party SDKs and HTTP clients with their own debug logging that writes headers, bypassing your logger entirely.
Exception trackers capturing local variables and request context automatically, which is a separate pipeline with its own retention and its own leak.
A signed webhook payload logged for verification debugging — payloads regularly contain tokens, personal data and payment identifiers (Webhook Signature Verification).
Redaction implemented, and applied only to the application logger while the access log still records the full URL with its query string.
The redaction itself becoming a hot path: deep-scanning every log object costs CPU proportional to payload size (What Serialization Costs).

What can race

A secret rotated after a leak is still valid in caches, sidecars and pods that have not reloaded configuration. Confirm the old value is refused before declaring the rotation complete (Configuration: Separating Code From Environment).

Security

What an attacker gets: a working credential with the privileges of whatever it authenticated. A leaked service API key is usually equivalent to the service; a leaked session cookie is that user until it expires.
The reachable population is much larger than the engineering team — log platforms are widely granted, often to support and analytics roles, and frequently to a third-party vendor.
Treat any secret that has appeared in a log as compromised and rotate it. There is no way to prove it was not read, and "the log store is internal" is the same reasoning The Trust Boundary rejects.
The presence of secrets in logs also blocks otherwise-useful practices: you cannot share a log export with a vendor, or hand a developer read access, without a review that would be unnecessary in a clean pipeline.

Misreads

"We redact at the aggregator, so we are covered." The value crossed the process boundary and the network before the filter ran, and the filter only knows the shapes it was told about.
"Only engineers can read the logs." Log platforms are among the most widely granted internal systems, and the grants outlive the roles.
"We deleted the log entries." Deletion changes future exposure. Rotation is what changes the value of what leaked.
"It is an internal service token, so it is low severity." Internal tokens are usually the ones with broad privileges and no user-level scoping.
"Structured logging solves this." Structured logging makes secrets *searchable and indexed*. It is a control for readability, not for confidentiality.

Operating it

How you see it in production

Plant a canary credential that is valid nowhere and search the log store for it on a schedule. If it appears, a code path is dumping objects.
Run secret-pattern detection over a sample of log volume — high-entropy strings, known key prefixes, JWT shapes. Providers publish recognisable prefixes precisely so this works.
Alert on log lines exceeding a size threshold. A 40 KB log line is almost always a serialized object, and almost always contains something it should not.
Audit who can read production logs, and review it as seriously as database access, because it is frequently equivalent (Security-Safe Logging in Security Engineering).

What changes at 10x and 100x

Log volume grows faster than traffic because people add lines. At 10x, sampling and level discipline become necessary for cost, and the sampling decision is also a leak-surface decision (The Log Bill and What It Is Buying in Observability & Performance).
With more services, redaction has to live in a shared logging library rather than in each service, or coverage becomes a per-team accident.
At larger scale the log pipeline crosses regions and vendors, which turns a leak into a data-residency and contractual problem as well as a security one.

What this costs

Field allow-lists are more work per log line and occasionally omit the field you needed during an incident. That is a real debugging cost, paid deliberately.
A non-printable secret type is friction everywhere a credential is legitimately used, and it will annoy someone every month. It is also the only mechanism that works without anyone remembering it.
Aggressive pipeline redaction can mangle legitimate content — a customer message containing the word "token" — and makes some incidents harder to reconstruct.
Time-boxed debug capture is genuinely useful and is itself a control that has to be audited, or it becomes the permanent leak it was meant to avoid.

Where this applies

Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.

GENERALTrue of every stack and every log pipeline. The routes in are the same everywhere because they come from how HTTP clients and loggers are built, not from any particular library.
FRAMEWORK-SPECIFICSome loggers serialize known request/response types with built-in redaction of standard headers; others serialize whatever they are given. Check what yours does with an Error from your HTTP client specifically — that object is the most common carrier, and the answer differs per library.

Where the depth lives

This domain teaches the application-side mechanism and hands the rest off.

Securitylogging-security secrets-management secret-lifecycle data-classification audit-logs

Performancelogs-and-secrets structured-logging log-cost-and-sampling cardinality

Cloudcloud-secrets-management infrastructure-logs

Agenticsecrets-and-untrusted-output