What You Just Wrote Into a Log Half the Company Can Read
Log storage has a wider read audience, a longer retention and weaker access controls than the database the data came from. A token logged once is a token in a search index, in backups, and in whatever third-party service ships your logs — and no rotation policy knows it is there.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Why log storage is the wrong place for a secret
The database holding user records is usually the most carefully controlled system you own: narrow access, audited queries, encryption at rest, a retention policy. Log storage is typically the opposite — broad read access for anyone debugging, months of retention, full-text search across everything, replication into a third-party SaaS, and inclusion in backups nobody enumerates.
So a value that was well protected in the database becomes badly protected the moment it is logged. This is a privilege inversion: the copy is easier to reach than the original. It is also durable — the value is now in a search index, in whatever the retention window is, and in every backup taken since, and none of those know they contain a credential.
The operational consequence is that a logged token cannot be rotated by rotating the token. The credential is invalidated, yes, but the log store still contains it, and if that store is where an attacker gets a foothold later, they get history. Treat "we logged a secret" as an incident with a cleanup task, not as a bug with a code fix.
| Property | Application database | Log storage |
|---|---|---|
| Read access | Narrow: a few service accounts and DBAs | Broad: most engineers, often the whole org |
| Retention | Governed by a data policy | Whatever the log budget allows — often months |
| Searchability | Requires a query and access | Full-text search across every field |
| Third parties | Rare and contractually scoped | Common — log shipping to a SaaS by default |
| Backups | Enumerated and access-controlled | Included and rarely inventoried |
| Audit of reads | Usually present (see audit-logs) | Usually absent |
Leaks come from objects, not from strings
Almost nobody writes log.info("password: " + password). Leaks arrive by aggregation: someone logs a whole request, a whole config, a whole exception, or a whole third-party response, and a sensitive field rides along inside it. The log call looks harmless because the secret is not named anywhere in the source line.
The four recurring shapes are worth memorizing. Whole-object dumps (log.debug("request", req)) carry headers, cookies and bodies. Exception logging carries whatever was in scope — including the query string that contained a token, or the connection string with the password. Third-party responses logged for debugging carry their credentials and personal data. And URLs carry tokens in query parameters, which is exactly why putting credentials in query strings is discouraged in the first place.
The corollary is that the fix cannot be discipline at the call site. It has to be a redaction layer in the shared logging helper that runs on every record, with a denylist of field names, pattern matching for token- and card-shaped values, and — best of all — a typed wrapper that makes sensitive values print as [redacted] unless explicitly unwrapped. If a secret cannot be stringified accidentally, it cannot be logged accidentally.
1log.debug("incoming request", req)2# -> headers.authorization = "Bearer eyJhbGciOi..."3# -> cookies.session = "s%3Aab12..."4 5log.error("payment failed", exc)6# -> exc.request.url = ".../charge?api_key=sk_live_9f2..."7 8log.info("provider response", resp.body)9# -> { card: { number: "4111111111111111", cvc: "737" }, ... }10 11log.warn("db connect failed", { dsn: DATABASE_URL })12# -> "postgres://app:hunter2@db.internal:5432/prod"1# 1. denylist by field name, applied to every record2REDACT_KEYS = {"authorization", "cookie", "set-cookie", "password",3 "api_key", "token", "secret", "dsn", "card", "cvc"}4 5# 2. pattern redaction for values that slip through under other names6REDACT_PATTERNS = [BEARER_RE, PAN_RE, PRIVATE_KEY_RE, CONNECTION_STRING_RE]7 8# 3. strongest: make the type unprintable9class Secret:10 def __str__(self): return "[redacted]"11 def __repr__(self): return "[redacted]"12 def reveal(self): return self._value # explicit, greppable, reviewable13 14api_key = Secret(os.environ["API_KEY"])15log.debug("incoming request", req) # authorization -> "[redacted]"16log.warn("db connect failed", {"dsn": Secret(DATABASE_URL)})17 18# select explicit fields rather than whole objects:19log.info("request", route=route_template(req), status=res.status,20 duration_ms=ms, request_id=rid)The bad version depends on every engineer remembering, forever, which fields inside every object are sensitive. The good version makes the safe outcome the default and requires an explicit, reviewable reveal() to do anything else.
Personal data, and what to do after a leak
Beyond credentials there is a larger, quieter category: personal data that is legitimately in the database and does not need to be in the log store. Full names, email addresses, physical addresses, phone numbers, national identifiers, health or financial detail. Logging an internal user id is almost always sufficient for debugging and dramatically less sensitive than logging the email, since the id resolves to a person only for someone who already has database access.
Log retention makes this sharper than it looks. A deletion request that removes a user from the database does not remove them from six months of logs, and most log stores cannot perform targeted deletion at all. The practical mitigation is to not put the data there: identifiers over attributes, and a documented classification of what may appear in a log field (see Structured Logging: Fields a Program Can Read on the field dictionary).
When a leak is found, the response has a shape. Stop the source, rotate every affected credential, scope the exposure (which fields, which window, which stores and backups), purge what can be purged, and record what could not be. Then add the detection: a scheduled scan of the log store for secret-shaped patterns catches the next one in days rather than at the next audit.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| Bearer-token pattern matches | 1,842 lines, 3 services | Live credentials in the index — rotate, then find the log call | smoking gun |
| Connection-string pattern matches | 76 lines, all from one error path | A DSN with an embedded password logged on connection failure | smoking gun |
| PAN-shaped (card) matches | 0 | Clean — but re-check after any change to payment error handling | normal |
| Email addresses in log fields | 412k lines | Not a credential, but personal data with no deletion path and months of retention | suspect |
| Debug-level volume in production | 4.1% | Debug lines are the most common leak source; confirm it is scoped and time-boxed | suspect |
Key points
- Log storage has broader access, longer retention and weaker auditing than the database the data came from — logging a secret inverts its protection.
- Leaks come from whole-object dumps, exception payloads, third-party responses and URLs, not from log calls that name a secret.
- Redaction must live in the shared logging helper; per-call discipline does not survive five teams and three years.
- A type whose string representation is
[redacted]prevents accidental logging structurally, which no denylist can fully do. - Rotating a leaked credential does not remove it from the log index or from backups — treat a leak as an incident with a cleanup task.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Debug call → record: a whole request object is logged, carrying the
authorizationheader inside it. - 2Record → shipper: the record is forwarded to the log platform with no redaction stage in the path.
- 3Shipper → index: the token is now full-text searchable by every engineer with log access.
- 4Index → backups: nightly backups copy the index, and no backup inventory records that it contains credentials.
- 5Audit → incident: months later a scan finds live tokens; rotation invalidates them, but the historical copies remain in backups that cannot be selectively purged.
- • "We never log passwords" — the leak is in the object you logged, not in a line that mentions a password.
- • "It is only in debug logs" — debug output is still shipped, indexed, retained and backed up like everything else.
- • "We rotated the token, so it is handled" — rotation closes the credential and leaves every copy in the index and the backups.
- • "Logs are internal" — internal access is broad access, and log platforms are frequently third-party services outside your perimeter.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Run a scheduled pattern scan over the log index for bearer tokens, private keys, connection strings and card-shaped numbers.
- • Inventory which log fields carry personal data and check them against your data classification.
- • Audit who can read the log store and compare that list against who can read the source database.
- • Check for whole-object log calls in code review and in static analysis — they are the highest-yield leak source.
- • Add a redaction stage to the shared logging helper: field-name denylist plus value-pattern matching, applied to every record.
- • Wrap credentials in a type whose string representation is `[redacted]`, so accidental stringification is impossible.
- • Replace whole-object logging with explicit field selection at the call site.
- • Log identifiers rather than personal attributes, and document what each field may contain in the field dictionary.
- • Re-run the pattern scan and confirm zero matches for credential shapes over a full retention window after the fix.
- • Unit-test the redaction layer against realistic request, exception and third-party response objects.
- • Verify a deliberately-logged test credential in staging appears as `[redacted]` end to end, including in the shipped index.
- • Aggressive redaction removes detail that is genuinely useful for debugging, and over-broad patterns can redact harmless values.
- • Explicit field selection is more code at every call site than dumping an object.
- • A `Secret` wrapper type is a cross-cutting change to how credentials are passed around, and needs an escape hatch that is easy to grep for and review.
- • Schedule the secret-shape scan and alert on any match, so the next leak is caught in days.
- • Add a CI rule rejecting log calls that pass whole request, response, exception or config objects.
- • Include log-field review in the checklist for any change touching authentication, payment or third-party integration code.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEMatch counts, field names and the example values are invented. Real scans depend entirely on which patterns you look for; a scan is evidence of what it matched, never proof that nothing else leaked.
- ENVIRONMENT-SPECIFICRetention, access model, third-party shipping and whether targeted deletion is possible are properties of your log platform and change the severity and the cleanup options substantially.