SerializationGENERALFRAMEWORK-SPECIFICDATABASE-SPECIFIC

Schema Leakage

Returning the row is publishing the schema. It is a contract you did not write, cannot see, and will be held to.

What actually happensHow to build it

The requirement, the obvious build, and why it breaks

Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.

The question

What exactly do you promise a client when you hand it a database row as JSON?

The requirement

A quick internal endpoint returns orders so the ops dashboard can list them. It is one query and one res.json, and it ships on a Tuesday.

The obvious build

SELECT * and return the rows. The dashboard needs the data, the columns are already named sensibly, and writing out a field list is duplication that will drift from the table anyway.

Why it breaks

A year later, orders.total must become total_minor because floats lost cents. The column rename is a five-minute migration and a six-week client migration, because four consumers read total (Expand and Contract Migrations).

How it breaks in production
  • A year later, orders.total must become total_minor because floats lost cents. The column rename is a five-minute migration and a six-week client migration, because four consumers read total (Expand and Contract Migrations).
  • Someone adds internal_margin to power a report. It appears in the public response on the next deploy. Nobody wrote a line of API code and the API changed.
  • A deleted_at column is added for soft deletes. Clients start filtering on it, and now your soft-delete strategy is part of the public contract.
  • A consumer starts depending on the column *order* in a CSV export, or on the fact that customer_id is an integer. Both were incidental; both are now load-bearing.
  • The dashboard is exposed to customers "temporarily". The row includes tenant_id, acquisition_cost and an internal note field, and the leak is discovered from a support ticket.
Requirement→API Contract→Application Logic→Data Access→External Deps→Concurrency→Failure→Security→Observability→Deployment→Scale

What is actually happening

  • A response is a contract whether or not you wrote one. Consumers code against what they observe. Every field you emit is a field somebody may read, and you find out which ones only by breaking them (Backward Compatibility: The Real Rules in API Design).
  • Passthrough inverts the direction of change. Normally a contract change is a decision. With SELECT * to JSON, a schema change *is* a contract change, made by whoever wrote the migration, reviewed by whoever reviews migrations.
  • Silence publishes. Adding a column requires no API review, no changelog entry and no version bump. The default for a new column becomes "public".
  • It couples two things with very different change rates. Storage wants to evolve — new indexes, denormalisation, column splits, a different engine. The public shape wants to stay still. Passthrough welds them.
  • It removes the only place redaction can live. With no response type, there is no line of code where "this field is not for this audience" can be written (Three Models, Not One).

The contract nobody signed

Follow the coupling in one picture. A migration is authored by a backend engineer, reviewed by another backend engineer, and lands. If nothing sits between the row and the response, that migration has just amended a public interface, and the review that would have caught it was never invited.

The seam in the second half of the diagram is small — one function, one type — and it is the entire difference between "a column changed" and "the API changed".

reviewed as storageno seamcontract changed, silentlywith a seamcontract changes only hereMigration adds a columnorders tableSELECT *Response typeres.json(row)Client reads only what you listedClient reads the field
UserLLMAgentToolDataDecisionHumanGuardrail

What each schema change does to the API

These are ordinary migrations. Every one of them is correct database work, and every one of them has a second effect that nothing in the migration review surfaces.

Read the last column as the general rule rather than the individual fix: in each row, the response is enumerated somewhere, so the schema change stops being an API change.

TriggerSymptomCauseResponse
Add a column for an internal reportAn internal metric is visible in a public responseFields are published by existing, not by decisionAllowlist the response fields; new columns are private by default
Rename a column for correctnessEvery consumer breaks at once on deployThe storage name was the wire nameRename at the boundary; storage names stay internal
Add deleted_at for soft deletesClients build logic on it; the strategy is now publicAn implementation detail became observableFilter in the query; never emit the marker
Change an integer id to a UUIDClients that parsed ids as numbers failThe column type was the contract typeIds as opaque strings in the response from day one (Serialization: Objects to Bytes)
Denormalise for query performanceThe response shape changes with the storage layoutThe response mirrored the join structureResponse shape is designed for the consumer and mapped (Three Models, Not One)
Multi-tenant column exposedCallers learn other tenants' identifiersThe row contained the isolation keyNever emit tenant or ownership keys the caller does not already have (Tenant Isolation)

Two lines, two contracts

The difference in code is small enough to look pedantic and structural enough to change who can alter your API. That asymmetry is why this is a lesson and not a style note.

The version on the right is also cheaper: it fetches fewer columns, builds smaller objects and encodes fewer nodes. It is rare that the safe choice is also the faster one, and here it is.

The orders list endpoint
The row is the response
const rows = await db.query('select * from orders where customer_id = $1', [id])
res.json(rows)

// Public contract = current table definition.
// Anyone who can write a migration can change the API.
// Adding a column is a disclosure decision made by silence.
The response is a decision
const rows = await db.query(
  'select id, status, total_minor, currency, created_at\n     from orders where customer_id = $1 and deleted_at is null',
  [id]
)
res.json(rows.map(toOrderSummary))

// Public contract = toOrderSummary + OrderSummary.
// A migration cannot change it. A PR that changes it
// shows up as a diff on the contract test.

The right-hand version puts a named, reviewable artefact between the storage schema and the consumer. That artefact is what makes a column rename an afternoon instead of a deprecation programme — and it is what makes a new column private until someone decides otherwise.

How to build it

Most important first.

  • Enumerate the response fields somewhere — a response type, a serializer, an explicit projection. It does not matter which; it matters that adding a column is not enough to change the API.
  • Select the columns you serialize rather than SELECT *. That single change kills the leak and reduces query, construction and encode cost at the same time (What Serialization Costs).
  • Make new columns private by default. The rule is: a field is in the API because someone put it there, never because it exists.
  • Rename at the boundary. Internal names should be free to be internal — total_minor in the table can be total.amount in the response and neither side has to compromise.
  • When you must break a shape, use expand and contract: add the new field, run both, migrate consumers, remove the old one on a published schedule (Expand and Contract Migrations, Removing Fields Without Removing Consumers in API Design).
  • Test the exact key set of each response. A test that fails when a field appears is the mechanism that turns "we should be careful" into a check (Contract Tests Between Services).

What can go wrong

Failure modes
  • The projection is applied in the query but the ORM returns a full entity anyway because it hydrates all mapped columns — the leak is closed in the SQL and open in the object.
  • A serializer is added for the main endpoint and not for the search endpoint, the export, the webhook payload or the error response that echoes the record (Outbound Webhooks).
  • A denylist is used instead of an allowlist. It protects the fields that existed when it was written, and every column added afterwards is public again.
  • Internal endpoints leak because "it is internal" — until an API gateway, a partner integration or a new frontend makes it not internal.
  • The response is fixed and the same row still reaches clients through logs, analytics events or a cached copy (Secrets in Logs).
What can race
  • When a column rename ships under expand-and-contract, both names exist for a window and writers must update both. A write that sets only the old column while a reader has already moved to the new one produces a value that appears stale for reasons no query explains (Expand and Contract Migrations).
Security
  • This is the mechanism behind a large share of accidental data disclosure: no exploit, no injection, just a column that was returned because it existed. Password hashes, TOTP secrets, internal scores, other tenants' identifiers and soft-delete state all reach clients this way.
  • In a multi-tenant system, returning the row returns tenant_id and other tenants' foreign keys, which hands an attacker exactly the identifiers needed to probe object-level authorization (Multi-Tenancy, Object-Level Authorization).
  • Field names are information. Publishing your column names describes your data model to anyone deciding where to spend effort.
  • Redaction is per audience, not per field. The same order shown to a buyer, a seller and an internal admin is three response shapes — a UI that hides a field is not a control (Where the Check Belongs).
Misreads
  • "It is an internal API." Internal is a deployment fact, not a property of the code. Internal endpoints become external by gateway, by acquisition, or by a frontend that fetches them directly.
  • "We can remove the field later." You can remove it when every consumer has stopped reading it, and you do not know who is reading it.
  • "A denylist is the same thing." It protects the columns that existed the day it was written. Allowlist and denylist differ exactly on the columns that do not exist yet.
  • "Adding a field is a backwards-compatible change, so passthrough is fine." Adding a field is compatible for clients and is a disclosure decision for you. Those are different questions with different reviewers.

Operating it

How you see it in production
  • A contract test asserting the exact keys of each response. This is the highest-value check in the module.
  • A schema artefact — OpenAPI or equivalent — generated and committed, so a response shape change appears as a diff in review.
  • Grep the codebase for SELECT * and for handlers that pass a repository result directly into a response. Both are mechanical and both find real instances.
  • A migration checklist item: does this column appear in any response, log or event? If nothing can answer that quickly, the leak is structural.
  • Response-size-per-route: a step change on a route you did not deploy usually means a column arrived.
What changes at 10x and 100x
  • The cost scales with consumers you cannot deploy. One internal consumer: a rename is a chat message. Four external ones: a deprecation programme.
  • At many services, passthrough turns your database schema into a distributed dependency — other teams' code breaks when you change a column, and the coupling appears in no dependency graph.
  • It also compounds with data growth: wide tables mean every response carries every column, so the leak and the bandwidth bill grow together.
What this costs
  • Enumerating fields costs a list per resource that someone must maintain, and there will be a day it lags the entity by a week.
  • Renaming at the boundary means the name a client reports in a bug is not the name in your table, and engineers must learn the mapping.
  • Strict contract tests fail on every intentional change too, which is the point and is also friction on every legitimate PR.

Where this applies

Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.

  • GENERALAny service that hands storage-shaped data to a consumer it does not deploy has this coupling, whatever the stack or transport.
  • FRAMEWORK-SPECIFICSome stacks make the leak the default and some make it visible: Django REST Framework's fields = '__all__' and Rails' render json: model publish every attribute, while a Go handler returning a struct publishes exactly the tagged fields — so in Go the leak requires editing the struct, and in the others it requires editing nothing.
  • DATABASE-SPECIFICEngines differ in how cheap a schema change is, which changes how often this bites: adding a column is near-instant in Postgres and MySQL for most cases, while column renames and type changes vary in whether they rewrite the table. A cheap migration makes the accidental contract change more frequent, not less dangerous.

Where the depth lives

This domain teaches the application-side mechanism and hands the rest off.

API Designresponse-contractsremoving-fieldsbackward-compatibilitydeprecationversioningapi-anti-patterns
Databasedenormalizationrequirements-to-schema
Architecturelayered-architecture
Securitymass-assignmentdata-classificationmulti-tenant-isolation
Domains that do not exist yet
  • — Testing & Reliability Engineering — golden-response tests and schema snapshots, the checks that make an accidental contract change fail in CI rather than in a client.