Data Minimization
Store only what is needed — against the equally correct rule that says keep everything because you cannot recreate it. Both are right, and the resolution is structural.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Do we need to store this field at all, and how do we square that with the rule that says never throw away raw data?
A privacy function that wants the smallest possible footprint, and an analyst who will one day ask a question about a field nobody kept. These two consumers want opposite things and both are legitimate, which is why this lesson exists as a trade rather than as a principle.
The unit of a minimization decision is one field, at one layer, for one horizon. Deciding at table granularity — "we keep raw" or "we drop PII" — is what makes the debate unresolvable, because the correct answer for a delivery address at the raw layer for thirty days differs from the answer for the same field in a mart for five years.
Keep everything. Storage is the cheapest resource in the platform, the data might be useful later, and the one thing you cannot do is recreate history you discarded. Land the full source payload, retain it indefinitely, and let modelling decide what to expose. This is deliberate and defensible advice, and it is what Keeping Raw History: The Recovery Position and the Liability argues for.
Every field kept is a field that must be classified, access-controlled, masked, retained on a clock and reached by every deletion request, forever. The obligation is per-field and permanent; the storage saving from dropping it is not the point (Deletion Requests).
- Every field kept is a field that must be classified, access-controlled, masked, retained on a clock and reached by every deletion request, forever. The obligation is per-field and permanent; the storage saving from dropping it is not the point (Deletion Requests).
- The fields most likely to be kept "just in case" are the ones nobody has a use for, which means nobody notices when they are wrong, and nobody notices when they are exposed (Dataset Documentation).
- An incident's blast radius is measured in fields. A breach of a table holding a token and a behaviour is an incident; a breach of a table holding a name, an address, a birth date and a payment reference is a different category of event, from the same query.
- The full-payload landing zone captures fields the source added for its own reasons — internal flags, debug context, a merged upstream object — that no consumer asked for and no one classified (The Raw Landing Zone).
- Keeping everything and keeping it *usably* are different. An indefinitely retained raw layer that nobody can query because it has no schema history is a liability with no compensating asset (Schema Evolution).
- Minimization applied naively breaks the other way: a team drops a field, and eighteen months later a regulatory question, a fraud investigation or a metric restatement requires it for a period that no longer exists. That loss is total and no engineering can undo it (Backfills).
What is actually happening
- The two rules conflict because they are answering different questions. Minimization asks what obligations you are taking on; the cost of a field is legal and operational and lasts as long as the field does. Keeping raw asks what you can reconstruct; the value of a field is optionality, and it is destroyed irreversibly by deletion. Neither is a subset of the other.
- The conflict is only unresolvable if the unit of decision is the whole dataset. Split the row into identity and behaviour and the tension mostly evaporates: behavioural history is what you cannot recreate and is what analysis needs, while identity is what carries the obligations and is what analysis usually does not need (Surrogate Keys).
- That split is implemented at the ingestion boundary by tokenisation: identity becomes a stable surrogate, the mapping lives in a separately governed vault, and the platform keeps unlimited behavioural history against a token. You keep everything that matters and hold almost none of what is regulated (PII in Pipelines).
- The second structural resolution is tiering by layer: raw retained short because it is the least filtered copy, modelled layers retained long because they are minimized, classified and useful. This inverts the usual instinct, which retains raw longest because it is cheapest to keep, and it is the single most valuable inversion in this area.
- The third is generalisation: keeping the attribute and dropping the precision. A birth year, a postcode district, a region, an hour bucket. This preserves most analytical value at a fraction of the identifying power, and unlike deletion it is a design decision rather than a loss (Grain: What Does One Row Represent?).
- Purpose limitation is the version of this that engineers usually skip: a field kept for one purpose is not automatically available for another, and platforms that ignore this end up with a marketing use of data collected for fulfilment, which is a legal problem rather than a technical one.
Two rules that are both correct
This is the only place in the domain where two well-established pieces of advice point in opposite directions and both survive scrutiny. "Keep raw data forever, because you cannot recreate it" is correct: every serious backfill, every metric restatement and every "what actually happened in March" question depends on it (Keeping Raw History: The Recovery Position and the Liability). "Store only what you need" is also correct: every field you hold is an obligation you carry until you delete it, and an incident's scope is the union of the fields involved.
Engineers usually resolve this by picking a side according to temperament, and both choices are wrong in the same way — they apply a single answer to a whole dataset. The productive move is to notice that the two rules are talking about different parts of the row. Keeping-raw is an argument about *behaviour*: the events, the amounts, the timestamps, the sequence. Minimization is an argument about *identity*: who the row is about.
Once the row is split, the trade largely dissolves. You can keep unbounded behavioural history against a token and hold almost no identity at all. What remains is a genuine residue — cases where analysis needs the real value — and that residue is small enough to govern properly, which is the whole objective.
Land the complete source payload in raw, retain it indefinitely, and drop or mask personal fields when building staging and marts. The warehouse looks clean, the marts contain no direct identifiers, and the privacy review passes on the layer that was shown.
Allow-list the fields that cross into the platform, tokenise direct identifiers against a separately governed vault at that boundary, generalise precision that is not needed, and retain raw for a horizon sized to your realistic backfill range rather than indefinitely.
A field dropped downstream is still present in the raw layer, which means it is still classified, still in scope for every deletion request, still in the backup, and still readable by anyone with lake access. The obligation attaches to the bytes, not to the modelled table, so minimization applied after landing reduces visibility rather than exposure.
What a field actually costs you
The reason "storage is cheap" wins arguments is that storage is the only cost anyone measures. The recurring cost of a retained personal field is mostly not storage, and the drivers below are relative weights within that ongoing cost rather than a budget.
Read the shape rather than the bars. The largest contributors are the per-field obligations that repeat forever — every deletion request must reach this field, every access review must consider it, every classification change must be applied to it — and they scale with the number of governed fields and the number of copies, not with the size of the data.
This is also why minimizing late is so much more expensive than minimizing at the boundary. A field dropped before it lands has zero copies to find; a field dropped after two years of pipelines has as many copies as the platform has stages, plus the operational ones nobody enumerated (PII in Pipelines).
Every erasure must locate and rewrite this field in every copy, forever. Scales with copy count and request rate, and it is the driver that never goes away.
Raw, staging, marts, extracts, logs, backups and training sets. Each copy multiplies every other driver in this list.
Not a running cost but a tail risk. The scope of a breach is the union of fields exposed, so each field widens the worst case.
A classified field pulls its dataset into review cycles and slows every access grant that touches it, permanently.
Policies must be written, tested and kept correct through every schema change to the field or its derived columns.
The only driver anybody measures, and typically the smallest. A narrow column, however long it is retained, is a small share of a platform's storage.
Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.
Weights are relative to the largest driver in this comparison and describe shape, not magnitude. The point is the ordering: the cheapest-to-measure driver is the least important one, which is why the argument is usually lost by the side that is right.
Resolving the tension in practice
The three structural resolutions — tiering, tokenising, and separating identity from behaviour — are not alternatives so much as layers of the same design, and most mature platforms end up with all three. The decision below frames them by what the analysis actually needs, because that is the only input that distinguishes them.
The one option people forget is the last: keeping the field and shortening its horizon. Retention is a dial, and a field that must exist can still exist for a bounded period. It is often the answer when the analytical need is real and recent — fraud investigation, support, reconciliation — and the historical need is not.
Whatever is chosen, write it into the dataset's contract next to the schema, with the date it took effect. A minimization decision creates a discontinuity in history, and a consumer who computes a metric across that date will see it change with no bug to find (Data Contracts).
You need history you cannot recreate, and you do not want the obligations. Which resolution applies to this field?
when Analysis needs to link records to a consistent subject over time but never needs the identifier itself. The common case for user and customer keys.
cost A vault in the ingest path, with its own availability and audit requirements. The token still links individuals, so obligations follow it.
when You need reprocessing ability for a bounded window but not an indefinite one, and the modelled layers are minimized enough to keep long.
cost Your maximum backfill range is now the raw horizon. Any bug older than it can only be fixed from modelled data, if at all (What Backfills Break).
when The attribute matters and its precision does not — a region rather than coordinates, a year rather than a date, an hour rather than a millisecond.
cost Precision is unrecoverable, and the coarsening that prevents re-identification depends on the whole row rather than the field, so it must be checked jointly.
when A small number of workflows genuinely need real identity and everything else needs none. Common where fulfilment and analytics share a platform.
cost Two systems, an audited join path between them, and the permanent risk that someone materialises the join for convenience.
when The need is recent rather than historical: investigation, support, reconciliation against a source that also expires it.
cost The obligation is unchanged while the data exists, so this reduces exposure duration rather than exposure surface (Data Retention).
when A legal or contractual obligation requires the field for a defined period, or the analytical need is central and unavoidable.
cost Full obligation, full blast radius, and a documented justification that must be reviewed when the purpose changes. This is a legitimate option and it should be an explicit one.
How to build it
Most important first.
- Decide per field, per layer, per horizon. Write the decision down next to the schema rather than in a policy document, so it is reviewed when the schema is (Data Contracts).
- Tokenise direct identifiers at ingest and keep the behavioural history unbounded. This is the resolution that gives both rules what they actually want, and it must be done at the boundary because retrofitting it does nothing for data already landed.
- Retain raw shortest, not longest. Size the raw horizon to your realistic maximum backfill range plus a margin, and accept that beyond it the modelled layers are your history (Data Retention).
- Prefer generalisation to deletion where the analysis tolerates it. Coarsening is reversible in the sense that you can start keeping precision again; deletion of history is not.
- Drop fields at the boundary, not in the modelled layer. A field dropped in transformation is still in raw, still classified, still in scope for deletion, and has bought nothing (Where the Transformation Actually Runs).
- Review by usage. Query logs tell you which columns have been read in the last year; a personal field with no reads and a long retention is the cheapest minimization decision available and nobody makes it because nobody looks (What Actually Drives Data Platform Cost).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Dropping a field at the boundary guarantees the platform never holds it — for data arriving after the change. It guarantees nothing about the source system, which still holds it, or about data already landed.
- Tokenisation guarantees that the analytical platform cannot resolve a token to a person without the vault. It does not make the data non-personal: a stable token still distinguishes and links individuals, and obligations follow that property, not the presence of an email address.
- Generalisation guarantees a loss of precision and nothing about anonymity. A coarse attribute combined with several other coarse attributes can still be unique to one person in a row (Data Classification).
- Nothing here guarantees you will not need the field later. Minimization is a bet, and the honest framing is that you are trading a certain, ongoing obligation against an uncertain, future analytical need.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The usage audit: join the catalog's column inventory against warehouse query logs for the last year, and list every column tagged
personalwith zero reads. Each row is a field being retained at full obligation for no consumer, and the list is usually much longer than anyone expects. - It misses columns read only by pipelines rather than by humans — a field that feeds a model looks unused by this query and is not — so join against lineage as well as query logs before proposing a drop (Column-Level Lineage).
- It also misses fields that are genuinely needed rarely: an annual regulatory report reads a column once a year, and a one-year window that starts the day after it ran will call it unused.
- Minimization has no effect on freshness in the pipeline sense. It affects historical depth, which is the other axis: how far back a consumer can ask, rather than how recently the data arrived.
- Dropping a field takes effect from the moment of the change forward, so every minimization decision creates a discontinuity in history at a known date. That date must be documented, because a metric computed across it will change shape without any bug (Semantic Changes).
- Tokenisation at ingest similarly creates a before-and-after: records landed before it hold the raw value and records after hold a token, and joining across the boundary requires the vault.
- A source adding fields means the minimization decision is not a one-time event. Every new field arriving in raw is a default-keep unless the boundary explicitly allow-lists, which is the argument for allow-listing rather than deny-listing at ingest (Data Contracts).
- A field that becomes needed after being dropped cannot be recovered for the past, only started for the future. Design the reversal path anyway: know how to turn a field back on and how to communicate the gap to consumers.
- Changing the tokenisation of an existing identifier — a new salt, a new vault — breaks joins across the change boundary in the same way a key change does, and it is a migration rather than a configuration edit (Surrogate Keys).
- There is no recovery from deletion of history. That asymmetry is the reason keeping raw is such strong advice and the reason minimization must be a considered decision rather than a default.
- The mitigation is to make the decision reversible in one direction only, deliberately: keep a full-fidelity copy for a bounded, short window under strict access, and minimize everything beyond it. You lose the ability to answer questions about last year and keep it for last month, which is where most "we should have kept that" moments actually land.
- Where a dropped field is needed for a historical period, the only path is back to the source system, which has its own retention and usually a shorter one. Check that before assuming it exists (Source of Truth).
What can go wrong
- Minimization is declared as a principle and implemented in the modelled layer, where it achieves nothing because raw still holds everything.
- A field is dropped in a hurry during an incident, and eighteen months of history is discovered missing during an audit.
- Tokenisation is applied to the obvious identifier and not to the free-text column that contains it, which happens on nearly every first attempt (PII in Pipelines).
- The vault becomes a bottleneck or an outage-shaped dependency, so a team caches the mapping in the warehouse for performance and undoes the entire design (Caching Patterns).
- Generalisation is applied per field, and the combination of three generalised fields remains unique to individuals — anonymity was assumed rather than tested.
- The usage audit is run once, produces a long list, and nothing is dropped because nobody will sign off on a deletion. The analysis was correct and the mechanism — a default expiry that requires an active decision to extend — was missing.
- "Storage is cheap, so keep everything." Storage is cheap and obligation is not. The recurring cost of a personal field is classification, access review, masking, deletion scope and audit, none of which appear on a storage line (What Actually Drives Data Platform Cost).
- "Minimization contradicts keeping raw history." They conflict only at dataset granularity. At field granularity — tokenise identity, keep behaviour, retain raw short and modelled long — both rules are satisfiable at once, and that is the whole content of this lesson.
- "Pseudonymised data is not personal data." A stable token still distinguishes and links individuals. It reduces risk substantially and it does not remove the obligations, which is a distinction that has surprised a lot of teams in an audit.
- "We aggregate, so it is anonymous." Aggregation is anonymising in proportion to group size. A count of one is a disclosure, and thresholds must be applied explicitly (Data Classification).
- "We can always get it from the source." The source has its own retention, usually shorter than yours, and it is under no obligation to keep a field it stopped using (Source of Truth).
- Minimization is the only governance mechanism that reduces work everywhere else: a field that does not exist needs no classification, no mask, no access review, no retention rule and no deletion handling.
- It is also the mechanism with the most severe irreversible failure mode, which is why it must be a documented decision with a named owner and not a cleanup task.
- The decision belongs at the ingestion boundary, where it is cheap, rather than in the modelled layer, where it is decoration (The Raw Landing Zone).
Operating it
- Count of columns tagged
personalper layer, tracked over time. It should be near zero in the modelled layers and bounded in raw; a rising line in marts means identity is leaking downstream (The Data Catalog). - Read counts per column over a rolling year, from query history, joined to classification. The quadrant of high classification and zero reads is the work queue.
- Retention horizon per layer against actual oldest object per layer, which is the check that the tiering you designed is the tiering that exists (Storage Lifecycle).
- Vault lookup rate and the identity of the principals doing the looking up. A rising rate means the tokenisation boundary is being routed around (Audit Logs for Privileged Actions).
- At 10x fields, per-field decisions must be made by policy — allow-list at the boundary, default-drop for unrequested fields — rather than by review.
- At 100x subjects, the vault becomes a serious system with its own availability, latency and scaling story, and its outage takes ingestion down with it (Ingestion Failure & Recovery).
- More sources multiplies the boundary count. Every new source is a new place to decide what crosses, and a source integrated without that decision defaults to keeping everything.
- More consumers increases pressure against minimization: each new team has a plausible reason to want one more field, and there is no natural counter-pressure unless the cost of a field is made visible (Cost Attribution).
- Retained bytes multiplied by retention horizon is the obvious driver and the least important one. Storage of a narrow column is small; the obligation attached to it is not.
- The dominant driver is per-field obligation: classification, access review, masking policy, deletion scope and audit — all of which scale with the number of governed fields and none of which scale with bytes.
- Tokenisation costs a lookup per record at ingest and a vault to operate continuously. Vault cost scales with subject count and lookup rate, not with the size of the behavioural history.
- Minimizing late costs more than minimizing at the boundary, because every copy made in between must be found and rewritten (Deletion Requests).
- Minimization buys a smaller obligation surface and a smaller incident scope, and costs optionality that cannot be bought back. There is no version of this where you keep both.
- Tokenisation resolves most of the tension and adds a vault: a new system, a new dependency in the ingest path, and a concentration of the risk you spread out before. It is the right trade for most platforms and it is a real trade.
- Short raw retention reduces exposure and directly bounds your maximum backfill window. Any bug older than the raw horizon cannot be fixed from raw, only from whatever modelled layer survives (Planning a Backfill).
- Generalisation preserves analysis and forecloses precision. Deciding the bucket size is deciding which future questions are answerable, and it is usually decided by whoever writes the transformation.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe structural resolutions — split identity from behaviour, tier retention by layer, generalise rather than delete — apply to any platform, because they follow from the shape of analytical data rather than from any tool or regulation.
- ORG-SPECIFICWhether pseudonymised data remains in scope, what purpose limitation requires and how long a field may be justified are legal determinations that differ by jurisdiction and industry; a health platform and an advertising platform will draw these lines in different places from the same engineering.
- SCALE-SPECIFICBelow a few sources, per-field decisions can be made in review and the raw layer can be kept indefinitely without much consequence; above a few dozen the boundary must allow-list by default, because the rate of new fields exceeds the rate at which anyone can decide about them.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns the review path for a change that stops collecting a field. It is an irreversible production change with no rollback, and it deserves the ceremony that implies rather than being a one-line diff in an ingestion config.