GovernanceORG-SPECIFICGENERALTOOL-SPECIFIC

Data Classification

Public, internal, confidential, personal, highly sensitive — what the tiers mean, why the unit is the column, and why classification is worthless unless it propagates.

What actually happens How to build it Can I trust it?

Who needs this, what one row is, and why the obvious build breaks

Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.

The question

What class of data does this column hold, who decided that, and does every column derived from it carry the same answer?

Who needs this

The policy engine, first and foremost — a classification exists so that something can act on it. Then a security reviewer scoping an assessment, a privacy function producing a record of processing, an engineer deciding whether a new dataset may be joined to another, and an analyst who wants to know why a column comes back masked.

What one row is

One column of one dataset, plus the class assigned to it and the provenance of that assignment. Table-level classification is a lossy aggregate of this — it is the maximum class across the table's columns, and applying that maximum to every column is why table-level policies make ordinary work painful without making sensitive work safe.

The obvious build

Define four tiers in a policy, add a classification label to each table when it is created, and require the label at code review. It is a real improvement over nothing, it is cheap, and for a platform with a few dozen human-created tables it is genuinely adequate.

Why it breaks

A staging model written as SELECT * FROM source inherits new upstream columns automatically. The label was set once, the column set was not, and the model now carries a field nobody classified (Schema Evolution).

How it breaks with real data

A staging model written as SELECT * FROM source inherits new upstream columns automatically. The label was set once, the column set was not, and the model now carries a field nobody classified (Schema Evolution).
The column is metadata and its type is JSON. Its class is whatever a producer put in it last week, which is not a schema property and cannot be labelled (Semantic Changes).
Free text defeats it entirely. A support-ticket body, a delivery note, a search query and an LLM prompt log are all text columns that contain whatever a human typed, including things no tier anticipated (PII in Pipelines).
Two columns that are individually innocuous become identifying together. A postcode, a birth date and a gender are each internal; the three of them in one row are frequently unique to a person, and no per-column label expresses that (Data Minimization).
A join reclassifies without touching a label. fct_events was internal; joined to dim_customers for an email address, the derived model holds personal data and kept the event table's class (Column-Level Lineage).
An automatic classifier that samples values reports both false positives (an order reference that looks like a national identifier) and false negatives (an email column that was empty in the sample), and its output is trusted as if it were a declaration.

Source→Ingestion→Raw→Transformation→Validation→Storage Model→Serving→Consumer→Observability

What is actually happening

A classification is metadata attached to a column and consumed by a policy. It has three possible provenances, and keeping them distinct matters: declared by a human who knows the source system, inherited through lineage from a declared column, and inferred by a scanner that looked at values. Inferred is a hint, not a decision, and treating it as a decision is how columns get wrongly downgraded.
Tiers are a total order in most organisations — public, internal, confidential, restricted — and personal data is usually a *cross-cutting tag* rather than a tier, because a personal-data column can be at any confidentiality level and carries obligations (deletion, minimization, purpose) that confidentiality tiers do not express. Organisations that model personal data as a tier lose the ability to say "confidential but not personal", which is most of their financial data.
Special categories — health, biometrics, precise location, political or religious affiliation, credentials, payment credentials — behave differently again: they attract additional obligations regardless of tier, and the practical consequence is a separate tag with a separate policy rather than a higher number — Security Engineering owns the regulatory framing, and secLinks below points at it.
Propagation is a graph problem over column-level lineage. Every derived column takes the maximum class of its inputs, plus any tag its inputs carried. Aggregation is the interesting exception: count(*) GROUP BY country genuinely produces something less sensitive, and any propagation rule that cannot express downgrade-on-aggregation will over-classify the entire warehouse into uselessness (Column-Level Lineage).
Downgrade must therefore be an explicit, reviewed act rather than an inference. The rule that works is: inheritance raises automatically, lowering requires a human declaration recorded against the model.
The scanner works by sampling values and matching patterns and dictionaries. It is a coverage tool: good at finding the column you forgot, bad at deciding what a column means.

The ladder, and the tag that is not on it

ORG-SPECIFICThese five names and their boundaries are one organisation's worked example. Yours will differ in count and in where the lines fall, and the definitions must be written against your own datasets to be applied consistently; what transfers is the tier-plus-tag structure, not the rows.

Almost every organisation ends up with a ladder of confidentiality tiers and a set of cross-cutting tags. The ladder answers "how bad is disclosure"; the tags answer "what special obligations does this attract". Conflating them is the most common structural mistake, and it produces masking policies that fire on the wrong columns.

The table below is a worked example, not a standard. Read the right-hand column first: the obligations are what actually differ between rows, and if two of your tiers have the same obligations then you have one tier with two names.

Notice that the personal-data row is not a rung. A customer email is confidential and personal. A quarterly revenue figure before announcement is confidential and not personal. A support transcript may be confidential, personal *and* special-category, all at once. One ladder cannot say that.

Class	Example column	Who may read it by default	What it obliges you to do
`public`	Published product catalogue attributes	Anyone, including outside the organisation	Nothing beyond integrity — the risk here is tampering, not disclosure
`internal`	Order counts by day, model run metadata	All employees	Access logging only. This is the default tier and therefore the one that quietly absorbs things that do not belong in it
`confidential`	Unannounced revenue, contract terms, salary bands	A named group, granted by the owner	Explicit grants, periodic access review, no export to unmanaged destinations
`restricted`	Payment credentials, credentials, health data	A small named group, usually with additional approval	Everything above plus encryption with separately managed keys, masking by default, and stricter retention
tag `personal`	Email, address, device identifier, precise location	Orthogonal — applies at any tier above	Deletion on request, purpose limitation, minimization, retention justified rather than assumed
tag `special-category`	Health, biometrics, affiliation, precise location	Orthogonal — usually forces `restricted`	Additional legal basis, tighter access, and usually a decision not to copy it into the analytical platform at all

Classification propagates or it decays

A source declaration is a single fact. The warehouse turns it into thousands, because every model that reads that column produces derived columns that should carry the same obligations. Nobody maintains that by hand, so the interesting engineering question is not "how do we label things" but "how does a label travel".

The rule is maximum-of-inputs, with one deliberate exception. A derived column takes the highest tier and the union of the tags of every column feeding it. Aggregation may lower the result, but only as a recorded decision by the model's owner, because "this GROUP BY makes it anonymous" is a claim that is frequently false at small group sizes.

The chain below is what that looks like for one column. The couldCorrupt entry on each hop is the specific way the label goes wrong there — and note that two of the four are silent downgrades rather than losses of the label entirely, which is worse because the column still looks governed.

One classification travelling from source to dashboard

`app.users.email` (source)
holds Declared confidential + personal by the team that owns the operational schema.
could corrupt Declaration made once at table creation and never revisited when the column's use changed.
↑ reads from
`raw.users` (landing)
holds A verbatim copy, inheriting the declaration.
could corrupt Landing as a JSON blob, so there is no email column to carry a label — the class attaches to an opaque payload field and stops being actionable.
↑ reads from
`stg_users` (staging)
holds Typed, renamed columns; class inherited per column through the rename.
could corrupt A SELECT * that picks up a newly added backup_email column with no declaration, which then inherits the model's table-level label instead of the source column's.
↑ reads from
`dim_customers` (dimension)
holds One row per customer, still carrying the personal tag on the contact columns.
could corrupt A concatenated customer_label built from name and city that is personal data with no personal ancestor the parser recognised.
↑ reads from
`fct_orders_enriched`
holds Order facts joined to customer attributes — the join is where the fact table becomes personal data.
could corrupt Retaining the fact table's original internal label because the join added columns without triggering reclassification.
↑ reads from
`rpt_orders_by_region`
holds Aggregated counts by region; a legitimate downgrade candidate.
could corrupt Downgrading by rule rather than by decision, so a region with a single customer is published as anonymous when it identifies exactly one person.

Two of these hops lose the label by structure (a blob, an unparsed expression) and two lose it by policy (a table-level label, an automatic downgrade). Only the structural ones are visible in a lineage graph.

The column that arrives unclassified

The routine event that breaks classification is not a redesign. It is a product team adding a field. The migration is correct, the pipeline does not fail, the schema check passes because the change is additive, and a new column appears in the warehouse carrying whatever the model it landed in was labelled (Backward Compatibility).

The diff below is the specific case worth internalising: a support-contact field added to a users table. Every consumer keeps working. One of them starts holding personal data it is not authorised to hold, and the only evidence is a column name in a schema nobody diffed.

The fix is structural rather than procedural. Additive changes must produce an unclassified column that the policy engine treats as blocked until declared — which will annoy people, and is the only version of this that works. A process that relies on the producing team remembering to classify has the same reliability as every other process that relies on remembering.

An additive change that reclassifies a dataset without reclassifying anything

Before

`user_id` — `internal`
`created_at` — `internal`
`email` — `confidential` + `personal`
`plan_tier` — `internal`

After

`user_id` — `internal`
`created_at` — `internal`
`email` — `confidential` + `personal`
`plan_tier` — `internal`
`emergency_contact_phone` — **unclassified**

change The product team adds an optional emergency contact phone number. The migration is additive and backward compatible; every downstream model continues to run.

Consumer	Effect	How it shows up
`stg_users`, built with `SELECT *`	Gains the column immediately and applies its own table-level `internal` label to it.	Silently — no error, wrong result
Masking policy keyed on the `personal` tag	Does not fire, because the new column carries no tag. The phone number is returned in clear text to everyone with table access.	Silently — no error, wrong result
Retention rule for personal data	Does not apply. The column is retained under the general internal horizon, which is longer.	Silently — no error, wrong result
Deletion-request tooling, which enumerates personal columns	Omits the column, so a completed erasure leaves the phone number behind.	Silently — no error, wrong result
Analysts	See a new column and start using it, creating downstream dependencies before anyone has classified it.	Loudly — it raises
Classification scanner, on its next run	Flags a phone-number pattern and raises a finding — days or weeks after the column arrived and after copies exist.	Loudly — it raises

How to build it

Most important first.

Classify at the column level and derive the table's class, never the other way round. Every masking, row-filter and minimization decision downstream needs column granularity, and a table-level label cannot be refined back into one.
Model personal data as a tag orthogonal to the confidentiality tier, and give special categories their own tags. Two axes cost almost nothing to implement and remove the most common modelling failure in this area.
Propagate by taking the maximum over inputs at every derived column, automatically, and require an explicit reviewed exception to lower a class (Data Lineage).
Treat a lineage gap as unclassified-and-blocked rather than unclassified-and-permitted. A Python model whose column mapping cannot be resolved should raise a governance finding, not quietly produce internal columns (Fail Open vs Fail Closed).
Run the scanner continuously against samples of new and changed columns, and route its output to the owning team as a question rather than into the policy engine as a fact.
Write the tier definitions with concrete examples from your own datasets. Abstract definitions ("data whose disclosure would cause serious harm") produce inconsistent labels; "customer email is confidential + personal; aggregated order counts by country are internal" produces consistent ones.

What this actually promises

Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.

A classification guarantees that a claim about a column exists. It guarantees nothing about the values actually in that column — a text column labelled internal can contain anything a user typed.
Inherited classification is exactly as complete as column-level lineage is. Where lineage is derived from parsed SQL, coverage stops at the first transformation the parser cannot read, and that boundary is invisible in the resulting labels (Column-Level Lineage).
Scanner output guarantees only that a pattern matched or did not match in a sample. Absence of a match is not evidence of absence, especially for columns that are usually null.
Nothing here guarantees that two datasets with the same class are equally sensitive together. Combination risk is a property of the join, not of either column (Data Minimization).

Can I trust it?

A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.

The check that would catch this

The propagation test: for every model, compare the classification of each output column against the maximum classification of the columns it reads. Any output lower than its inputs must have a recorded, reviewed downgrade. Anything else is a finding.
It misses the source declarations being wrong, and it misses everything downstream of a lineage gap — a model the parser could not read produces no comparisons at all, so it passes by having nothing to check.
It also cannot see combination risk. Three internal columns that jointly identify a person will satisfy the test perfectly.

Freshness

Classification lags schema. The relevant window is between a column appearing and being classified, and the design question is whether that window is bounded by a blocking check at publish time or by a scan schedule.
A scan-based approach means a new sensitive column is unclassified until the next scan, which is exactly the period in which a new pipeline is most likely to be copying it somewhere.
Reclassification after a semantic change has no trigger at all unless someone raises it, which makes it the slowest-moving and least reliable freshness in the whole governance area (Semantic Changes).

When the schema or meaning changes

Adding a column upstream is the highest-risk routine event here, because SELECT * propagation carries it into models whose labels were set for a different column set (Backward Compatibility).
Renaming a column breaks the link between a declaration and the thing it described. Classification keyed on column name is fragile for exactly the reason every other name-keyed policy is; keying on a stable column identity where the catalog offers one is better (Breaking Schema Changes).
Tier definitions themselves evolve, and when they do the change is retroactive across everything already labelled. This is the argument for storing the tag and evaluating policy at query time — a redefinition is then a policy change, not a data migration.

How to re-run this safely

Recovering from under-classification means finding every copy made while the label was wrong, which is the same problem as a deletion request and is solved by the same inventory (Deletion Requests).
Recovering from over-classification is a review, and is cheap — but the intermediate damage is that people worked around the mask, so check for the extracts created during the period (Data Masking, Tokenisation & Encryption).
Re-deriving lost classifications from lineage works and is worth automating: walk upstream to the nearest declared column and take its class. It produces conservative answers, which is the correct direction to be wrong in.

What can go wrong

Failure modes

Everything is labelled internal because that was the default, so the labels carry no information and no policy can act on them.
Everything sensitive is labelled correctly and nothing derived is, because propagation was never built.
The classifier's inferences are written directly into the policy engine, and a false negative on a sparsely populated column downgrades a genuinely personal field.
Personal data is modelled as a confidentiality tier, so masking policies fire on financial data that is confidential and not personal, and analysts learn that masking is noise.
Free-text columns are classified by what they were designed to hold rather than what they contain, which is how personal data ends up in a dataset that every engineer can read (PII in Pipelines).
Labels exist, propagate and are correct — and no enforcement point reads them, which is the failure the previous lesson is about (Data Governance).

Misreads

"Hashed identifiers are not personal data." A deterministic hash of an identifier still distinguishes individuals and still links records across datasets, which is what the obligations attach to. Whether it is reversible is a separate question with an uncomfortable answer (Data Masking, Tokenisation & Encryption).
"Aggregates are not sensitive." An aggregate over a small group can identify its members — a count of one is a disclosure. Aggregation reduces sensitivity in proportion to group size, which means it needs a threshold, not an assumption.
"Internal means safe." Internal means the whole company, which in most organisations is a larger and less vetted population than the label implies. Most real exposure is internal (Least Privilege).
"We classified the tables." Then you classified the maximum and applied it uniformly, and the first analyst who hits a masked order id will start asking colleagues for extracts.
"The classifier found all the PII." It found what its patterns match in what it sampled. Free text and encoded values are outside both.

Privacy, retention and access

Classification is the input to every other mechanism in this module: masking policies, row filters, retention horizons and deletion scope are all expressed in terms of it. Its errors therefore propagate into all of them, in the same direction.
The provenance of a classification — declared, inherited or inferred — should be stored alongside it and visible to anyone acting on it. A policy engine treating an inference as a declaration is the specific mechanism by which automated classification makes things worse.
Combination risk is a genuine gap in per-column classification and should be handled where it lives: in review of the model that performs the join, not in the labels of the columns being joined (Data Minimization).

Operating it

How you see it in production

Distribution of classes across columns. A histogram with one enormous bar is a labelling failure regardless of which bar it is.
Count of columns whose class was inherited versus declared versus inferred, tracked over time. A healthy platform is mostly inherited, with a small stable set of declarations at the sources.
Count of models where output class is lower than input class without a recorded downgrade — the direct measure of propagation health (Impact Analysis).
Scanner hit rate on columns already classified at or above the matched pattern's class. A rising rate of surprises means new sources are arriving faster than declarations.

What changes at 10x and 100x

At 10x columns, declaration by humans is already impossible except at sources, and the design must be source-declared plus inherited.
At 100x, the binding constraint is lineage parse coverage. Every percentage of models the parser cannot read becomes a subtree of unclassified columns, and those subtrees are where the platform's risk concentrates.
More source systems scales declaration work linearly and combination risk super-linearly, because each new source adds potential joins against everything already present (Data Minimization).

What drives cost here

Scanning costs are sampling costs: bytes read per column per scan, multiplied by column count and scan frequency. Sampling a bounded number of rows per column keeps this flat as data volume grows, which is why the scan should be per-column and not per-table.
Propagation costs are lineage costs — parsing every model and maintaining a column-level graph, which grows with model count rather than data volume (Column-Level Lineage).
The real cost is over-classification: masked columns that did not need masking push analysts toward extracts and copies, and every copy is a new governance liability. Classifying too high is not the safe direction it appears to be.

What this approach costs

Two axes — tier plus tags — are more expressive and more work to operate than a single ladder. The single ladder is genuinely simpler and cannot express "confidential, not personal", which is the distinction most masking policies need.
Automatic propagation with maximum-of-inputs is conservative and will over-classify aggregates unless downgrade is supported, and supporting downgrade introduces a human decision that can be wrong.
Blocking publication on missing classification is the only thing that keeps coverage high, and it makes the platform harder to move quickly in. Pipelines will be blocked at inconvenient times, and that is the cost of the guarantee.

Where this applies

Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.

ORG-SPECIFICThe number of tiers, their names and their boundaries are defined per organisation and per regulator. Four tiers plus a personal-data tag is a common shape, not a standard, and a design that hard-codes tier names into pipeline code will not survive contact with a second organisation.
GENERALColumn granularity, provenance tracking, maximum-of-inputs propagation and the inference-is-not-declaration rule hold everywhere, because they follow from how derived data is produced rather than from any particular regulation.
TOOL-SPECIFICWhether classification propagates automatically depends entirely on the catalog and its column-level lineage support; some tools parse SQL and infer column mappings, others require tags to be declared per model, and the operational burden differs by an order of magnitude between the two.

Where the depth lives

This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.

Backendmulti-tenancy

API Designscopes

Securitydata-classification least-privilege cia-triad risk-and-residual-risk

Cloudkms-and-encryption

Domains that do not exist yet

— DevOps / Production Engineering owns the delivery path for tier definitions and policy code. A change to what restricted means is a change that alters query results across the platform, and it needs staged rollout and a rollback plan like any other.