Data Products
Owner, schema, semantics, quality, documentation, SLO, access policy. Seven commitments, and what a team has to start doing on the day it makes them.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
A team publishes a table that other teams read. What has to be true before that table is a product rather than a shared file with a name on it?
Whoever reads it without being able to ask the person who built it: an analyst joining it to two other datasets, a downstream pipeline that will fail at 03:00 when its shape changes, a model training on last year of it, and an agent retrieving from it with no way to notice that it is stale. What every one of them needs is not more rows — it is a stated grain, a stated freshness, a stated meaning per column, and a name to call (Who Actually Consumes This Data).
The unit is one published dataset at one version, with a declared grain of its own. That nesting is the point: a product must state what one of *its* rows represents, and a product whose grain statement is missing cannot be joined safely by anyone who did not build it (Grain: What Does One Row Represent?).
Add an owner column to the catalog, point it at whichever team seems closest, and call the warehouse's existing tables data products. This is where nearly every organisation starts and it is not stupid — it makes ownership visible for the first time, and visible-but-nominal ownership is genuinely better than none.
The owner is a distribution list. When freshness breaks at 03:00 nobody is paged, because a mailing list has no rota, and the first report of the incident is an analyst at 09:30 (Data Ownership).
- The owner is a distribution list. When freshness breaks at 03:00 nobody is paged, because a mailing list has no rota, and the first report of the incident is an analyst at 09:30 (Data Ownership).
- The SLO was written once, in a wiki page, and nothing measures it. A dataset that has been eleven hours late every day for a month still shows a four-hour SLO, and the document is treated as the fact (The Freshness SLO).
- The schema is declared and the semantics are not.
revenueis documented as "revenue"; whether it is gross or net, whether it includes tax, and which timestamp puts a row in a day are all unwritten, so two teams compute two different correct numbers (Semantic Changes, The Metrics Layer). - The access policy is whatever the warehouse default was when the schema was created, which in practice means every employee can read a table containing customer email addresses, and nobody made that decision (Data Access Control).
- Documentation was written at launch and the table has changed four times since. A stale description is worse than none, because it is believed (Dataset Documentation).
- The product has no consumers. Publishing was easy, retiring is nobody's job, and the catalog now lists three tables with the same name and different suffixes (Data Discovery, Storage Lifecycle).
What is actually happening
- A product is a promise plus a mechanism that keeps it. Each of the seven commitments is a sentence anyone can write in an afternoon, and each one is only real once something in the platform fails when it is broken. That is the entire difference between a data product programme that works and one that produced a spreadsheet of owners (Contract Enforcement).
- What changes operationally is that publishing becomes a release. A table that is only read by the team that wrote it can be altered at will; a product has consumers who are not in the room, so a change to it is a deployment with a blast radius, a compatibility question and an announcement (Breaking Schema Changes, Impact Analysis).
- The owner commitment is the one that costs real money, because it means an on-call rotation for a dataset. Everything else on the list is work; this one is a rota, a pager and a budget line, and it is the commitment organisations most often make in name only (Data Ownership).
- The SLO commitment converts freshness from a property people observe into a number the team is accountable for. It also forces the pipeline to be measured end to end rather than task by task, because an SLO stated against the consumer's clock cannot be satisfied by a green DAG (Pipeline SLOs, Freshness Monitoring).
- The semantics commitment is the one with no technical enforcement anywhere. Types are checked by the engine; meaning is checked by a human reading a sentence somebody wrote. That asymmetry is why semantic drift is the failure this whole domain keeps returning to (Semantic Changes).
- The access-policy commitment forces classification to happen at publish time rather than at audit time. A product that declares what it contains can be governed automatically; a table that does not has to be reviewed by a person who was not there when it was built (Data Classification). Taken together the seven turn a dataset into something a stranger can evaluate without talking to anyone — which is the actual capability being bought, and the reason this matters more the more teams there are (The Data Catalog).
Seven commitments, and the mechanism that makes each one real
The list is easy to agree with and easy to fake, so the column that matters here is the third one. Every commitment has a mechanism that makes it enforceable and a hollow version that satisfies an auditor and nobody else, and the hollow version is what most "data product" initiatives actually ship.
Read the fourth column as a diagnostic. If you can look at your own catalog and recognise three of those descriptions, the programme is at the vocabulary stage — which is a normal place to be, and is fixed by choosing one commitment and making it mechanical rather than by adding an eighth.
Notice which commitment has no automated enforcement anywhere: semantics. A type system checks types; nothing checks that revenue means net of refunds. That gap is why semantic descriptions have to be reviewed by a human at publish time, and why the review is worth the friction (Semantic Changes).
| Commitment | What it means concretely | The mechanism that makes it real | What "declared but not real" looks like |
|---|---|---|---|
| Owner | A named team that is paged when this dataset breaks and that decides its roadmap. | An on-call rotation with the dataset attached to it, and alert routing that reaches a human at 03:00. | A distribution list, a departed employee, or "data team" — a field that identifies nobody who can act (Data Ownership). |
| Schema | The declared fields, types and nullability, versioned, with a compatibility policy for changes. | A contract checked at publish time, so an incompatible change fails the way a failing test fails (Data Contracts). | A schema that is whatever the last transformation happened to produce, discovered by consumers reading information_schema. |
| Semantics | What one row represents, and what each column means — units, inclusions, which timestamp decides a row's day. | A written grain sentence and per-column descriptions, reviewed by a human before first publish and on any change of meaning (Grain: What Does One Row Represent?). | revenue: "the revenue". Present, non-empty, and carrying no information a stranger could use. |
| Quality | The assertions that must hold, run before consumers see the data, with thresholds someone chose. | Tests in the publish path that block promotion on failure, not tests that write to a log (Data Tests). | Tests that exist, run after publish, and alert into a channel nobody reads (Alert Fatigue: The Page Nobody Reads). |
| Documentation | How to use it, what it is for, what it is not for, and the known gaps. | Generated from the same declaration the gate checks, so it cannot drift from the schema it describes (Dataset Documentation). | A wiki page written at launch, four schema changes ago, and believed because it exists. |
| SLO | A freshness and completeness promise stated against the consumer's clock. | Continuous measurement of achieved versus promised, published beside the promise (The Freshness SLO). | A number in a document that nothing measures, on a dataset that has missed it daily for a month. |
| Access policy | Who may read it, at what granularity, with what masking and what retention. | A declared classification that the platform turns into grants, masking and lifecycle rules automatically (Data Classification). | Whatever the warehouse default was when the schema was created — a decision nobody made and everybody inherited (Data Access Control). |
The commitments are ordered by what they cost. The first is a budget decision, the middle five are engineering work, and the last is a governance decision — which is why programmes that start at the top stall and programmes that start in the middle make visible progress.
Publishing becomes a release
The operational change that surprises teams is not the documentation. It is that a table they used to CREATE OR REPLACE at will now has consumers who are not in the room, which turns every change into a deployment with a blast radius.
The stages below are what a publish looks like once that is true. Read the guarantees column: the gate stage is the only one that promises anything about the seven commitments, and the publish stage is the only one that promises consumers never see a half-written state. Everything else is ordinary pipeline work that was happening anyway (Atomic Publish).
The last two stages are the ones that get skipped, and they are the ones the model exists for. A product nobody announced is a product whose consumers will discover the change by breaking, and a product with no deprecation path is a product that can never be retired — which is where large catalogs of unretirable tables come from (Impact Analysis).
- 1Build candidate
Runs the transformation into a location consumers are not reading.
guarantees Nothing about correctness. Only that a candidate exists and the code ran (The Pipeline Succeeded. The Data Is Wrong.).
fails by Writing directly into the served location, which makes every later stage advisory rather than protective.
- 2Validate data
Runs the declared tests: grain uniqueness, referential integrity, volume against history, freshness of inputs.
guarantees That the assertions someone wrote hold on this candidate. Never that the data is right (Data Tests).
fails by Running the tests and publishing anyway, because failure was wired to a notification instead of to a gate.
- 3Conformance gate
Checks the seven commitments are present and non-vacuous before the candidate may be promoted.
guarantees That a consumer will find an owner, a grain sentence, semantics, tests, docs, an SLO and a classification.
fails by Checking presence rather than content, so
grain: variouspasses and the gate certifies nothing. - 4Publish version
Makes the candidate the served version atomically — a swap, a pointer move, or a metadata commit.
guarantees Readers see the whole previous version or the whole new one, never a partial state (Atomic Publish).
fails by Publishing partition by partition, so a consumer reading mid-publish sees a period that is genuinely incomplete and looks merely low.
- 5Announce
Records the version, its changes and their compatibility class where consumers and the catalog can see them.
guarantees That a change is discoverable before it is felt — but only to consumers you can enumerate (The Data Catalog).
fails by Relying on a voluntary consumer registry, which is wrong within a quarter and wrong in the direction that matters.
- 6Measure the SLO
Compares achieved freshness and completeness against the promise, continuously, and publishes both.
guarantees That the commitment is a fact rather than a document (Freshness Monitoring).
fails by Measuring pipeline runtime instead of data age, which is green during exactly the incidents that matter.
- 7Deprecate
Announces a retirement date, tracks consumer migration, then removes the version.
guarantees That the catalog can shrink. This is the only stage that makes retirement possible at all (Storage Lifecycle).
fails by Having no enumerated consumer list, so nobody dares delete anything and the catalog only ever grows.
Two stages are new relative to an ordinary pipeline — the conformance gate and the deprecation path — and they are the two that decide whether a product programme produces a catalog people trust or a catalog people avoid.
What the gate catches, and what it never will
Conformance checking is mechanical, cheap and easy to over-trust. Every row below runs in the publish path and every one of them has a blind spot that a competent team walks into, because the check can only see whether a declaration exists and roughly what shape it has.
The row worth sitting with is the grain one. A grain statement is the highest-value sentence in a data product and no automated check can tell whether it is true — asserting uniqueness on the key the statement names catches a contradiction, but only if the statement named a key at all, and a wrong grain statement with a matching uniqueness test is perfectly self-consistent and completely wrong (Grain: What Does One Row Represent?).
The last row is the one that changes behaviour rather than catching a defect. Measuring achieved SLO against promised SLO turns the freshness commitment from something a team states into something a team is answerable for, and it is the single check most likely to change how a product is operated.
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| All seven commitments are present and each exceeds a minimum content bar — owner resolves to a rota, grain is a sentence, every column has a description. | A stranger can evaluate this dataset without talking to anyone. | Datasets promoted by copying a template, missing classifications, owner fields pointing at a mailing list or a team that no longer exists. | Declarations that are well-formed and wrong. A confident, fluent, incorrect grain sentence passes every content bar you can automate. |
| Uniqueness holds on the key named by the grain statement. | The declared unit is actually the unit. | A join that fanned out, a re-run that appended, a dimension that gained duplicate rows and multiplied every fact joined to it (Duplicate Rows). | A grain statement that names the wrong key. The test then passes on a unit nobody downstream expects, and every join is silently at the wrong grain. |
| Schema change against the previous published version is classified as additive, compatible or breaking. | Consumers can be told what a new version does to them. | Removals, renames and retypes shipped without a deprecation window (Breaking Schema Changes). | Every change of meaning that leaves the schema identical — a redefinition of revenue, a change in which timestamp decides a row's day (Semantic Changes). |
| Every column carrying personal data has a classification, and the classification produces the grants and masking actually applied. | The access policy is enforced rather than described. | A new column added to a governed product without classification, and drift between the declared policy and the grants in the warehouse (Data Access Control). | Personal data in a column nobody thought was personal — free-text notes and error payloads are where it usually is (PII in Pipelines). |
| Achieved freshness versus promised SLO, per product, over a rolling window. | The promise is being kept. | An SLO written once and never met, a pipeline that has been silently late since a schedule change, a product whose upstream got slower (The Freshness SLO). | Fresh data that is wrong. Freshness and correctness are independent, and a product that is always on time can be always incorrect. |
Four of the five check a declaration against reality. That is the shape of the whole model: the commitments are only worth something because something compares what a team said to what a team did, and every remaining blind spot is a claim about meaning that only a person can adjudicate.
How to build it
Most important first.
- Split the namespace in two before anything else. Internal working tables are private and may change freely; published products live somewhere separate and carry all seven commitments. Without that line every table is either over-governed or under-governed, and teams stop creating intermediate models to avoid the ceremony (Raw, Staging, Curated: Layers by Purpose).
- Make the seven a publish-time gate, not a review checklist. A publish that lacks an owner, a grain statement, a semantic description per column, at least one test, an SLO, documentation and a classification should fail the way a failing test fails (Contract Enforcement).
- Write the grain as one sentence in the dataset's own documentation: "one row is one settled payment for one order line, at the time the payment provider confirmed it". If that sentence cannot be written, the dataset is not ready to be a product and the modelling is not finished (Grain: What Does One Row Represent?).
- Measure the SLO from the consumer's side — the age of the newest complete record versus now — and publish the measurement next to the promise. An SLO nobody measures is a sentence; an SLO measured and displayed is a commitment (The Freshness SLO).
- Enumerate consumers automatically from query logs and lineage rather than asking people to register. Voluntary registration produces a list that is wrong within a quarter, and the whole deprecation process depends on that list being right (Data Lineage, Impact Analysis).
- Version the product and give it a deprecation procedure with a date attached, because the ability to retire a product is what stops the catalog filling with things nobody will admit to owning (Storage Lifecycle). Treat the first consumer as a design partner rather than a recipient. A product designed without one is a modelling exercise, and modelling exercises are where unused datasets come from (Who Actually Consumes This Data).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- A product guarantees exactly what its contract says and nothing implied by its name.
orders_dailyguarantees one row per order per day only if that sentence appears somewhere a consumer can read (Data Contracts). - The seven commitments guarantee evaluability, not correctness. A well-formed product can be completely wrong about its domain; what the commitments buy is that a consumer can tell what it is claiming and check the claim (Data Quality).
- The SLO is a promise about freshness and, if written properly, about completeness at that freshness. It says nothing about the accuracy of any value in the dataset (The Dimensions of Data Quality).
- Nothing about being a product guarantees stability of meaning. Semantics can change while every declared field stays identical, which is why the semantic description belongs in the contract and in the change process (Semantic Changes).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check this arrangement needs first is a conformance check run by the platform at publish time: does the candidate declare an owner, a grain sentence, per-column semantics, a classification, an SLO, documentation and at least one data test? It is cheap, it is mechanical, and it is the only thing standing between a product programme and a spreadsheet (Data Tests).
- It misses the product that declares all seven and declares them badly. "One row is one order" attached to a table with one row per order line passes the check and destroys every revenue number built on it (Grain: What Does One Row Represent?).
- Pair it with an SLO conformance measurement — the achieved freshness versus the promised freshness, per product, over a rolling window — because the commitment that decays first and most silently is the one nobody measures (Freshness Monitoring).
- Becoming a product does not make a dataset fresher. It makes its freshness stated and measured, which usually reveals that the number people assumed was hours is in fact a day, and that revelation is most of the initial value.
- The SLO should be expressed against the event that matters to the consumer — "orders placed before midnight are complete by 06:00" — rather than against the pipeline's own schedule. A schedule is an implementation detail and consumers should not have to know it (The Freshness SLO).
- Products that depend on other products inherit the worst freshness in their chain, and the inheritance is invisible unless the platform computes it. Publishing a product with a four-hour SLO on top of an input that arrives daily is a promise that was never available (Data Lineage).
- A published product is a public interface. Additive changes are safe; renames, removals and retypes need a deprecation window with a date and an enumerated consumer list (Backward Compatibility, Schema Evolution).
- The change that needs the most process is the one with no schema footprint: redefining what a column means, or changing the grain. Both are breaking changes that every automated compatibility check will pass (Semantic Changes, Breaking Schema Changes).
- Version the product rather than mutating it when a change cannot be made compatibly. Two versions running side by side with a retirement date is more work than an in-place change and it is the only approach that does not require every consumer to move on your schedule (Atomic Publish).
- Ownership itself evolves: teams merge, split and are reorganised, and products outlive them. A re-assignment procedure is part of the model, or every reorganisation produces orphans with a stale name in the owner field (Data Ownership).
- Recovery is the ordinary discipline of this domain — immutable raw inputs, deterministic transformations, rebuild into a new location, validate, then swap (Reprocessing vs Retrying, Atomic Publish).
- What the product framing adds is that a correction is a consumer-visible event. If yesterday's figures were wrong and have been republished, every consumer needs to know, because some of them copied the number into a document that will not be recomputed (Validating a Backfill Before You Publish).
- Keep the previous published version until the new one has been validated by a consumer rather than only by a test. Rollback for a data product means pointing consumers back at a version that still exists, and it exists only if you decided in advance not to delete it (Rolling Back Data).
What can go wrong
- Ownership that is nominal: a name in a field, no rota, no budget, and every incident routing to the platform team by default (Who Owns Data Quality).
- An SLO that is written and never measured, which converts a commitment into a decoration and makes the platform look healthier than it is.
- Documentation that describes a previous version of the dataset and is trusted precisely because it exists (Dataset Documentation).
- A product whose grain statement is wrong, so every downstream join is at the wrong unit and every check passes (Grain: What Does One Row Represent?).
- Product proliferation: publishing is easy, retiring is unowned, and the catalog fills with near-duplicates nobody will delete (Data Discovery).
- The mitigation failing: a conformance gate that checks for the *presence* of the seven fields rather than their content, so
owner: data-teamandgrain: variouspass and the gate certifies nothing.
- "A data product is a table with an owner." Ownership is one of seven commitments and the other six are what make it evaluable. A table with an owner and no grain statement is still unjoinable by anyone who did not build it (Grain: What Does One Row Represent?).
- "Data products require a mesh." They are the unit of publication in any arrangement, central or federated. A central team can publish products, and doing so is the cheapest improvement available to most central warehouses (Data Mesh, The Central Warehouse).
- "We documented it, so it is a product." Documentation is one commitment, and the one that decays fastest. Without a measured SLO and a real owner it describes a dataset nobody is accountable for (Dataset Documentation).
- "The product is the pipeline." The product is what a consumer can rely on. The pipeline is an implementation detail the owning team is free to replace, and a consumer who depends on the pipeline's intermediate tables is depending on something nobody promised (Data Contracts).
- "Everything in the warehouse should be a product." Most tables are working intermediates and should be private. Promoting all of them makes the label meaningless and the gate hated (Raw, Staging, Curated: Layers by Purpose).
- The access-policy commitment is what makes governance mechanical rather than periodic: a product declares its classification, and the platform applies masking, row filtering and retention from that declaration (Data Classification, Row and Column Security).
- Retention belongs in the contract too. A product whose retention is "whatever the warehouse default is" is one that will be found by an audit rather than by its owner (Data Retention).
- A deletion request has to reach every product derived from the one holding the subject, which is why the enumerated consumer list is a governance artefact and not only an engineering convenience (Deletion Requests, Data Lineage).
Operating it
- Achieved versus promised freshness per product, over a rolling window, published where consumers see it rather than where the owning team sees it (Freshness Monitoring).
- Products whose named owner has acknowledged an alert in the last quarter, as a fraction of all published products. This is the honest ownership number and it is always lower than the owner column suggests (Data Ownership).
- Consumers per product, derived from query logs and lineage rather than from a registration form. Zero-consumer products are pure cost and a growing count of them is the first sign that publishing is easier than retiring (Data Lineage, Cost Attribution).
- Conformance rate at publish time, broken down by which of the seven commitments fails most often. In most organisations it is semantics, and knowing that tells you what to invest in (The Data Catalog).
- Time from a breaking change being proposed to every consumer being notified. If that is measured in days because the consumer list is manual, the deprecation process is theatre (Impact Analysis).
- Below a handful of datasets and one team, the seven commitments are ceremony: everyone knows the grain, the owner is sitting there, and the SLO is "when Ana runs it". Writing them down still helps, but a gate that fails a publish is overhead (The Central Warehouse).
- The value climbs with consumer count and team count, not with data volume. The commitments exist so that a stranger can evaluate a dataset without a conversation, and strangers only appear above a certain organisational size.
- At high product counts, discovery and deprecation become the binding constraints. Finding the authoritative product for a concept is harder than building a new one, which is exactly the dynamic that produces four hundred datasets (Data Discovery, The Self-Service Data Platform).
- At high consumer counts, the enumerated consumer list stops being maintainable by hand and the lineage-derived one becomes load-bearing, because a deprecation you cannot announce accurately is a deprecation you will not do (Impact Analysis).
- The dominant cost is ongoing product work that the producing team was not previously doing: documentation, SLO measurement, deprecation notices, consumer questions. It is recurring, it is unglamorous, and it is almost never resourced when the programme is announced (Agreement Costs Round Trips).
- A second version running beside the first during a deprecation window costs storage and compute for the overlap period. That is the price of not forcing every consumer onto your schedule, and it is usually worth it (Compute Waste).
- Unretired products cost retained bytes and maintenance attention permanently. A lifecycle policy with an owner is cheaper than the storage it deletes, and the reason it rarely exists is that deletion needs consumer certainty (Storage Lifecycle).
- The conformance machinery itself — catalog integration, publish-time checks, SLO measurement — is a fixed platform cost that does not scale with the number of products, which is what makes it worth building once centrally (Data Platform Engineering).
- The commitments slow publishing down. That is the mechanism, not a side effect: a dataset that takes an afternoon of documentation and threshold-setting to publish is a dataset somebody decided was worth publishing.
- A strict conformance gate creates an incentive to keep datasets private and share them informally instead. The counter is a two-tier namespace where private is a legitimate, easy option — not a stricter gate (Raw, Staging, Curated: Layers by Purpose).
- Per-product SLOs give consumers honest numbers and give them a heterogeneous platform, where joining two products means reasoning about two freshness promises. A single platform-wide SLO is simpler and is a fiction in any organisation with more than one source (The Freshness SLO).
Dataset review questions
This lesson uses the shared review exercise.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- ORG-SPECIFICSix of the seven commitments are cheap work; the owner commitment is a rota, a pager and a budget line, and whether an organisation will fund that is not a technical question. Two companies with identical platforms get completely different results depending on whether ownership arrived with on-call or only with a name in a catalog field.
- GENERALThe seven commitments are independent of storage, engine and ownership model: a central team publishing into one warehouse can make all of them, and a federated organisation must. What differs is only how many teams have to be taught the vocabulary, not what a product has to declare.
- SCALE-SPECIFICBelow roughly one team and a handful of datasets the commitments are ceremony, because every consumer can ask the author directly. They start paying above the point where a dataset is read by someone who cannot identify who built it, which is a consumer-count threshold rather than a data-volume one.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns the release mechanics this lesson borrows wholesale — versioning, gates in the promotion path, announcements, deprecation windows and rollback. A data product is software whose artefact is a dataset, and it deserves the same delivery discipline.
- — Distributed Systems owns what a consumer of a published dataset can assume when the producer is a separate system with its own failures, and why "the contract held" and "the data arrived" are different statements.