Who Actually Consumes This Data
Analysts, finance, product analytics, ML, agents and operational read-back each need a different freshness, tolerate a different amount of revision, and fail in a different way. Design from them inward.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
Before designing a dataset, what should I know about the people who will read it — and which of their needs will silently invalidate my design if I guess?
This lesson's own consumer is the engineer about to create a table. The argument it makes is that the first question is never "what does the source have" but "who reads this, how fresh do they need it, and what happens to them when it is wrong" — and that those three answers determine most of the architecture that follows.
The unit is one consumer-question pair: not "the finance team" but "the finance team closing a month". The same people have different requirements at different moments — a controller exploring a variance tolerates staleness that the same controller publishing a statement does not — and designing for the team rather than the question produces a dataset that serves neither well.
Model whatever the source system has, publish it, and let consumers work out what to do with it. It is fast, it is honest about what exists, and it avoids guessing at requirements nobody has written down.
The dataset reflects the source's shape, so every consumer writes the same twelve lines of reconstruction logic — differently. Within a quarter there are four definitions of an active customer and no way to say which is right (The Metrics Layer).
- The dataset reflects the source's shape, so every consumer writes the same twelve lines of reconstruction logic — differently. Within a quarter there are four definitions of an active customer and no way to say which is right (The Metrics Layer).
- Freshness is set by whatever the pipeline could easily do, and one consumer needed it before the morning stand-up while another only needed it monthly. The first builds their own extract; the second is paying for hourly compute nobody uses (Cost vs Freshness).
- A dataset is republished after a correction and finance discovers that a figure they filed externally has changed. Nothing in the design ever said whether a closed period was immutable (What Backfills Break).
- A model trains on a feature computed with information that was not available at prediction time. Evaluation looks excellent, production does not, and the gap is discovered weeks later with no obvious cause (Feature Pipelines).
- An agent retrieves a document that was superseded a month ago and answers a customer confidently from it. There is no error, no alert and no downstream signal at all (Data Engineering for Agents).
What is actually happening
- Consumers differ along three axes that are usually collapsed into one. Freshness — how far behind the present can the data be before the decision changes. Revision tolerance — what happens if a number they have already used is later corrected. Failure reaction — what it costs them, and whether they notice at all.
- Revision tolerance is the axis engineers most often ignore, and it is the one with the sharpest consequences. An analyst re-runs a query and shrugs. A finance controller who filed a figure externally cannot re-run anything; a change to a closed period is a restatement, and restatements have process attached (Snapshot Tables).
- Failure reaction sorts consumers into two groups that need completely different investment. Some fail loudly and immediately — an operational read-back that sends the wrong email is a customer-visible event within minutes. Others fail silently and late — a training set with subtle leakage, a retrieval corpus with stale documents, an experiment read against a changed definition (The Pipeline Succeeded. The Data Is Wrong.).
- Designing from the source outward optimises for what is easy to produce. Designing from the consumer inward optimises for what is needed, and the difference shows up as grain: source-out modelling produces tables shaped like the upstream system, and consumer-in modelling produces tables shaped like the questions (Analytical Data Modeling).
- The requirement that most often surprises is point-in-time correctness: several consumers need to know what was true at a past moment, not what is true now. Finance needs the value as reported; ML needs the value as known at prediction time; an audit needs the value as recorded. A model that stores only current state cannot serve any of them (Slowly Changing Dimensions).
Six consumers, six different sets of requirements
The columns that matter are the last three. Freshness is the one everybody discusses; revision tolerance and failure reaction are the ones that actually decide the design, and they are almost never asked about.
Read down the "reaction to being wrong" column first. Three of these six consumers cannot detect that they were served bad data — a model, a retrieval corpus and an experiment all absorb wrong inputs and produce plausible outputs. Those three need signals built for them, because they will never generate one.
Then read the revision column. It splits the six cleanly into consumers who can re-run and consumers who have already published, and that split determines whether a closed period can ever be restated. It is a modelling and process decision, and if nobody makes it, finance experiences the default as "no, and we found out the hard way" (Snapshot Tables).
| Consumer | What they are deciding | Freshness they actually need | Tolerance for revision | What happens when it is wrong |
|---|---|---|---|---|
| Analysts / BI | Exploring a question, building a chart, sizing an opportunity. | Daily is almost always enough. Requests for hourly are usually requests for reliability. | High. They re-run the query and move on, as long as they knew it changed. | They lose trust in the dataset and build a private extract. The damage is a fork in the definitions, not a wrong decision (Two Dashboards, Two Numbers). |
| Finance / accounting | Closing a period, filing a statement, reporting externally. | Days, but with a hard boundary: the period must be complete and then frozen. | Near zero after close. A change to a filed figure is a restatement with process attached. | Escalation, audit questions, and a permanent loss of confidence in the platform. This is the consumer whose failures are most expensive per occurrence (Reconciliation). |
| Product analytics | Ending an experiment, shipping or not shipping a feature. | Enough to close the experiment on schedule — usually daily, occasionally hourly. | Moderate, but definition stability matters more than freshness: a metric that changes mid-experiment invalidates it. | A wrong ship/no-ship decision that nobody attributes to data, because the experiment reported cleanly (Semantic Changes). |
| ML training + features | Training a model, and serving features to it at prediction time. | Training tolerates staleness; serving usually does not. The two paths must agree. | Low, and for an unusual reason: retraining is expensive and everything scored in between is suspect. | Evaluation looks good, production is worse, and the cause is found weeks later if at all — classically point-in-time leakage (Feature Pipelines). |
| Agents / retrieval | Answering a user from a corpus, grounded in retrieved documents. | Corpus freshness is a product property: a superseded document answers today's question wrongly. | Low. There is no re-run — the answer was already given to a person. | A confident, fluent, wrong answer with no error anywhere. The quietest failure on this table (Data Engineering for Agents). |
| Operational read-back | Taking an action in a product: an email, a suspension, a price, a segment. | Minutes, and inside a user-facing loop rather than a reporting one. | None. The action already happened and cannot be un-sent. | A customer-visible event within minutes — the loudest failure here, and the reason this consumer justifies stricter guarantees than its volume suggests (Deduplication). |
One schema change, six different reactions
The clearest way to see why consumer analysis is not optional is to take a single, modest, well-intentioned upstream change and follow it into each consumer. The change below is one people make weekly: a currency amount is split into an amount and a currency code, and the old column is kept for compatibility but its meaning quietly narrows to the reporting currency.
Every schema check passes. The old column still exists, still has the same type, still has no nulls. Contract enforcement based on shape sees nothing at all, because nothing about the shape changed (Semantic Changes).
What follows is six different outcomes, three of them silent. The silent flag is the important column: a break that raises is a break somebody fixes today, and a break that does not raise is one somebody discovers in a quarterly review, or never.
- Four of six break silently. No test that checks shape, nullability or row count would have fired on any of them.
- The two loud breaks are the two consumers closest to a human decision — finance and operational read-back — which is why those two so often end up being the platform's de facto quality monitors.
- The fix is not a better schema check. It is a contract that includes meaning, and a change process that enumerates consumers before the change rather than after the incident (Data Contracts).
- order_id: string
- ordered_at: timestamp
- amount_minor: integer -- in the order's own currency
- customer_id: string
- order_id: string
- ordered_at: timestamp
- amount_minor: integer -- NOW: converted to the reporting currency
- amount_original_minor: integer
- currency_code: string
- customer_id: string
change Two columns added; no column removed, renamed or retyped. amount_minor keeps its name and type and changes what it means: it is now converted at the rate on the day of the load rather than being the amount the customer was charged.
| Consumer | Effect | How it shows up |
|---|---|---|
| Analysts / BI | Totals shift for non-domestic orders. Someone eventually notices the trend has a step in it and asks; the answer takes a week to find because nothing in the schema changed. | Silently — no error, wrong result |
| Finance | A previously-closed period now sums differently, because the conversion applies to history on every full refresh. This is the failure that becomes an incident with a formal review. | Loudly — it raises |
| Product analytics | A running experiment measuring revenue per user has its metric definition changed mid-flight. The result is reported cleanly and is not comparable across the change. | Silently — no error, wrong result |
| ML features | A feature computed from amount_minor changes distribution. Training on post-change data and scoring on pre-change history — or the reverse — degrades the model with no error anywhere. | Silently — no error, wrong result |
| Agents / retrieval | Any summary or answer grounded in a table that mixes both meanings quotes figures that are individually defensible and jointly inconsistent. | Silently — no error, wrong result |
| Operational read-back | A threshold rule ("spend over X triggers an outreach") starts firing on a different population. This one is loud within days, because a human notices the wrong customers being contacted. | Loudly — it raises |
Designing from the consumer inward
Consumer-first design is a sequencing rule, not a philosophy. It says: establish the questions, the freshness and the revision policy before choosing the grain, and choose the grain before choosing anything about storage, layout or tooling. Reversing that order is the single most common way a data platform ends up full of datasets that are technically excellent and nobody uses (Analytical Data Modeling).
The sequencing matters most for the decisions that cannot be revisited. Grain, history and point-in-time answerability all have to be right early: a model that stored only current state cannot be made to answer "what was true in March" retroactively, because the information was never kept (Slowly Changing Dimensions).
It also disciplines the freshness conversation, which is otherwise unbounded. The bars below show what actually moves when a consumer asks for fresher data, relative to each other. Note what dominates: the requirement does not mostly buy compute, it buys operational surface — more runs, more failures, more state, more things to be on call for (Cost vs Freshness).
The practical form of this is a short interview. What decision does this drive; how stale can it be before the decision changes; what happens if we correct it later. Three questions, and the answers usually settle the grain, the schedule and the immutability policy in one conversation.
The dominant cost and the least anticipated. Going from daily to hourly multiplies every failure mode by the number of runs, and going to continuous adds state that has to be checkpointed and recovered (Checkpointing).
A frequent schedule with a full-refresh model recomputes all of history every run. This is where freshness requirements silently become the largest line in the platform (Full Refresh vs Incremental).
Fresher means publishing before all the data has arrived, which converts a batch problem into an event-time problem with watermarks and allowed lateness (Late-Arriving Data).
More frequent writes produce more, smaller files, which makes every subsequent read more expensive until compaction is a scheduled job of its own (File Compaction).
Usually the smallest of these and almost always the first one estimated, because it is the only one that appears as a line on a bill.
Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.
Relative weights for a batch platform being asked to run more often, shown to establish an ordering rather than a magnitude. The teaching is the ordering: freshness is mostly bought with operational burden and reprocessing, not with compute, which is why "just run it hourly" is a much larger decision than it sounds.
Model what the source has. Publish `stg_orders`, `stg_customers`, `stg_payments` shaped like their upstream tables, and let each consumer join, filter and define what they need.
Start from the questions that must be answerable, at what freshness, with how much history. Derive the grain from the questions, the schedule from the freshness, the history model from the point-in-time needs — then build the staging models that get you there.
Source-shaped datasets push definition work onto every consumer independently, so the same concept gets defined several ways and the divergence is only discovered when two of them are shown side by side. Consumer-shaped datasets centralise that work once, where it can be reviewed, tested and owned — and they surface the irreversible decisions, grain and history, early enough to actually make them.
How to build it
Most important first.
- Interview before modelling. Three questions per consumer: what decision does this drive, how stale can it be before the decision changes, and what happens if it is later corrected. Most designs are settled by the answers (Data Products).
- Publish grain, freshness and revision policy as part of the dataset, not as tribal knowledge. "This table is at one row per order-day, refreshed hourly, and closed periods are never restated without notice" is a contract and it prevents most of the arguments (Data Contracts).
- Serve the strictest consumer with a separate dataset rather than raising everyone's guarantees. Making the whole platform finance-grade because one report is finance-grade is how platforms become unaffordable (Data Marts).
- Preserve point-in-time answerability wherever any consumer needs it, and decide that early — it is a modelling decision that cannot be retrofitted once history has been overwritten (SCD Type 2 in Practice).
- Give silent-failure consumers explicit signals, because they will not generate one themselves. A retrieval corpus needs a staleness metric; a feature pipeline needs training-serving consistency checks (Embedding Pipelines).
- Say no in the contract. A dataset that declines to promise sub-hourly freshness is more useful than one that promises it and misses, because consumers plan against the promise (The Freshness SLO).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- What you promise a consumer is the union of every hop's weakest promise, plus whatever validation you added. Promising more than that is not optimism, it is a defect you have not observed yet (The Data Loop).
- A stated freshness is a commitment to a distribution, not to a maximum. "Hourly" that is met 95% of the time is a different product from "hourly" that is met always, and the consumers who care about the difference are exactly the ones who will not tell you until it fails (Pipeline SLOs).
- Immutability of a closed period is a guarantee you either make or do not. It is not a default in either direction, and the absence of a decision is experienced by finance as the absence of the guarantee (Snapshot Tables).
- Nothing guarantees a consumer uses the dataset the way you intended. A mart built for one aggregate will be joined back to an order-level table by someone, and the guarantee you offered says nothing about that composition (Grain: What Does One Row Represent?).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that matters is per consumer, not per table: assert the property that consumer depends on. Uniqueness for the read-back that sends emails, point-in-time correctness for the feature pipeline, period totals for finance (Data Tests).
- Add a usage check alongside the quality checks: which consumers actually queried this dataset in the last month. A dataset with tests and no readers is a maintained liability (Data Discovery).
- These miss the consumer who copied the data out months ago and has been reading a snapshot ever since. Nothing upstream can see them, and they will report an incident about a number you fixed in March.
- Freshness needs cluster far more coarsely than people request. In practice most consumers need daily, a few need hourly, and a small number genuinely need minutes — and the ones who ask for minutes are frequently reading the dashboard once a day (Cost vs Freshness).
- The question that separates real requirements from stated ones is: what decision would you make differently if this were an hour older? A requirement that survives that question is real and worth paying for.
- Operational read-back is the consumer whose freshness need is genuinely different in kind. It is inside a user-facing loop, so its latency is a product property rather than an analytics one, and it should usually not be served from the same path as reporting (The Event-Driven Data Platform).
- The same schema change lands differently on every consumer, and the ones it breaks silently are rarely the ones consulted before the change (Schema Evolution).
- A definition change is worse than a schema change for exactly the consumers with the lowest revision tolerance. Finance can absorb a new column; they cannot absorb revenue quietly moving from gross to net between two closes (Semantic Changes).
- Consumers are added far faster than they are removed, so the compatibility burden grows monotonically. Deprecation needs a process from the beginning, because retrofitting one onto eighty unknown readers is not possible (Impact Analysis).
- Recovery is consumer-specific and must be planned as such. An analyst needs the table fixed. Finance needs the table fixed *and* a statement of which previously-published figures moved and by how much (Validating a Backfill Before You Publish).
- A model trained on bad data is not fixed by fixing the data; it has to be retrained, and everything it scored in between is suspect. That is a longer recovery than any pipeline re-run (Feature Pipelines).
- An operational read-back that acted on wrong data cannot be recovered at all in the usual sense — the email was sent, the account was suspended. Recovery there is a customer-communication problem, which is why that consumer justifies stricter guarantees than its volume suggests.
What can go wrong
- Designing for the loudest consumer rather than the strictest one. The team that complains most is rarely the team whose failure is most expensive.
- One dataset asked to serve every consumer, so it carries the strictest freshness, the strictest immutability and the widest schema — and is too expensive to run and too rigid to change (Data Platform Anti-Patterns).
- A consumer nobody knew about, reading a table that was changed on the assumption it had two readers (Impact Analysis).
- A silent-failure consumer with no monitoring, discovered by a customer rather than by a signal (Agent Observability Data).
- Requirements gathered once, at project start, and never revisited as the consumers' decisions changed underneath them.
- "Consumers do not know what they want, so ask the source." They know what decision they are making, which is the only requirement that matters. They cannot specify a schema, and they should not be asked to.
- "Everyone wants real-time." Most people want *reliable*, and say real-time because staleness and unreliability feel identical from the outside. A dataset that is always four hours old is far more usable than one that is sometimes minutes old and sometimes two days (Freshness Monitoring).
- "ML is just another SQL consumer." It is the consumer with the strictest point-in-time requirement and the quietest failure mode, and treating it as an analyst with a bigger extract is how training-serving skew gets built in (Feature Pipelines).
- "If nobody complained, the dataset is fine." Half of these consumers cannot complain, because their failure produces no visible error — it produces a slightly worse model or a confidently wrong answer (Data Engineering for Agents).
- Consumers determine the access surface. A dataset serving finance, ML and an operational read-back has three different access patterns and probably three different sensitivity requirements on the same columns (Data Access Control).
- Personal data reaches consumers who never needed it, because the dataset was modelled from the source rather than from the question. Consumer-first design is also the cheapest form of data minimisation (Data Minimization).
- A training set or a retrieval corpus is a copy that outlives its source, and a deletion request has to reach it. Enumerate those consumers explicitly — they are the ones most often missing from the list (Deletion Requests).
Operating it
- Query logs by consumer and by dataset: who reads what, how often, and when they stopped. This is the most under-used signal in most warehouses and it answers both "who breaks if I change this" and "what can I delete" (Data Discovery).
- Freshness achieved versus freshness promised, per serving dataset, as a distribution rather than an average (The Freshness SLO).
- Republish events on closed periods, with the consumers notified. If that list is empty and republishes happen, the revision policy is not being honoured (Rolling Back Data).
- For silent consumers specifically: corpus staleness for retrieval, training-serving consistency for features, and duplicate-action counts for operational read-back (Agent Observability Data).
- At a handful of consumers, requirements live in conversations and that works. The breaking point is when a change requires asking people you cannot name (The Data Catalog).
- At 10x consumers the marginal cost of a change is dominated by coordination, so contracts and deprecation windows stop being ceremony and start being the only way to change anything (Data Contracts).
- At 100x, self-service becomes the only workable model: consumers compose their own answers from governed, documented, contracted models, and the platform team serves models rather than answers (The Self-Service Data Platform).
- Freshness is the most expensive requirement a consumer can state, and the cost is not linear — moving from daily to hourly multiplies runs, and moving from hourly to continuous changes the architecture (Cost vs Freshness).
- Serving many consumers from one wide model costs scan on every query; serving each from a purpose-built mart costs storage, pipeline complexity and one more thing that can be stale. The trade turns on how stable the query patterns are (Data Marts).
- The cheapest thing you can do is delete a dataset nobody reads, and the reason it does not happen is that nobody measures readership (Cost Attribution).
- Consumer-first design produces better datasets and takes longer to start. There is a real cost to interviewing three teams before writing a table, and it is worth paying only for datasets that will be depended on.
- Purpose-built marts make each consumer fast and multiply the number of things that can drift from their parent model. Every mart is a promise to keep it reconciled (Data Marts).
- Strict contracts protect consumers and slow producers. The producing team now needs a compatibility review for a change that used to be a migration, and that friction is the price of not surprising people (Contract Enforcement).
Six consumers, six obligations
Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.
Read once each morning, glanced at, and quoted in a meeting an hour later by someone who did not open it.
The previous day, complete, before the morning it is read. Anything fresher than that is spent effort — this table is looked at once and the decisions behind it move on a weekly cadence.
A number that changes after it has been said out loud costs credibility, even when the new number is more correct. The platform owes a *stated* revision window rather than silence.
- ·A completeness guarantee for a closed period, not a best-effort one for an open one.
- ·A visible “as of” marker, so a stale table cannot pretend to be a current one.
- ·A revision policy in writing: what may still move, and until when.
It is served from a table that is still being written to. The morning read then lands mid-period and reports a day that is genuinely incomplete as if it were done.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALThe three axes — freshness need, revision tolerance, failure reaction — apply to any data consumer in any stack, including a single scheduled email report. What varies is which consumers exist: a company with no ML has one fewer failure mode and one fewer set of point-in-time requirements.
- ORG-SPECIFICThe strictness ordering of consumers is a property of the business rather than of the technology. In a regulated financial firm the finance consumer dominates every design decision; in a consumer product the operational read-back usually does, because its failures are visible to customers within minutes.
- SIMPLIFIEDSix consumer types is a working taxonomy, not a complete one — data-sharing partners, regulators and embedded customer-facing analytics each behave differently again. The six are chosen because they span the range of freshness needs and failure reactions.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — Distributed Systems owns why a consumer reading a copy is always reading a past state, and what the available consistency choices cost. This lesson takes that as given and asks which consumers can live with it — which turns out to be most of them, and precisely not the operational read-back.
- — DevOps / Production Engineering owns the deprecation and release discipline this lesson leans on: announcing a change, giving consumers a window, and knowing who to announce it to.