Cost Attribution
You cannot manage what you cannot attribute — and in a shared platform every cost belongs to everyone, which means it belongs to nobody.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
A shared warehouse serves eleven teams. Which of them caused this month's increase, and what would it take to answer that without starting an argument?
The consumer of attribution is not a dashboard, it is a conversation. A team lead asked to reduce their platform spend needs to know which of their datasets is responsible and what changed; a platform owner needs to be able to say why the total moved without accusing anyone; finance needs a breakdown that survives being questioned. If the output of your attribution cannot survive someone saying "that is not ours", it has not attributed anything (Data Platform Engineering).
The unit of attribution is one dataset, not one person and not one query. A query is too fine — it is a momentary act with no owner once the analyst has moved on. A person is the wrong axis entirely — people change teams, service accounts run everything, and the analyst who wrote an expensive query usually did so because a model made it expensive to answer their question. A dataset is durable, has an owner, has a dependency graph, and is the thing that can actually be changed (Data Products).
Tag the compute resources by team and report spend per tag. This is the standard first move, it is genuinely useful, and it works well while each team has its own compute. It stops working the moment there is one shared warehouse — which is the architecture most platforms converge on precisely because sharing is cheaper — and at that point every tag reads the same and the report becomes a single unlabelled block.
Everything runs as the orchestrator's service account, so the largest line on the report is one entry named after a robot, covering eighty per cent of spend and pointing at nobody (Orchestration).
- Everything runs as the orchestrator's service account, so the largest line on the report is one entry named after a robot, covering eighty per cent of spend and pointing at nobody (Orchestration).
- A shared warehouse has one tag. Splitting it by team requires attributing individual statements, and the statements do not carry a team — they carry a user, and the user is that service account.
- A team is charged for a query they wrote, but the query is expensive because an upstream model is unpartitioned and wide. The cost lands on the reader and the fix belongs to the producer, so the report generates resentment and no change (Physical Data Layout).
- Attribution is done by person, and a well-meaning analyst appears at the top of the list. The organisational effect is that people stop running queries, including the ones the company needed (Who Actually Consumes This Data).
- A shared upstream model is read by six teams. Attributing its production cost to whoever built it punishes the team that did the most useful work in the platform (Model Layering).
- Nobody owns the platform total, so every team optimises to its own line and the shared drivers — small files, an oversized cluster, a wasteful shared model — belong to none of them and are never touched (Data Ownership).
What is actually happening
- Attribution is a join between a cost record and a dependency graph. The cost record says a statement consumed some chargeable units; the graph says which dataset it read and which dataset it produced. Without the graph you have a per-statement bill, which nobody can act on (Data Lineage).
- Cost flows downstream along the graph. A serving table's true cost is its own production plus a share of everything upstream that exists to feed it, and a table whose only consumer is one dashboard carries the whole chain behind it (Column-Level Lineage).
- Attributing to datasets rather than to people works because a dataset is the object that can be changed. You can repartition it, narrow it, make it incremental or retire it; you cannot do any of those to a person, and telling someone their queries are expensive gives them no lever except to stop asking.
- A dataset also has a durable owner, which a query does not. Ownership is what turns a number on a report into a decision, and it is the reason dataset-level attribution keeps working after a re-org while person-level attribution resets (Data Ownership).
- The political problem is structural, not cultural. On shared infrastructure, one team's saving is diluted across everyone and one team's waste is subsidised by everyone, so the individually rational move is to do nothing. This is the commons problem, and no amount of encouragement fixes it — only making each team's share visible and attributable does (Data Mesh).
- Perfect attribution is not achievable and not the goal. Shared storage, shared caches and shared query capacity genuinely cannot be split without arbitrary rules. The goal is an allocation that is defensible enough to act on, with the arbitrary parts written down rather than hidden (Cost per Service and the Attribution Problem).
Cost flows down the dependency graph
The instinct when a bill grows is to look for the expensive query. That instinct finds the symptom reliably and the cause almost never, because in a layered platform the expensive query is usually the last hop in a chain, and the reason it is expensive was decided several hops upstream.
The chain below is an ordinary one. A dashboard reads a serving table; the serving table is built from a fact model; the fact model reads staging; staging reads raw. Every hop holds cost and every hop can corrupt the attribution of the hops after it — by being wide, by being unpartitioned, by rebuilding history, or by existing at all when nothing reads it.
Reading the couldCorrupt column is the exercise. In each case the cost lands somewhere other than where the decision was made, which is precisely why attribution has to follow the graph rather than the invoice. A report that shows only the last hop tells eleven teams that their dashboards are expensive, and none of them can do anything about it.
- Raw landing zone
holds Every byte that arrived, retained for as long as the policy says, in whatever file sizes the writer produced.
could corrupt A high-frequency writer producing many small objects makes every downstream read pay metadata cost, and that cost is charged to the readers rather than to the ingestion job that caused it (File Size and the Small-Files Problem).
↑ reads from - Staging model
holds Cleaned, typed rows at source grain, usually rebuilt in full because it is cheap to write and nobody revisits it.
could corrupt If it drops the partition column or lands unpartitioned, every downstream model loses the ability to prune, and each of them appears as an expensive job for a reason that lives here (Partitioning).
↑ reads from - Fact model
holds The business grain — one row per order, per event, per interaction — with the joins to dimensions already performed.
could corrupt A full nightly rebuild here is charged to the team that owns the model, while the reason it must be rebuilt is often that an upstream source offers no reliable watermark (The High-Water Mark).
↑ reads from - Serving table or mart
holds A narrow aggregate at the grain a consumer displays, refreshed on the schedule the consumer asked for.
could corrupt If it does not exist, every dashboard reads the fact model directly and the fact model's scan cost is multiplied by the number of consumers — with each consumer appearing as the culprit (Data Marts).
↑ reads from - Dashboard or extract
holds A generated query issued on a refresh interval, by a tool, under a shared connection.
could corrupt It is the only hop most cost reports can see, so it absorbs the blame for every decision above it. It also multiplies: one auto-refresh setting changes the cost of the whole chain behind it (Cost vs Freshness).
The useful attribution question is not "which query cost the most" but "which decision, at which node, is responsible for the cost that shows up at the leaves". That question is only answerable with the graph in hand.
Attributing chargeable units to a dataset
Every analytical platform keeps a history of statements executed and what each consumed. The names differ and the columns differ, but the shape is consistent enough to write against: a statement, an identity, a timestamp, a measure of chargeable work, and — sometimes — the tables it touched.
The query below is the core of an attribution model. It joins statement-level history to a lineage table that maps a statement to the datasets it read and the dataset it wrote, and produces two numbers per dataset: what it cost to build and what it cost everyone else to read. Those two columns answer different questions and mixing them into a single "cost" is the most common way an attribution report becomes unusable.
Note the deliberate omission: there is no user in the output. Resolving the caller matters for the drill-down and it is the wrong axis for the headline, because a person is not something you can repartition. The team column comes from an ownership mapping on the dataset, maintained separately, so a re-org edits one small table rather than rewriting cost history.
1-- query_history: one row per executed statement2-- statement_id, started_at, principal, chargeable_units3-- statement_datasets: which datasets a statement touched4-- statement_id, dataset, access ('read' | 'write')5-- dataset_owner: the separately maintained ownership mapping6-- dataset, owning_team7 8WITH produced AS (9 SELECT sd.dataset,10 sum(qh.chargeable_units) AS units_to_produce11 FROM query_history qh12 JOIN statement_datasets sd USING (statement_id)13 WHERE sd.access = 'write'14 AND qh.started_at >= DATE '2025-03-01'15 GROUP BY sd.dataset16),17consumed AS (18 -- A statement that reads three datasets to write one gets its units19 -- split evenly across what it read. The rule is arbitrary; what matters20 -- is that it is written down here rather than assumed.21 SELECT sd.dataset,22 sum(qh.chargeable_units / rc.n_read) AS units_to_read23 FROM query_history qh24 JOIN statement_datasets sd USING (statement_id)25 JOIN (SELECT statement_id, count(*) AS n_read26 FROM statement_datasets27 WHERE access = 'read'28 GROUP BY statement_id) rc USING (statement_id)29 WHERE sd.access = 'read'30 AND qh.started_at >= DATE '2025-03-01'31 GROUP BY sd.dataset32)33SELECT o.owning_team,34 d.dataset,35 coalesce(p.units_to_produce, 0) AS units_to_produce,36 coalesce(c.units_to_read, 0) AS units_to_read37FROM dataset_owner o38JOIN dataset d USING (dataset)39LEFT JOIN produced p USING (dataset)40LEFT JOIN consumed c USING (dataset)41ORDER BY coalesce(p.units_to_produce, 0) + coalesce(c.units_to_read, 0) DESC;The even split in consumed is the arbitrary part, and it is arbitrary on purpose — a statement reading three tables cannot be apportioned correctly without knowing how much of each it read. Write the rule down, keep it stable, and restate history if you change it. The alternative, an unstated rule, is what makes cost reports unarguable-with and therefore ignored.
Every major warehouse exposes something like query_history and something like a per-statement measure of consumption, and every one names them differently, retains them for a different period, and differs in whether the touched-table list is available without parsing the SQL yourself. Two things to confirm in current documentation: the retention of the history, which must exceed your longest-period consumer's interval to support retirement decisions, and whether the recorded consumption is what was read or what was billed, since caching and result reuse make those diverge.
Who do you send it to?
Once the numbers exist, the harder question begins: what unit do you report them against. This is the decision that determines whether attribution changes anything, because each axis produces a different behaviour and only one of them produces the behaviour you want.
The failure to avoid is person-level reporting. It is the easiest axis to build — every statement has a principal — and it produces an immediate, visible effect: people run fewer queries. That looks like success on the cost report and is a straightforward loss for the organisation, since the queries were how anyone learned anything (Trusting Data).
The commons problem sits underneath all of these options and none of them dissolves it. On shared infrastructure, one team's saving is diluted across everyone while one team's waste is subsidised by everyone, so doing nothing is individually rational. Attribution does not fix that; it makes each team's share visible, which is the precondition for anyone choosing to act (Data Mesh).
- Report produce-cost and read-cost as separate columns. Merging them hides which half of the platform can fix the problem (Model Layering).
- Keep an explicit unallocated line owned by the platform team. Smearing shared overhead across teams to make the total balance destroys the credibility of every other line.
- Attribution becomes chargeback only when the method is trusted. Attaching budget consequences to a contested number ends the programme rather than the spending (Who Owns Data Quality).
You have per-statement cost records and a lineage graph. What do you attribute cost to?
when Almost always, as the primary axis. A dataset has an owner, a dependency graph and a set of properties — partitioning, width, refresh frequency, retention — every one of which is a lever someone can actually pull.
cost Requires lineage that is complete enough to map statements to datasets, and an ownership mapping that is maintained. Shared upstream models need an explicit, arguable allocation rule.
when As the roll-up above datasets, for budget conversations. Derived from dataset ownership rather than measured directly, so a re-org edits a mapping table instead of invalidating history.
cost Team boundaries change more often than datasets, so the series needs restating periodically. Reading cost lands on the consuming team even when the fix belongs to the producer, which has to be handled by reporting produce and read separately.
when When you want to answer "what does this dashboard actually cost" including its whole upstream chain. The most honest number and the most useful one for deciding whether something is worth maintaining.
cost Requires walking the graph and apportioning shared upstream cost across consumers, which multiplies the arbitrary allocation decisions. Best used for a handful of important products rather than as a general report.
when Only as a drill-down during an investigation, never as a published report. Useful for "who issued this one enormous statement", useless as an ongoing metric.
cost Published person-level cost suppresses analysis, which is a real organisational cost your report does not measure. It also attributes to the one axis with no lever: a person cannot repartition their way out of an expensive upstream model.
when Genuinely correct below a few teams, where everyone knows every pipeline and a formal system would cost more than the spend it manages.
cost Stops working exactly when it matters. Once no single person can name every pipeline, an unattributed total is a number that generates meetings and no changes.
How to build it
Most important first.
- Attribute to datasets first, teams second, people never. The dataset is what can be changed; the team is who changes it; the person is noise (Data Products).
- Join query history to lineage so that both the production cost and the read cost of every dataset are visible. Producer-side cost without reader-side cost blames the wrong half of the platform (Data Lineage).
- Report two numbers per dataset: what it cost to build, and what it cost everyone to read. A dataset that is cheap to build and heavily read is where a narrower serving model repays itself many times over (Data Marts).
- Push shared upstream cost onto its consumers proportionally to reads, and say explicitly that you are doing so. Silent allocation rules are the thing people argue with; stated ones they argue about once (Raw, Staging, Curated: Layers by Purpose).
- Give every dataset an owner at creation. Attribution without ownership produces a report nobody is accountable for, which is a report (Data Ownership, The Data Catalog).
- Keep an unallocated bucket and show it. Shared capacity, idle time and platform overhead that genuinely cannot be split should appear as their own line owned by the platform team, rather than being smeared across everyone to make the numbers add up.
- Publish the method alongside the numbers. Attribution's only real output is a conversation, and a conversation about a number whose derivation is secret goes nowhere (Dataset Documentation).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Query history guarantees a record of what was executed and what it consumed, for as long as the platform retains it. That retention is a setting, and it is usually shorter than the interval of your least frequent consumer (The Data Catalog).
- Nothing guarantees that a statement identifies its human origin. Orchestrated work runs as a service account and BI tools run as a shared connection, so the mapping from statement to team is something you construct, not something you receive.
- Lineage guarantees the edges it was told about. Anything expressed dynamically — SQL assembled at runtime, a query issued by a tool outside the platform — is missing from the graph, and its cost lands in the unallocated bucket (Data Lineage).
- No allocation of shared cost is guaranteed to be fair. It is guaranteed to be *consistent* if the rule is written down, and consistency is what makes it actionable (Fixed vs Variable Cost).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check is a coverage assertion: the sum of attributed cost plus the declared unallocated bucket must equal the platform total, and the unallocated share must be trending down or explained. It catches lineage gaps, statements that could not be mapped to a dataset, and whole workloads running outside the accounting.
- It misses misattribution entirely. A report can be perfectly complete and assign a shared model's cost to the wrong consumer, and the sum will still reconcile — completeness and correctness are different properties here as everywhere else in this domain.
- Pair it with a spot audit: take the three largest attributed lines and trace them back to individual statements by hand. If a line cannot be explained in one sentence to the team that owns it, the attribution is not yet usable (Where Did This Number Come From?).
- Attribution data is inherently lagging. Query history lands after the fact and cost records are consolidated on the provider's schedule, so an attribution report is a description of last week rather than a control on this week.
- That lag decides how it can be used. It is good for finding structural problems — a model that has been wasteful for months — and poor for catching a runaway job, which needs a limit at execution rather than a report afterwards (Pipeline Metrics).
- Retention of the history matters more than its freshness. A report that only reaches back thirty days cannot see a quarterly consumer, and it is the quarterly consumer that makes retirement decisions dangerous (Storage Lifecycle).
- Attribution breaks when lineage breaks. A refactor that replaces a model with a differently named one resets its cost history, and the trend that would have shown the regression starts over (Data Lineage).
- Renaming a dataset renames its cost line. Keep a stable identity across renames or every reorganisation of the warehouse erases the evidence base for cost decisions.
- Team boundaries change more often than datasets do, which is another argument for attributing to the dataset and mapping datasets to teams as a separate, editable layer (Data Ownership).
- Attribution is a reporting layer, so mistakes in it corrupt decisions rather than data. The recovery is to recompute from the retained history — which only works if the history is retained, making retention the one irreversible part of this lesson.
- When an allocation rule turns out to be wrong, restate the history under the new rule rather than changing it going forward only. A cost series with a methodology change in the middle and no restatement is a series nobody will trust again.
- If a cost report has already caused a bad decision — a dataset retired that was needed — the recovery is the ordinary one: rebuild from raw if the inputs are retained, and treat the incident as evidence that the read-recency window was too short (Reprocessing vs Retrying).
What can go wrong
- Everything attributed to a service account, producing one enormous line that names a robot.
- Cost attributed to readers of a badly laid-out dataset rather than to its producer, so the report points at people who cannot fix it (Physical Data Layout).
- Person-level attribution suppressing legitimate analysis, which is a real cost the report does not capture.
- A shared upstream model's cost charged entirely to whoever built it, which penalises the most useful work in the platform.
- The mitigation failing: an attribution report built on lineage that misses dynamic SQL, so the fastest-growing workload is precisely the one in the unallocated bucket (Data Lineage).
- Attribution used as a chargeback with real budget consequences before the method is trusted, which converts an engineering conversation into a political one and ends the programme (Who Owns Data Quality).
- "Tagging solves attribution." Tags attribute *resources*. On a shared warehouse there is one resource, and the interesting question is inside it (Cost per Service and the Attribution Problem).
- "The team that runs the expensive query should pay for it." Sometimes. Frequently the query is expensive because of a layout decision made by the producing team, and charging the reader guarantees the fix never happens.
- "We need perfect attribution before we act." You need attribution that is defensible enough that the owner of the largest line agrees it is theirs. That threshold is far below perfect and is reached much sooner.
- "Attribution will make teams cost-conscious." It will make them conscious of the number you show them. If that number is per-person, they will run fewer queries; if it is per-dataset, they will fix datasets. The axis you choose is the behaviour you get.
- "The platform team owns platform cost." They own the shared overhead. They cannot own the consequences of eleven other teams' modelling decisions, and holding them responsible produces heroic tuning and no behavioural change (Who Owns Data Quality).
- Query history is a record of who asked what, which is access-log data about people. It is subject to access control in its own right, and an attribution dataset that exposes every analyst's queries to every other team is a governance problem wearing a cost report's clothes (Data Access Control).
- Attributing cost by dataset rather than by person has a privacy benefit as well as a practical one: the dataset-level aggregate answers the cost question without publishing individual behaviour (Data Minimization).
- Retaining query history long enough to be useful for retirement decisions means retaining personal activity data longer than you otherwise would, and that retention needs the same justification and expiry as any other personal data (Data Retention).
Operating it
- Cost per dataset, split into produced-by and read-by, trended weekly. The core artefact; almost everything else is a drill-down from it (Cost per Service and the Attribution Problem).
- The unallocated share of platform spend, trended. A rising unallocated share means the accounting is losing touch with the platform, which happens silently.
- Top readers per dataset, by chargeable units, with the caller resolved to a team rather than to a service account.
- Cost per dataset divided by reads of that dataset — the closest available proxy for value, and the one that identifies expensive datasets nobody uses (Impact Analysis).
- Change in cost per dataset week over week, ranked by absolute change rather than by size. The largest dataset is rarely the one that moved (Regression or Tuesday? Telling a Real Change from Noise).
- At 10x teams the shared-platform commons problem dominates everything technical, and attribution stops being a reporting nicety and becomes the mechanism by which any cost work happens at all (Data Mesh).
- At 10x datasets, attribution has to be automatic and derived from lineage, because a manually maintained mapping of dataset to owner drifts from reality within weeks (The Data Catalog).
- At 10x query volume, the history itself becomes a large dataset that needs partitioning and retention decisions of its own — the accounting system acquires the problems of the thing it accounts for.
- Below a few teams, none of this is worth building. Two engineers who know every pipeline can attribute cost by reading a list of jobs, and a formal system would be pure overhead (Data Platform Engineering).
- Attribution itself consumes the platform: query history is a dataset, lineage extraction is a job, and the reporting model is refreshed on a schedule like any other. It is overhead deliberately accepted in exchange for being able to see everything else.
- A quality suite or lineage extractor that reads metadata across every table in the platform is a full metadata sweep on a schedule, and on a large catalog that is not free.
- Retention of query history is retained storage, and it must be longer than the longest consumer interval to be useful — so this is one place where a longer retention window is the correct expensive choice.
- The largest cost of not attributing is the one that never appears: every cost decision made without evidence, which is most of them.
- Attribution costs engineering time and produces no saving by itself. It is a measurement system competing for resources with the fixes it would inform, and the argument for it is that without it the fixes are chosen by whoever is most persuasive.
- Any allocation of shared cost is arbitrary at the edges. Making the rule explicit invites argument about the rule; leaving it implicit invites argument about the numbers, which is worse and never ends.
- Chargeback — attribution with budget consequences — creates real accountability and real incentives to game it: teams will copy data to avoid reading a charged dataset, or move work outside the platform entirely (Data Platform Anti-Patterns).
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- ORG-SPECIFICThe commons problem this lesson is about only exists where multiple teams share infrastructure and none owns the total. A single-team platform has perfect attribution for free, and everything here is overhead until the second team arrives.
- WAREHOUSE-SPECIFICWhat you can attribute depends on what the platform records per statement — bytes scanned, slots, credits, warehouse-seconds — and on how long it keeps it. Some platforms make per-statement attribution straightforward and others make the shared-cluster case genuinely unsolvable without sampling.
- SCALE-SPECIFICBelow roughly three teams and a hundred datasets, cost can be attributed by reading a list of jobs, and a formal attribution system costs more than the spend it manages. It becomes necessary at the point where nobody can name every pipeline in the platform.
- CLOUD-SPECIFICTag propagation, billing-export granularity and how quickly cost records become available differ by provider, and some resources cannot carry a tag at all. The unallocated bucket exists partly because of those gaps, and its size is a provider property as much as an engineering one.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns the account, project and tag structure that decides how much of the platform can be attributed by infrastructure alone before any query-level work is needed.