Pipeline SLOs
A published, measured promise about a dataset — when it arrives, how fresh it is, how often it is right — agreed with the people who depend on it rather than declared by the team that runs it.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
What exactly are we promising the people who read this table, and how would either of us know the promise was broken?
The finance analyst who starts the close at 07:00, the marketing job that reads yesterday's attribution before it spends money, and the on-call engineer at 03:00 deciding whether a failed run can wait until morning. All three are currently guessing, and all three have guessed differently (Who Actually Consumes This Data).
The unit is one dataset, one promise. Not a platform, not a DAG, not a team: the thing an SLO can be written about is a table with a name, an owner and consumers who would notice its absence. A promise attached to anything larger cannot be measured and cannot be breached (Data Products).
Publish a run-success rate for the orchestrator — "99.5% of tasks succeeded this month" — and treat that as the reliability of the platform. It is easy to compute, it comes free with the scheduler, and for the failure mode it covers it is genuinely informative.
Every task succeeded and the upstream extract returned zero rows because a source API changed its pagination. The success rate is perfect and the table is empty (The Pipeline Succeeded. The Data Is Wrong.).
- Every task succeeded and the upstream extract returned zero rows because a source API changed its pagination. The success rate is perfect and the table is empty (The Pipeline Succeeded. The Data Is Wrong.).
- The DAG finished at 09:40 instead of 05:40 because a retry took four hours. Nothing failed, so nothing is counted, and the analyst who starts at 07:00 spent two hours looking at yesterday (Stale Dashboards).
- A task that runs once an hour and a task that runs once a month contribute equally to the numerator, so a single monthly finance model failing is invisible next to a healthy hourly one.
- The number is an average across two hundred datasets, so the three that matter to the business are averaged with a hundred and ninety-seven that nobody reads. A platform can be 99.5% reliable and completely broken for its most important consumer (Data Observability).
- Nobody agreed to it. It was computed, published on a dashboard, and never once used to decide whether to page, to invest, or to say no — which is the definition of a metric rather than an objective.
What is actually happening
- An SLO is three things and it is incomplete without all three: an indicator (a measurable property of the served data), an objective (the threshold and the window over which it is evaluated), and an agreement (a named consumer who accepts it and a named owner who is accountable for it) (SLIs: Measuring What the User Actually Feels).
- The indicator has to be measurable from the served side. "The data is correct" is not an indicator because nothing can compute it; "the newest complete hour is less than N behind now" is, because a query returns it (The Freshness SLO).
- The evaluation window converts the objective from a feeling into arithmetic. A rate over thirty rolling days behaves very differently from a rate over a calendar month — the second is forgiven on the first of every month, which is why a monthly window quietly rewards failing early (SLOs: A Target, a Window, and a Reason).
- The gap between the objective and one hundred per cent is the error budget, and it is the useful half. It converts an argument about whether to prioritise reliability work into a measurement: budget remaining, and burn rate against it (Error Budgets: Unreliability You Are Allowed to Spend).
- Data platforms need at least two indicator families because their failures are of two kinds. Timeliness — did the data arrive, and how fresh is it — is what a scheduling failure damages. Correctness — completeness, uniqueness, distribution — is what a silent failure damages, and no timeliness indicator sees it (The Dimensions of Data Quality).
- A breach is a consumer-visible event, not a task-visible one. That is what makes an SLO different from a monitor: the monitor watches your system, and the objective describes their experience (From Symptom to Root Cause).
Three shapes an objective takes
Almost every data SLO worth writing is one of three shapes. A deadline — the dataset is available by a stated hour. A freshness bound — the newest complete data is no older than a stated interval. A success proportion — some stated share of runs, or of periods, met their definition of done. Each protects a different consumer behaviour and none substitutes for the others.
The figures in the table below — 06:00, fifteen minutes, 99.9% — are examples of the shape, not recommendations. There is no defensible general answer to "how fresh should a table be", because the answer is a property of the decision the data drives and of nothing else. A number copied from another company's blog post is an objective with no consumer behind it, which is the one kind guaranteed to be wrong.
Read the last column first. What an objective does *not* say is the part that surprises people: a dataset can hit a punishing deadline every single morning and be missing a third of its rows, because a deadline is a statement about arrival and says nothing whatever about content.
| Shape | The indicator, measured how | The consumer behaviour it protects | What it does not say | Example — shape only |
|---|---|---|---|---|
| Deadline | Timestamp at which the dataset became complete for the period, read from the published table rather than from the scheduler. | A decision taken at a fixed hour: a close, a morning report, a scheduled spend. | Nothing about content, and nothing about the twenty-three hours between deadlines. | "Yesterday's orders are available by 06:00 on 99% of business days" |
| Freshness bound | Now minus the event time of the newest complete unit, evaluated continuously. | An unscheduled decision taken whenever someone looks — an operational dashboard, an alerting rule. | Whether the data that is there is right; a table of partial current data is fresh and wrong (The Freshness SLO). | "Newest complete data is under 15 minutes old, 99% of the time" |
| Success proportion | Share of scheduled periods that met their full definition of done, including validation. | Trust that the series has no holes — the thing a time-series analysis silently assumes. | Which periods failed, or whether the failures cluster. Ninety-nine per cent scattered and ninety-nine per cent consecutive are very different datasets. | "99.9% of daily runs publish and pass validation" |
| Completeness objective | Reconciled row count or monetary sum against the source for a closed period (Reconciliation). | Any use where a missing slice changes the conclusion — finance, regulatory, experiment analysis. | Whether values are correct. A reconciled sum with the wrong definition reconciles beautifully (Two Dashboards, Two Numbers). | "Closed-period order count matches source within a stated tolerance" |
| Platform-wide average | Mean attainment across all datasets. | Nothing. No consumer makes a decision on it. | Which datasets are broken — which is the only thing anyone wanted to know. | Listed here to be argued against, not adopted. |
Measuring the indicator honestly
An objective is only as good as the query behind it, and the most common way objectives become theatre is an indicator computed from the orchestrator rather than from the data. The scheduler knows when a task exited zero. It does not know whether the table gained rows, whether they were the right rows, or whether the period is complete — and every one of those is what the consumer meant.
The rule is that the indicator must be computable by the consumer. If an analyst could not run a query that reproduces your attainment number, then the number describes your system rather than their experience, and the first disagreement about whether an objective was met will be unresolvable.
Each check below has a blind spot, and the blind spots are the argument for having more than one. There is no single indicator that covers timeliness and correctness, and a platform that picks one has chosen which class of failure it is willing not to see.
| Check | Expresses | Catches | Still misses |
|---|---|---|---|
| Task exited zero within its timeout | The code ran. | Crashes, timeouts, unhandled exceptions, an unavailable dependency. | Every silent failure: zero rows loaded, a filter that dropped a category, a fan-out join, a cast that nulled a column. All exit zero (The Pipeline Succeeded. The Data Is Wrong.). |
| Completion marker present for the period, written by the publish step | The dataset was published as complete for that period. | Partial publishes, a run that wrote some partitions and stopped, a consumer reading mid-write (Atomic Publish). | A complete publish of wrong content. The marker asserts the publish finished, never that what it published is right. |
| Now minus event time of newest complete unit | How stale the freshest trustworthy data is. | A stopped pipeline, a stopped source, a run that produced nothing, an upstream that silently emptied. | Fresh and wrong. It also fires falsely for any period where the source genuinely produced nothing, which is why a quiet weekend pages somebody (The Freshness SLO). |
| Row count for the closed period versus source | Completeness — everything that happened arrived. | Missing rows, an extract window that closed early, a dropped partition (Missing Rows). | Duplicates that offset losses, anything in an unclosed period, and any error that preserves the count while corrupting values (Duplicate Rows). |
| Validation suite passed before publish | The assertions someone wrote hold on this batch. | Whatever was encoded — nulls, ranges, uniqueness, referential integrity (Data Tests). | Everything nobody thought to encode, which is where the incidents you have not had yet all live. |
Notice that the first row is the one most platforms use as their reliability number, and it is the row whose misses column contains the entire subject of this domain.
Writing one down
An objective that lives in a conversation is not an objective. Writing it as data — next to the dataset, in the same repository as the model that produces it — makes it reviewable, diffable, generatable into checks, and publishable to the catalog entry the consumer actually reads.
The fields that matter most are the ones teams leave out. Who agreed: a named consumer, because an objective with no counterparty cannot be breached in any meaningful sense. What is excluded: whether a source outage or a planned backfill consumes budget, decided in advance rather than during the incident. What happens when the budget is gone: the policy, written down while everyone is calm.
The window deserves one deliberate thought. A rolling window means today's miss is forgiven thirty days from now; a calendar window means everything is forgiven on the first. Rolling is almost always the honest choice, and it is the one that makes a burn-rate alert meaningful (Burn-Rate Alerts: How Fast Is the Budget Going?).
A dataset has consumers and no written promise. Where do you start?
when Every consumer reads it at a fixed hour — a close, a morning report, a scheduled downstream job.
cost Says nothing between deadlines and nothing about content. A dataset can meet it daily while being wrong (Data Quality).
when Consumption is continuous and unscheduled — an operational view, an alerting rule, a customer-facing figure.
cost Rewards publishing partial data quickly. Without a completeness definition of "complete", it actively encourages the wrong behaviour (The Freshness SLO).
when The dataset drives money or reporting. This is the common right answer for a core fact table.
cost Two indicators to measure and two conversations to have, plus a reconciliation against the source that has to be built and maintained (Reconciliation).
when The long tail: exploratory datasets, internal staging models, anything with no consumer who would be harmed.
cost Consumers will still assume something. Saying "best effort, no objective" in the catalog is the whole point, and it must be said rather than left blank (Dataset Documentation).
when One dataset serves several decisions and splitting it is not worth it.
cost Everybody pays for the strictest need, including the consumers who did not want it. Often correct, and it should be a decision rather than an accident (Data Products).
1dataset: analytics.fct_orders2owner: team-commerce # the team that can fix the cause, not the one that noticed3consumers:4 - name: finance-close5 decision: "monthly close begins 07:00 CET on business days"6 agreed_by: j.okoro7 - name: marketing-attribution8 decision: "daily spend allocation, reads at 08:00"9 agreed_by: l.andersson10 11objectives:12 - id: deadline13 indicator: "max(completed_at) for period d-1, read from the published table"14 threshold: "06:00 CET"15 window: "30 rolling business days"16 attainment: 0.99 # the shape; your consumers supply the number17 18 - id: completeness19 indicator: "count(*) for closed period d-1 vs source of record"20 threshold: "within agreed tolerance, stated per dataset"21 window: "30 rolling days"22 attainment: 0.99523 24excluded_from_budget:25 - "documented source-system outage, with an incident reference"26 - "planned backfill announced to consumers at least one business day ahead"27 28on_budget_exhausted:29 - "pause non-reliability work on this pipeline until the window recovers"30 - "or renegotiate the objective with both consumers, in writing"The half that teams omit is everything below objectives. An exclusion list decided in advance is what stops an objective being renegotiated in the middle of an incident, and a stated consequence is what separates an error budget from a chart.
How to build it
Most important first.
- Write the objective from the decision the consumer makes, and work backwards. "The close starts at 07:00" produces a deadline; "we spend budget on yesterday's attribution" produces a freshness bound; "we reconcile to the cent" produces a completeness objective. An objective derived from what the pipeline currently achieves is a description, not a commitment (Data Contracts).
- Set objectives per dataset, and only for datasets that have an owner and a consumer who signed. A platform-wide number cannot be breached by anything and therefore cannot be acted on (Data Ownership).
- Pick one indicator per failure family rather than ten indicators per dataset. In practice a deadline, a freshness bound and a completeness check cover most of what consumers actually notice, and each additional indicator dilutes attention (Alert Fatigue: The Page Nobody Reads).
- State the evaluation window and the exclusions explicitly. Does a source outage consume the budget? Does a planned backfill count as a breach? Answer those before the first incident, because answering them during one is how objectives get quietly renegotiated (Data Incidents).
- Alert on budget burn rate, not on every individual miss. One late run inside budget is information; a burn rate that will exhaust the budget before the window closes is a page (Burn-Rate Alerts: How Fast Is the Budget Going?).
- Publish the objective where the data is consumed — in the catalog entry, next to the table, on the dashboard — not only in the team's own runbook. An unpublished objective is a private opinion about someone else's expectations (Dataset Documentation).
- Review the objective when the consumer's decision changes, and be willing to lower it. An objective nobody can meet is abandoned, and abandoning it costs more credibility than never having set it (Who Owns Data Quality).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- An SLO guarantees nothing about any individual run. It is a statement about a distribution over a window, and a consumer who reads it as "my table will be there tomorrow" has misread it (Availability, SLOs and Error Budgets).
- It guarantees a shared definition of "broken" and a shared method for measuring it. That is genuinely most of its value, and it is available even when the objective itself is modest.
- It does not guarantee correctness unless a correctness indicator is one of the objectives. A dataset can meet a punishing freshness objective every day while being systematically wrong (The Pipeline Succeeded. The Data Is Wrong.).
- It is not a contract in the legal sense and not an SLA. An SLA has a consequence attached; an objective is an engineering target, and conflating them is how teams end up defending numbers rather than fixing pipelines (SLAs: The Promise With Money Attached).
- What is explicitly not guaranteed: that the objective is the right one. It encodes today's understanding of the consumer's decision, and consumers change what they do with data more often than they tell you (Semantic Changes).
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- Measure the indicator from the served table, using the same query a consumer could run. An indicator computed from orchestrator metadata measures your belief about the data rather than the data (Freshness Monitoring).
- Record every evaluation, not just the failures. Without the full series there is no burn rate, no trend, and no way to answer "was it better last quarter" (Pipeline Metrics).
- The indicator misses whatever it does not encode. A deadline says nothing about content; a completeness check says nothing about meaning; and all of them are blind to a value that is present, plausible and wrong (Two Dashboards, Two Numbers).
- The measurement itself can fail. A freshness check that reads a metadata column the pipeline stopped updating reports a stale value as stale forever, or worse, reports nothing and is interpreted as healthy (Quality Alerting).
- A deadline objective ("available by a stated hour") and a freshness objective ("no older than a stated interval") answer different questions and are both needed for a batch dataset. The deadline protects a scheduled decision; the freshness bound protects an unscheduled one.
- A deadline is checkable only once per period, which makes it a poor early-warning signal — by the time it is breached the harm is done. Pair it with a projected-completion signal derived from where the run currently is (Pipeline Observability).
- Freshness objectives compose badly along a chain. A mart is at best as fresh as its slowest input plus its own run time, so an objective on a leaf dataset is an implicit and usually unstated objective on everything upstream of it (Model Layering).
- Tightening any timeliness objective costs run frequency, which costs compute, and the relationship is direct enough that the objective should be set jointly with whoever pays for it (Cost vs Freshness).
- Changing the objective changes the history. A series of "met / missed" verdicts computed under one threshold cannot be compared with one computed under another, so a threshold change should start a new series rather than rewrite the old one (Semantic Changes).
- Changing the *indicator* is a larger change than it appears. Moving from "job finished" to "newest complete hour" typically reveals that the dataset was never meeting the objective anybody thought it was (The Freshness SLO).
- Adding a consumer with a stricter decision does not automatically change the objective, and this is the moment to renegotiate rather than to silently absorb. Two consumers with different needs are two objectives on one dataset, or one objective at the stricter level with the cost that implies (Who Actually Consumes This Data).
- When a dataset is deprecated its objective must be withdrawn explicitly. An orphaned objective generates pages for a table nobody reads, which is the fastest known route to a team ignoring all of them (Alert Fatigue: The Page Nobody Reads).
- A breach is not repaired by a re-run. The consumer already made a decision on missing or stale data, so recovery is two actions: restore the dataset, and tell the people who read it during the gap (Data Incidents).
- Budget consumed by an incident is not refunded, and it should not be. The budget exists so that a sequence of small unrepaired incidents becomes visible before it becomes a pattern (Error Budgets: Unreliability You Are Allowed to Spend).
- When the budget is exhausted, the useful response is a policy agreed in advance — pause feature work on that pipeline, or explicitly and publicly lower the objective. Deciding what "budget exhausted" means during the incident guarantees it means nothing (Reliability Patterns).
- Backfilling the data after a breach improves the dataset and does not retroactively meet the objective. Recording it as met because the numbers are now right destroys the series (Validating a Backfill Before You Publish).
What can go wrong
- An objective set from what the pipeline currently does, so it is always met and never informs a decision.
- An objective set from ambition, missed every week, and quietly ignored within a quarter (Alert Fatigue: The Page Nobody Reads).
- An indicator computed from orchestrator state rather than from the data, so every silent-correctness failure is scored as a success (The Pipeline Succeeded. The Data Is Wrong.).
- A platform average that hides the three datasets anyone would notice.
- A monthly evaluation window that forgives everything on the first of the month, so a bad week in the last days of a month costs nothing.
- The mitigation failing: burn-rate alerting configured with a window so long that the budget is gone before the alert fires (Burn-Rate Alerts: How Fast Is the Budget Going?).
- An objective owned by the platform team for a dataset produced by a product team, so the accountable party cannot fix the cause (Who Owns Data Quality).
- "Our DAGs are 99.9% successful, so the platform is reliable." Task success measures whether code ran. Every failure class this domain cares about most — missing rows, duplicates, staleness, semantic drift — produces successful runs (The Pipeline Succeeded. The Data Is Wrong.).
- "An SLO is a promise that the data will be there." It is a statement about a proportion over a window. Any individual day can miss without the objective being breached, and consumers must be told that in those words (Availability, SLOs and Error Budgets).
- "We should aim for 100%." An objective of one hundred per cent has no error budget, which means every incident is a crisis and no reliability work can ever be prioritised against features. It also cannot be met, because the sources are not yours (Error Budgets: Unreliability You Are Allowed to Spend).
- "The figures in the examples are the right ones." They are examples of the shape an objective takes, not recommendations. A deadline of 06:00, a freshness bound of fifteen minutes and a run-success target of 99.9% are placeholders for numbers only your consumers can supply.
- "The data team owns the SLO." The team that can fix the cause owns it. When the cause is a source system, an objective owned by the data team is an objective owned by someone with no lever (Who Owns Data Quality).
Operating it
- Per dataset: objective, current attainment over the window, budget remaining, and burn rate. Four numbers, on one row, visible to the consumer as well as the owner (Dashboards Built Around Questions).
- The indicator series itself, so trends are visible before the objective is breached. Attainment tells you what happened; the series tells you what is about to (Regression or Tuesday? Telling a Real Change from Noise).
- Breach events annotated with cause and with whether the budget was consumed, which is the raw material for the quarterly conversation about where reliability work should go (Reading a Timeline: Observation Order Is Not Causal Order).
- Deployment and schedule-change markers on the same axis, because a step change in attainment usually has a change behind it ("What Changed?" — Deploy Markers and the Invisible Deploys).
- At 10x datasets, objectives cannot be hand-written. They become a declared property of each dataset, generated into checks, or they do not exist (Data Contracts).
- At 100x, the population is stratified: a small tier of datasets with negotiated objectives and real pages, and a long tail with a default best-effort statement. Attempting a real objective on everything produces alert volume nobody reads (Alert Fatigue: The Page Nobody Reads).
- At 10x consumers per dataset, the objective becomes the coordination mechanism — it is cheaper to publish one promise than to answer the same question in eighty conversations (Data Discovery).
- Nothing about the mechanism changes with data volume. This is an organisational instrument that happens to be measured with a query (Data Governance).
- The measurement is cheap: a small query per dataset per evaluation, whose cost is a rounding error against the pipeline it measures — unless it is written as a full scan of the table it checks, which happens more often than it should (Scan Cost).
- The expensive part is meeting the objective. Every tightening buys latency with compute, redundancy or engineering attention, and each of those is paid continuously (Cost vs Freshness).
- The cost of *not* having one is paid in incident time and in trust, both of which are real and neither of which appears on any platform bill (Trusting Data).
- Attainment history is a small append-only series and is worth retaining well past the evaluation window — it is the cheapest artefact in the platform and the only evidence of what reliability used to be (Keeping Raw History: The Recovery Position and the Liability).
- A tight objective buys consumer confidence and costs run frequency, redundancy, on-call load and engineering attention that is then not spent on new datasets.
- A loose objective is cheap and honest and tells consumers less than they wanted. It is very often the right answer, and saying so out loud is more useful than a tight objective nobody meets.
- Per-dataset objectives buy actionability and cost maintenance: each one is a small ongoing obligation, and a hundred abandoned objectives are worse than five living ones.
- Error budgets buy a principled way to decide between reliability and features, and cost an organisation that must actually honour the policy when the budget runs out. A budget with no consequence is a chart (Error Budgets: Unreliability You Are Allowed to Spend).
Dataset review questions
This lesson uses the shared review exercise.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- GENERALIndicator, objective, window and error budget are the same instrument used in service reliability, applied to a dataset instead of an endpoint; what changes is that data has a second failure family — silent incorrectness — that has no equivalent in request-serving availability and needs its own indicator.
- ORG-SPECIFICAn error budget only functions where the organisation will honour the policy attached to it; in a company that ships regardless, the budget is a chart and the objective decays into a target nobody defends. The technical mechanism is identical in both cases and the outcome is not.
- SCALE-SPECIFICBelow roughly a dozen consumed datasets, a written objective per dataset is proportionate and a conversation would do most of the work; above a hundred, objectives must be declared as data and generated into checks, and a tail of datasets with no objective at all becomes the correct answer rather than a gap.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns the practice this borrows from — indicators, objectives, error budgets and the organisational agreement that gives a budget teeth — and owns the on-call rotation that a data page eventually lands in.
- — Distributed Systems owns why a hard guarantee is unavailable in the first place: the sources, the network and the compute are all things that fail independently, so a promise about a dataset can only ever be probabilistic.