Pipeline Observability
What an orchestrator genuinely knows, what it structurally cannot know, and how to make a task-level signal say something about data.
Who needs this, what one row is, and why the obvious build breaks
Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.
The DAG is green. Precisely which claims about my data does that entitle me to make?
The engineer on call, who needs to know within a minute whether anything is broken and where; and the analyst, who wants to know whether today's table is finished — a question the orchestrator can answer and almost never exposes (Who Actually Consumes This Data).
One task attempt: a single execution of one unit of work for one logical period. Attempts matter rather than tasks because a task that succeeded on its fourth attempt has a very different story than one that succeeded immediately, and the retry count is where that story lives (Retries in Pipelines).
Enable the orchestrator's built-in failure email and a duration SLA on the longest job. This is genuinely worth doing — loud failures are real failures and finding out from a machine beats finding out from a person.
The task succeeds having processed zero rows, because the upstream partition it read was empty. No orchestrator distinguishes "did nothing" from "had nothing to do" unless you make it (Partial Failure).
- The task succeeds having processed zero rows, because the upstream partition it read was empty. No orchestrator distinguishes "did nothing" from "had nothing to do" unless you make it (Partial Failure).
- A task is retried three times and eventually succeeds. The alert never fires, the run history shows green, and the intermittent fault that will take the platform down next month goes unrecorded.
- Two tasks depend on the same upstream and one of them was accidentally left out of the dependency graph, so it runs against yesterday's data every day and succeeds every day (Task Dependencies).
- The duration SLA fires whenever the source is busy. It is widened, then widened again, and now it cannot detect the run that hung for six hours.
- The DAG is refactored and a model is dropped. There is no failing task, because there is no task (The Transformation DAG).
What is actually happening
- An orchestrator maintains a state machine per task instance — queued, running, up-for-retry, success, failed, skipped, upstream-failed — and a database of those transitions. Everything it can tell you is a projection of that state machine (Orchestration).
- The critical word is *skipped*. Branching and short-circuit logic produce runs where a task never executed and the DAG is still green, which is correct behaviour and a common way for a table to stop being built without anything failing.
- Asset-oriented orchestrators change what the state machine is about: the tracked entity is the dataset that was materialised rather than the process that ran. That is an architectural difference, not a feature difference, and it is what lets the tool answer "when was this table last built" instead of "did this task run".
- A task can therefore be made to report on data if — and only if — it emits facts about what it wrote. Rows in, rows out, partitions touched, watermark advanced. None of that is inferable from an exit code; all of it is trivial for the task itself to record (Pipeline Metrics).
- Retries are the mechanism that most distorts the signal. A pipeline configured to retry on any exception converts a persistent data problem into a delayed success, and the run history that results is a record of how long the platform hid the fault (Retries in Pipelines).
What the orchestrator actually knows
It is worth being precise about the information content of a scheduler, because the phrase "the pipeline is monitored" usually means this and only this. The stages below are what exists; the guarantees column is what each stage entitles you to claim.
Read it as a chain of increasingly weak inferences. The executor knows a process exited zero. The task state machine knows that transition happened for this logical period. The DAG knows every upstream state was success or skipped. By the time you reach "the data is fine", every link has been an inference and none has touched a row.
The last stage is the one worth designing. A task that records what it wrote is the only place in this chain where a fact about data enters the system, and it costs a few lines inside work that already has the numbers in hand.
- 1Scheduler
Decides that a logical period is due and enqueues the tasks whose dependencies are satisfied.
guarantees That a run was created for a period it knew about. Nothing about periods it was not configured for.
fails by Not scheduling at all — a paused DAG, a changed cron, a removed task. There is no failure event, so failure-only alerting sees nothing.
- 2Executor
Places the task on a worker and starts the process.
guarantees The process started, and its exit status will be recorded.
fails by Queueing behind other work indefinitely. A task waiting is neither succeeded nor failed and is easy to alert on only if you look for queue age.
- 3Task attempt
Runs the code for one task, once.
guarantees An exit status and a duration. Nothing whatsoever about rows.
fails by Exiting zero after catching every exception, or exiting zero having processed an empty input.
- 4Retry policy
Re-runs a failed attempt up to a bound, usually with backoff.
guarantees Bounded automatic recovery from transient faults, if the task is idempotent.
fails by Turning a persistent fault into a late success, so the first human-visible symptom is stale data rather than a failure.
- 5Dependency state
Releases downstream tasks when upstream tasks reach an acceptable state.
guarantees The upstream *task* reached success or skipped.
fails by Treating skipped as satisfied, so a branch that did not run releases work that reads a table nobody rebuilt.
- 6Run record
Persists the outcome, timings and attempt count for the period.
guarantees An auditable history of executions, retained for as long as the metadata database is retained.
fails by Being reset by a rename, or retained so briefly that no baseline for duration or retry rate exists.
- 7Emitted data facts
Records rows read, rows written, partitions touched and the watermark reached — if the task volunteers them.
guarantees A fact about the dataset, sourced from the process that produced it.
fails by Not existing. This stage is optional in every orchestrator and is where the control plane stops being blind.
Six stages that describe processes and one that describes data. Everything this module adds afterwards exists because the seventh stage is the only one anybody downstream cares about.
Turning a green run into a claim about data
The gap between "the task succeeded" and "the dataset is right" is closed in one place: the task itself, at the moment it writes. It is the only point in the system that knows both the intent and the outcome, and it costs almost nothing to make it say so.
The pattern is to assert against an expectation rather than against zero. Zero rows is sometimes correct — a Sunday, a closed market, a source that genuinely had nothing — and a check that cannot express that difference will be muted within a month. An expectation derived from the period's own history survives, and the false-positive rate is what decides whether the check still exists next year (Volume Anomalies).
The second half is failing *loudly* when the assertion breaks. A task that logs a warning and publishes anyway has converted a silent data problem into a slightly less silent one. Failing the run keeps the previous good version in place, which is almost always what a consumer would choose if asked (Atomic Publish).
The task reads yesterday's partition, transforms it, writes the result and exits zero. The orchestrator records success and duration. If the input partition was empty, the output is empty and the run is green.
The same task records rows read, rows written, the period processed and the watermark it advanced to; asserts that rows written is within the band established by the same weekday historically; and fails the run when it is not, leaving the previous published version in place.
An exit code is a statement about the process and there is no inference from it to the contents of a table. The task is the only component that holds both numbers at once, so instrumenting anywhere else means re-querying the warehouse to learn something the job already knew.
1-- One row per task attempt, written by the task itself.2-- The columns that matter are the last four: no orchestrator fills them in.3create table pipeline_run_facts (4 dag text not null,5 task text not null,6 logical_period date not null, -- the period processed, not the wall clock7 attempt int not null,8 started_at timestamptz not null,9 ended_at timestamptz,10 status text not null, -- success | failed | skipped11 rows_read bigint,12 rows_written bigint,13 partitions_written int,14 watermark_to timestamptz15);16 17-- Runs that were expected and never happened. Failure-only alerting18-- cannot see these, and they are indistinguishable from staleness19-- to every consumer downstream.20select c.task, c.logical_period21from expected_runs c22left join pipeline_run_facts f23 on f.task = c.task24 and f.logical_period = c.logical_period25 and f.status = 'success'26where c.logical_period < current_date27and f.task is null;28 29-- Succeeded, and wrote nothing. The domain's signature failure.30select dag, task, logical_period, attempt31from pipeline_run_facts32where status = 'success'33and coalesce(rows_written, 0) = 034and logical_period >= current_date - 7;Both queries are impossible without the four columns the task volunteers. The second one is the query that finds the incident everyone else finds from a dashboard.
What deserves a human at three in the morning
Every signal in this module can be wired to a page, and wiring all of them is how a platform ends up with an alert channel that people mute. The decision is not which signals to compute — compute all of them, they are cheap — but which ones interrupt a person.
The useful discriminator is the consumer, not the technology. A dataset that a finance close depends on at 08:00 has a hard deadline and earns a page at 03:00 because there is time to fix it. The same technical failure on an exploratory table earns a row in a report. That is a tiering decision and it belongs in the catalog next to the dataset, not in the alerting tool (Data Ownership).
The second discriminator is whether a human can do anything. Paging for a failure whose only remedy is "wait for the upstream vendor" trains people to ignore pages. Route it to a status board instead, and page on the consequence — the freshness breach — if the wait becomes long enough to matter (The Freshness SLO).
A signal has fired. Who needs to know, how fast, and what can they do about it?
when A tier-1 dataset with a hard consumer deadline will miss it, and a human can still act — a failed run, a freshness breach with time left, a contract rejection on a table finance closes on.
cost Interrupts a person at night. Every false positive here is repaid with interest in future ignored pages, so the tier list must be short and honest.
when The data is wrong or late but no deadline is imminent, and the fix belongs to a specific team — a distribution drift, a schema change that needs a producer conversation.
cost Slower response, and tickets rot. Needs a named owner and a review cadence or it is a queue that only grows (Data Ownership).
when Consumers need to know the dataset is degraded but no engineering action is available — a source outage, a vendor incident, a period genuinely still filling.
cost Only works if consumers actually look. Requires the board to be linked from the dashboards themselves rather than living in a wiki.
when The signal is informative rather than actionable — retry counts, duration drift, coverage percentage.
cost Nobody reads it unless it is reviewed on a schedule. A trend with no review meeting is storage.
when The signal has fired repeatedly without a real incident behind it.
cost The honest and least-taken option. Suppressing without fixing is how a platform accumulates checks that can no longer fire (Alert Fatigue: The Page Nobody Reads).
Orchestrators differ architecturally in what they track: some model tasks and some model the assets those tasks materialise, and transformation frameworks add their own model-level test results on top. Which metadata is exposed, and whether it is queryable rather than only rendered, changes between versions — verify against current documentation before designing alerting around a specific field.
How to build it
Most important first.
- Make every task emit what it wrote: rows read, rows written, the period it processed, and the high-water mark it advanced to. This is the cheapest change with the largest effect in this module (The High-Water Mark).
- Assert on emptiness explicitly. A task that legitimately produces no rows should say so and pass; a task that produces no rows because its input vanished should fail. Only the pipeline author can tell those apart, and only at write time.
- Separate "the code failed" from "the data is unacceptable" in the alert. They have different owners, different urgency and different fixes, and merging them into one channel guarantees both are triaged badly.
- Alert on *runs that did not happen*, not only on runs that failed. A missing run and a failed run look identical to a consumer and completely different to a monitor watching only failures (When a Task Fails Mid-DAG).
- Bound retries and record them. Three attempts with backoff is a resilience feature; unlimited retries is a way of turning an incident into a mystery (Without Jitter, Every Client That Failed Together Retries Together).
- Publish the run history as a queryable dataset rather than only as a UI. "Which tasks have retried most this month" is a design question, and it needs SQL (The Data Quality Dashboard).
What this actually promises
Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.
- Task success guarantees the process exited zero inside its timeout. That is the entire content of the signal.
- Dependency satisfaction guarantees the upstream task reached a success state — not that the upstream *dataset* is complete, which is a different claim that only a data-plane check can make.
- A green run guarantees nothing about periods other than the one it processed. A backfill that failed silently three weeks ago is invisible in today's green run (Backfills).
- Nothing in the orchestrator guarantees that the set of tasks is the set of tasks you need. Coverage of the DAG over the datasets that exist is an assumption, and it decays.
Can I trust it?
A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.
- The check that turns pipeline observability into data observability: assert
rows_written > 0— or against a period-specific expectation — inside the task, and fail the run when it is violated. It converts the domain's signature silent failure into a loud one. - It misses everything about the rows that were written. A task that writes the right number of completely wrong rows passes it happily.
- It also misses the run that never started, because a check inside a task cannot fire when the task does not exist (Freshness Monitoring).
- Task-level signals are as fresh as the schedule. An hourly DAG can tell you within the hour; a monthly one tells you a month late, which is why cadence-matched data checks matter more the coarser the schedule is.
- Retries add latency between the first symptom and the first alert, exactly proportional to attempts times backoff. That interval is invisible unless retries are recorded as their own metric.
- A dependency-blocked task reports nothing while it waits. Long queue waits therefore look like silence rather than like a problem — the fix is to alert on queue age, borrowing the pattern from Depth Is Not an Emergency; Age Is.
- DAG changes are code changes and deserve review as such: a removed model is a removed dataset for everyone downstream, and the pull request is the last cheap moment to notice (Impact Analysis).
- Renaming a task resets its history in most orchestrators, so duration baselines and retry statistics start from zero. That is a real cost of refactoring and an argument for keeping stable identifiers.
- When schedules change, historical run comparisons stop being comparable. Record the schedule interval alongside the run so that "this run took longer" can be distinguished from "this run covered more".
- Re-running a task must be safe by default, which makes idempotency a monitoring concern and not only a correctness one: an operator who is unsure whether a re-run is safe will delay it, and the delay is the outage (Idempotent Data Pipelines).
- Clearing and re-running a range must be visible in the run history as exactly that, so a later investigation can tell a backfill apart from the original run (Planning a Backfill).
- After a fix, re-run forward from the earliest affected period rather than re-running everything. The run history is what tells you which period that was, which is a reason to retain it well past the point where it feels useful (Reprocessing vs Retrying).
What can go wrong
- A task that catches every exception and exits zero — the most damaging four lines of code in a data platform.
- Retry-until-success masking a persistent upstream problem, so the first alert anyone sees is a freshness breach hours later.
- A sensor or wait-for-upstream step that times out into
skippedrather thanfailed, producing a green run with a missing model. - Alerting configured per task in a UI, so the checks drift from the DAG and nobody can say which tasks are actually alerted on.
- Monitoring so tightly coupled to the orchestrator that a scheduler outage removes both the pipeline and the ability to see it is gone.
- "The DAG is green, so the data landed." Green means the processes exited zero. Skipped tasks, empty inputs and removed models all produce green (The Pipeline Succeeded. The Data Is Wrong.).
- "Retries make the pipeline reliable." Retries make transient faults invisible, which is valuable, and make persistent faults invisible for longer, which is not. Their count is a signal and should be treated as one.
- "We do not need data checks, our tasks validate their inputs." Input validation catches what the producer sent wrong. It cannot catch what the producer never sent (Missing Rows).
- "Duration is a good proxy for health." A broken run is often the fastest run of the week, because there is nothing to process and nothing to raise.
Operating it
- Run outcome, attempt count and duration per task per period, stored as data rather than only rendered in a UI.
- Rows read and rows written per task attempt — the single most useful field to add, and the one no orchestrator populates for you (Pipeline Metrics).
- Runs expected versus runs observed per schedule, which is how a task that stopped being scheduled becomes visible.
- Queue wait time before execution, which distinguishes "the job is slow" from "the cluster is busy" (The Backlog Arithmetic: Four Levers and a Drain Time).
- At ten times the number of tasks, per-task alert configuration collapses. Alerting has to be derived from dataset tier rather than attached to tasks by hand.
- At a hundred times, the run-history table itself becomes an analytical dataset with partitioning and retention decisions of its own — a pleasing and slightly embarrassing recursion.
- The scheduler becomes a shared bottleneck long before the compute does. A DAG with thousands of short tasks spends most of its wall-clock time in scheduling overhead, which shows up as duration drift with no code change (Straggler Tasks).
- Run metadata is small and grows linearly with tasks times periods. The cost that surprises people is retention of task *logs*, which grows with verbosity and dwarfs the metadata by a wide margin (The Log Bill and What It Is Buying).
- Retries multiply compute for the failing task and, when the task is expensive, make a transient fault an expensive one. Bounding attempts is a cost control as much as a reliability one (Compute Waste).
- Emitting row counts is nearly free when the task already has the dataframe or the result cursor, and expensive when it requires a second query against the warehouse. Prefer the former; the difference is a design decision at write time.
- Making tasks fail on unacceptable data converts silent wrongness into loud outages. That is the correct trade and it will page someone at three in the morning for a problem the business would not have noticed until Tuesday.
- Rich per-task instrumentation is code in every task. It is worth writing once as a shared wrapper and is a maintenance burden if it is copied into two hundred models.
- Asset-oriented orchestration gives dataset-level state for free and constrains how you express work. Task-oriented orchestration is more flexible and leaves the dataset question entirely to you.
Where this applies
Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.
- TOOL-SPECIFICTask-centric and asset-centric orchestrators model different entities: one records that a process ran, the other records that a dataset was materialised. The same "green" therefore supports a stronger claim in the second than in the first, and porting alerting between them is not a translation.
- GENERALThe underlying limit is universal and not a tooling gap: an exit code is a statement about a process, and no scheduler can infer row-level outcomes from it without the task volunteering them.
- SCALE-SPECIFICPer-task alert configuration is workable for tens of tasks and unmanageable for thousands, where alerting must be generated from dataset tier instead. The crossover is roughly where nobody can name every DAG from memory.
Where the depth lives
This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.
- — DevOps / Production Engineering owns how DAG code and check definitions are tested, versioned and rolled back. A removed model is a deployment event, and treating it as one is the cheapest place to catch it.