ConnectionsGENERALTOOL-SPECIFICORG-SPECIFIC

Data Engineering and Observability

Observability & Performance owns why it is slow. This domain owns whether it is correct, complete and fresh. Different questions, different signals, different toolkits.

What actually happens How to build it Can I trust it?

Who needs this, what one row is, and why the obvious build breaks

Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.

The question

Why is monitoring a data platform a different discipline from monitoring a service, and which signals belong to which?

Who needs this

The on-call engineer holding a page at 07:00 who needs to know, within minutes, whether this is a system that is unhealthy or data that is wrong — because those have entirely different first moves and the dashboards for one tell you almost nothing about the other (Debugging a Data Incident).

What one row is

The unit being observed is one dataset partition, per run. Not a request, not a span, not a host. Its row count, its freshness, its schema and its value distributions are the four signals, and every one of them is a property of a dataset rather than of the process that produced it.

The obvious build

Instrument the data platform the way the services are instrumented: task success and failure, run duration, CPU and memory, an alert when a DAG fails. It is familiar, it reuses the existing tooling, and it catches every failure that raises an exception.

Why it breaks

The extract succeeds and returns zero rows because the source API changed its pagination. Every task is green, the transformation runs on an empty set, and the dashboard shows yesterday as a quiet day (Volume Anomalies).

How it breaks with real data

The extract succeeds and returns zero rows because the source API changed its pagination. Every task is green, the transformation runs on an empty set, and the dashboard shows yesterday as a quiet day (Volume Anomalies).
A dimension gains duplicate rows, the fact join fans out, the job finishes faster than usual because nothing errored, and revenue is reported at several times its true value (Duplicate Rows).
A source stops sending. Nothing fails — there is simply nothing to process — and the last-known number sits on the dashboard for three days looking exactly like a real number (Stale Dashboards, Freshness Checks).
The alert fires on task duration. The task is fast, because the input is empty. Duration monitoring is anti-correlated with this failure (Pipeline Observability).
A performance dashboard shows warehouse query latency rising and the data team spends a day tuning, when the real change is a source that started sending a hundred times more rows for one key (Data Skew).
Every quality check is defined per table by whoever built it, so during an incident nobody can answer "which datasets are currently failing" without opening six tools (The Data Quality Dashboard).

Source→Ingestion→Raw→Transformation→Validation→Storage Model→Serving→Consumer→Observability

What is actually happening

The two disciplines answer different questions and the signals do not substitute for each other. Observability & Performance asks: is the system healthy, where is the bottleneck, what is the tail latency, which resource is saturated. This domain asks: did everything arrive, is it counted once, does it mean what it meant last week, and how old is it (Data Observability).
The four data signals are freshness, volume, schema and distribution. Freshness answers "how old is the newest complete record". Volume answers "does today look like a normal day". Schema answers "did the shape change". Distribution answers "did the values change without the shape changing" (Pipeline Metrics, Distribution Tests).
Task-level monitoring is structurally blind to the failures that matter here, because those failures produce successful runs by construction. A green DAG is evidence that code ran (The Pipeline Succeeded. The Data Is Wrong.).
Lineage is this domain's equivalent of a distributed trace. A trace answers "which service call was slow" by following one request across systems; lineage answers "which upstream produced this number" by following one dataset across models. Both exist because the failure is never where the symptom is (Data Lineage, Lineage Debugging).
Some questions genuinely belong to the neighbour and should be handed over immediately. Why a query is slow, where a shuffle spilled, why a cluster is saturated, why tail latency moved — that is performance work with a performance toolkit, and re-deriving it here would produce a second, worse answer (Query Optimizers).
The two are coupled in one direction that matters: a freshness SLO is a latency budget. If a dataset promises hourly freshness, every stage's duration and every queue's backlog is part of that budget, and that is where the performance toolkit becomes directly relevant to a data commitment (The Freshness SLO, Pipeline SLOs).
Alert design fails the same way in both disciplines and for the same reason. A check on every column of every table produces noise that gets muted; a check on nothing produces incidents reported by executives. The portfolio is chosen from what a consumer would actually notice (Quality Alerting, Alert Fatigue: The Page Nobody Reads).

Two questions that look like one

Observability & Performance is a mature discipline with a strong toolkit: metrics, traces, logs, profiles, percentiles, saturation, queueing. Every bit of it is about whether a system is healthy and where its time goes. None of it is about whether a number is right.

This domain asks a question that toolkit cannot express. "Is fct_orders missing four hundred rows for last Tuesday" is not a latency question, has no percentile, produces no error, and appears in no trace. It is answered by comparing a dataset to a source, and by knowing what one row is supposed to represent.

The division that works in practice: if the complaint is "it is slow", it belongs to the neighbour. If the complaint is "it is wrong", it belongs here. The overlap is exactly one signal — freshness — because being late is both a latency outcome and a completeness statement.

We teach	Depth lives in	The mechanism that crosses
Is this dataset fresh enough for its consumer?	Observability: SLOs: A Target, a Window, and a Reason, SLIs: Measuring What the User Actually Feels, Error Budgets: Unreliability You Are Allowed to Spend	A freshness SLO is a latency budget spread across every stage. This is the one signal both disciplines genuinely share (The Freshness SLO).
Did today's data arrive completely?	Observability: Four Metric Types, Four Questions, Counters: The Slope Is the Signal, anomaly baselines	Row count per partition against its own history is a metric in their sense and a completeness statement in ours (Volume Anomalies).
Did the values change meaning?	Observability: Cardinality: The Label That Took Down Monitoring, Label Sets That Survive a Year	Distinct counts and category shares are the same measurement; here they are watched for semantic drift rather than for cost (Distribution Tests).
Why is this model slow?	Observability & Performance: The Slow Query Workflow, Reading a Flame Graph, Saturation: The Reading Utilization Cannot Give You	Hand it over. Bytes scanned, shuffle volume and stage skew are diagnosed with their toolkit, not with lineage (The Shuffle, Data Skew).
Why is the queue backing up?	Observability & Performance: Queueing: Why Systems Get Slow Before They Get Broken, Little's Law as Working Intuition, The Backlog Arithmetic: Four Levers and a Drain Time	Consumer lag is a queueing problem whose consequence is a completeness problem once retention passes (Retention and Replay).
Where did this number come from?	Observability: Distributed Tracing, Trace, Span, Attribute, Status	Lineage is the dataset-scale analogue of a trace: follow the dependency graph instead of the call graph (Lineage Debugging).
Did a deploy cause this?	Observability: "What Changed?" — Deploy Markers and the Invisible Deploys, Reading a Timeline: Observation Order Is Not Causal Order	The same technique, applied to freshness and quality charts rather than to latency charts (Data Incidents).
Which alerts are worth having?	Observability: Alerts Worth Waking Someone For, Alert Fatigue: The Page Nobody Reads, Dashboards Built Around Questions	Identical failure mode, different subject: a wall of unowned column assertions gets muted exactly like a wall of unowned host alerts (Quality Alerting).

The four signals and what each one still misses

A service is monitored on the golden signals — latency, traffic, errors, saturation. A dataset has its own four, and they map onto nothing in that list. Freshness, volume, schema and distribution answer, in order: is it late, is it short, did it change shape, did it change meaning.

The reason to learn them as a set is that each one's blind spot is another one's strength, and no single one is worth deploying alone. Volume without distribution catches a missing day and misses a corrupted column; distribution without freshness watches a table that stopped updating and reports it as perfectly stable.

The misses column is where the honesty lives. A team that deploys these four and believes it has coverage has swapped one false confidence for another — the difference is that this one is at least explicit about its edges (The Dimensions of Data Quality).

The four dataset-level signals

Check	Expresses	Catches	Still misses
Freshness: age of the newest complete partition against the dataset's SLO	The data is recent enough for the decision it drives.	A stopped pipeline, a stopped source, an upstream that silently produced nothing, a schedule that was disabled and forgotten.	Fresh data that is wrong — the most common state of a broken pipeline; and it raises a false alarm on any period where the source genuinely had nothing to send (Freshness Checks).
Volume: row count per partition against the same weekday historically	Today looks like a normal day for this dataset.	A partial load, a filter that became too broad, a source outage, a schema change that nulled a column used in a `WHERE` clause, a fan-out join.	Slow drift below the threshold; and every error that preserves row count, which includes almost all value-level bugs (Volume Anomalies).
Schema: declared columns, types and nullability against the contract	The shape the consumer was promised is the shape that arrived.	Added, removed, renamed or retyped fields; a type widening that will silently truncate downstream.	Every semantic change, which is the expensive category — a column that keeps its name and type and changes from gross to net passes cleanly (Semantic Changes).
Distribution: null rate, distinct count and category share per important column	The values mean what they meant last week.	A new enum value from an unannounced release, a null rate that steps, a currency or unit change, a segment that vanished.	Changes that are within normal variance; anything in a column nobody nominated as important; and a genuine business change, which it reports identically to a bug (Distribution Tests).
Reconciliation: aggregate for a closed period against the source system	What we hold matches what actually happened.	Missing rows, duplicated rows, bad joins, and the cumulative effect of everything the four signals above let through.	Any open period; errors present identically in source and warehouse; and it requires the source to be queryable, which excludes exactly the SaaS sources that need it most (Reconciliation).

Deploy them as a portfolio. The argument for each one is what the others cannot see, and a platform with only the first two has coverage of arrival and none of meaning.

Triage: is this slow, or is this wrong?

The most expensive minutes in a data incident are the ones spent in the wrong discipline. A performance investigation into a correctness problem produces a faster wrong number; a correctness investigation into a saturated warehouse produces a day of reading SQL that was never at fault.

The rows below are a triage table. Each starts from a symptom as it is actually reported — usually by a person, in a message, without a stack trace — and names the discriminating observation that decides which toolkit to open.

Note how many of the discriminators are cheap: a row count, a freshness value, a distinct-key comparison. The expensive part of a data incident is almost never the diagnosis, it is the scoping — which range, which consumers, which downstream models are already contaminated (Impact Analysis).

Which discipline owns this symptom

Trigger	Symptom	Cause	Response
"The dashboard is slow to load"	Query latency rising, no change in the numbers themselves.	A performance problem: scan volume, concurrency, layout, or a plan that changed when statistics did (Query Optimizers).	Hand it to the performance toolkit. Look at bytes scanned and the plan before looking at the SQL, and check whether partition pruning is still happening (Partition Pruning).
"Revenue looks wrong for last week"	A number moved and no system reported anything.	A correctness problem: missing rows, duplicates, a semantic change, or a BI-layer filter.	Compare distinct business keys with the source for the closed period, then walk lineage upstream to the first model where the count diverges (Lineage Debugging).
"The dashboard has not changed since Tuesday"	A stable, plausible number that is stale.	Ambiguous by design — a stopped pipeline, a stopped source, or a genuinely quiet period all look identical.	Read freshness per dataset, then row counts upstream. If the pipeline ran and produced an empty partition, the fault is upstream of the pipeline (Stale Dashboards).
"The nightly job used to take an hour and now takes four"	Duration growth with correct output.	Usually skew or file-count growth: one key or one partition now dominates, or the input became many small objects (Data Skew, File Size and the Small-Files Problem).	Performance toolkit, with one data-specific check first: look at per-key volume and per-partition file counts before tuning anything (Straggler Tasks).
"The job finished much faster than usual"	A green, fast run and a quiet dashboard.	Almost always an empty or truncated input. Speed is the symptom, not the good news.	Treat a large negative duration anomaly as a data alert, not a performance win. Check input row counts for the run immediately (Volume Anomalies).
"Consumer lag is growing"	A queueing symptom with a completeness deadline attached.	A performance problem now and a data-loss problem at the moment lag exceeds retention (The Backlog Arithmetic: Four Levers and a Drain Time).	Both toolkits. Diagnose the saturation with the performance toolkit and, in parallel, compute how long remains before retention makes the backlog unrecoverable (Retention and Replay).

How to build it

Most important first.

Publish freshness, volume, schema and quality status per *dataset*, in one place, so "is this table healthy" is answerable by the person reading it and not only by the team that built it (The Data Quality Dashboard).
Alert on the data signals, and use task status as one input rather than as the definition of health (Pipeline Observability).
Set freshness and completeness SLOs per dataset, with the consumer's decision as the justification, and treat the budget the same way a service does (The Freshness SLO, SLOs: A Target, a Window, and a Reason).
Build lineage before you need it. During an incident the two questions are always "what feeds this" and "what else does it feed", and both are unanswerable at 3am if the graph was never captured (Data Lineage, Impact Analysis).
Route the question to the right toolkit deliberately. "Slow" goes to profiling, plans and queueing; "wrong" goes to lineage, reconciliation and distribution (Debugging a Data Incident).
Keep the check count proportional to consequence. A small number of checks that people act on beats a wall of assertions that everyone has learned to mute (Quality Alerting).

What this actually promises

Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.

A data check guarantees only that the assertion you wrote held on the data you ran it against, at the moment you ran it. It never guarantees correctness (Data Tests).
A freshness monitor guarantees that data has arrived recently. It says nothing about whether that data is right, and it fires falsely for any period the source genuinely had nothing to send (Freshness Monitoring).
A pipeline SLO is a commitment about the platform's behaviour, not about the source's. If an upstream system stops producing, the SLO is breached by something you do not control, which is why the error budget conversation has to include the producer (Pipeline SLOs, Error Budgets: Unreliability You Are Allowed to Spend).
Lineage guarantees the dependencies it captured. A dependency expressed in a BI tool, a notebook or a hand-run script is invisible to it, and that is where impact analysis under-reports (Data Discovery).

Can I trust it?

A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.

The check that would catch this

The portfolio that catches the most for the least effort is four checks per served dataset: freshness against its SLO, volume against its own history, uniqueness on the business key, and reconciliation against the source for a closed period (Reconciliation).
Together they miss slow drift, any error that preserves both volume and uniqueness, and every semantic change — which is most value-level bugs.
They also miss anything that happens after the model: a filter or join applied in the BI layer is invisible to every test the data team wrote, and is a startlingly common root cause (Where Did This Number Come From?).

Freshness

Freshness is the one signal that is simultaneously a data-quality property and a latency property, which is exactly why it is the natural meeting point of the two disciplines.
Measure it as the age of the newest *complete* record, not the newest record. A partition that is half-written is not fresh, it is misleading (Freshness Checks).
Freshness monitoring has a structural false-positive: a quiet period looks identical to a broken pipeline, and distinguishing them requires knowing the source's expected pattern (Volume Anomalies).

When the schema or meaning changes

Schema monitoring is the cheapest early warning available, because a structural change is machine-detectable and usually precedes the semantic one by a release (Schema Evolution).
Distribution monitoring is what catches the semantic change that schema monitoring cannot — a null rate that jumps, an enum that gains a value, a mean that steps (Distribution Tests).
Checks themselves need versioning. A threshold that was tuned for last year's volume becomes noise at this year's, and a muted check is worse than no check because it looks like coverage.

How to re-run this safely

The first move in a data incident is to determine the affected range and the affected consumers, and to say so publicly. Repairing before scoping produces a second incident inside the first (Data Incidents).
Repair is a bounded backfill validated against the source before publishing, not a re-run of the schedule (Planning a Backfill, Validating a Backfill Before You Publish).
After repair, add the check that would have caught it — and write down what that check still misses, so the next person does not read its presence as coverage (Data Quality).

What can go wrong

Failure modes

Monitoring only task status, so every failure that produces a successful run is invisible.
Alerting on duration, which is anti-correlated with the empty-input failure it is expected to catch.
Freshness monitored per pipeline rather than per dataset, so a consumer cannot see the health of the table they are reading.
Alert fatigue: hundreds of column-level assertions, all owned by nobody, all muted within a quarter (Alert Fatigue: The Page Nobody Reads).
The mitigation failing: a freshness check that reads the pipeline's own metadata table rather than the data, so it stays green when the pipeline writes an empty partition on schedule.
A performance investigation opened for a correctness problem, or the reverse, costing a day before anyone re-frames the question.

Misreads

"The pipeline is green, so the data is fine." Task status reports on code execution. The failures that matter in this domain — missing rows, duplicates, late data, semantic drift — all produce green tasks.
"We have observability, we use a tracing platform." That platform answers why a request was slow. It has no opinion about whether a fact table is missing a day, and it should not be asked to have one.
"Data quality is a monitoring problem." Monitoring detects. Correctness is produced by the team that writes the field and by the contract at the boundary; a monitor placed downstream can only report (Who Owns Data Quality).
"More checks means more confidence." Beyond the point where alerts are muted, more checks means less confidence, because coverage now looks larger than it is (Alert Fatigue: The Page Nobody Reads).
"Freshness is a performance metric." It is a commitment about completeness over time. It has a latency budget inside it, which is exactly why it is the one signal that belongs to both disciplines (Pipeline SLOs).

Operating it

How you see it in production

Freshness per served dataset, exposed next to the data itself rather than in an engineering dashboard (Freshness Monitoring).
Row count per partition compared with the same weekday historically — the cheapest broad detector in the domain (Volume Anomalies).
Null rate, distinct count and category share per important column, trended rather than thresholded, because the step change is the signal (Distribution Tests).
Lineage edges from every serving table to its upstreams and its consumers, queryable during an incident (Data Lineage).
For the neighbour's half: bytes scanned, bytes shuffled, queue backlog and stage skew, which is where "slow" is diagnosed (The Shuffle, The Backlog Arithmetic: Four Levers and a Drain Time).

What changes at 10x and 100x

At 10x datasets, per-table hand-written checks stop being maintainable and the platform needs defaults applied automatically to every served dataset (Data Platform Engineering).
At 100x, check *cost* becomes a real budget line and checks need to be incremental — asserting over the new partition rather than the whole table.
Consumer count scales the cost of every incident, because scoping "who is affected" is proportional to the number of downstream dependencies rather than to the size of the error (Impact Analysis).

What drives cost here

Data checks cost query compute proportional to how much they scan, and a naive full-table assertion on every run can cost more than the model it guards (Scan Cost).
Metadata and lineage collection cost storage and pipeline overhead, and both are small relative to a single incident that could not be scoped.
The largest cost on this boundary is not compute at all — it is the human time an unscoped incident consumes, which is the thing observability is actually buying down (Data Incidents).

What this approach costs

Data observability is a second monitoring system with its own alerts, its own thresholds and its own maintenance. It buys detection before a human notices; it costs ongoing tuning that nobody enjoys.
Tight freshness SLOs make lateness visible and make every upstream hiccup a page. Loose ones keep the peace and let a stale dashboard sit for a day. The right answer comes from the decision the data drives, not from ambition (The Freshness SLO).
Automatic checks on every dataset buy coverage and generate noise. Selective checks buy signal and leave gaps. Both failure modes are real and the balance is a judgement about who acts on the alert (Who Owns Data Quality).

Where this applies

Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.

GENERALThat correctness and completeness are not observable from process health, and that freshness, volume, schema and distribution are the four dataset-level signals, holds for any platform regardless of tooling.
TOOL-SPECIFICSome transformation frameworks run assertions as part of the model run and fail the run on breach; dedicated observability tools profile tables on a schedule and detect anomalies without assertions being written. The first gives you a gate before publish, the second gives you coverage without authorship — and neither substitutes for the other.
ORG-SPECIFICWhether a failing data check pages anyone depends on whether a dataset has an owner. In organisations where datasets are unowned, adding checks measurably increases alert volume and does not improve response time, which is an ownership problem wearing a tooling costume (Who Owns Data Quality).

Where the depth lives

This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.

Backendbackend-metrics structured-logging

Observabilitywhat-observability-is golden-signals slo sli error-budgets burn-rate-alerts alerting alert-fatigue dashboard-design metric-types metric-labels cardinality counters distributed-tracing trace-anatomy incident-timeline incident-debugging symptom-to-signal queueing littles-law queue-backlog queue-age saturation slow-query-workflow flame-graphs deployment-markers measure-before-optimizing correlation-vs-causation averages-lie percentiles

Domains that do not exist yet

— DevOps / Production Engineering owns incident command — declaring, running and reviewing an incident. This domain contributes the part that is specific to data: scoping the affected *range* and the affected *consumers* before repairing, because a repair applied before scoping tends to create a second incident inside the first.