ObservabilityGENERALSOURCE-SPECIFICORG-SPECIFIC

Freshness Monitoring

Freshness is a per-dataset property. Averaging it across a platform hides the one table that has not updated since Friday — and the false-positive rate decides whether anyone still reads the alert in six months.

What actually happens How to build it Can I trust it?

Who needs this, what one row is, and why the obvious build breaks

Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.

The question

How old is the newest complete record in this dataset, and how old is it allowed to get before someone should be told?

Who needs this

The analyst deciding whether today's figure is final, the finance close that cannot begin until yesterday is complete, and the retrieval index whose answers age with its corpus. All three want one number per dataset that they can read without asking an engineer (Who Actually Consumes This Data).

What one row is

One freshness observation per dataset: a single age, measured now, for one table or one logical partition. It is deliberately not per-platform and deliberately not per-pipeline — a pipeline can be perfectly healthy while the table it feeds is stale because the source stopped producing.

The obvious build

Put one "data is fresh" tile on the platform dashboard, computed as the average age across all datasets, and go green when it is under an hour. It renders beautifully and everybody understands it immediately.

Why it breaks

Four hundred tables refresh hourly and one has not updated since Friday. The mean age barely moves, the tile stays green, and the stale table is the one finance uses (Stale Dashboards).

How it breaks with real data

Four hundred tables refresh hourly and one has not updated since Friday. The mean age barely moves, the tile stays green, and the stale table is the one finance uses (Stale Dashboards).
Freshness is computed as "time since the last successful run", so a pipeline that runs every hour and writes nothing every hour reports perfect freshness forever (The Pipeline Succeeded. The Data Is Wrong.).
Freshness is computed from max(ingested_at), which is set by the pipeline. It therefore measures how recently the pipeline ran and not how recently the world changed — the number is fresh by construction (Ingestion Time).
The source genuinely produced nothing overnight — a closed market, a holiday, a batch partner that skips weekends — and the alert fires. It fires every weekend, and by the second month it is muted (Alert Fatigue: The Page Nobody Reads).
A late-arriving batch fills in yesterday, so max(event_time) jumps backwards relative to what was reported an hour ago. Nobody expected freshness to be non-monotonic and the alert logic assumed it was (Late-Arriving Data).

Source→Ingestion→Raw→Transformation→Validation→Storage Model→Serving→Consumer→Observability

What is actually happening

Freshness is a difference between two clocks, and there are four different pairs people mean by it. The newest event time in the table against now; the newest arrival time against now; the time of the last successful run; and the table's last physical modification. They answer different questions and diverge exactly when something is wrong (Event Time, Processing Time).
The only one that measures what a consumer cares about is event-time freshness: how recently did the world, as recorded here, happen. The other three measure the pipeline, and a pipeline can be busy and useless.
The word "complete" in "newest complete record" is load-bearing. A partition that is still filling contains recent records and is not yet a truthful answer to any question about that period, so freshness should be measured against the newest *closed* period rather than the newest row (Windows).
Freshness is not monotonic when late data is allowed. Backfills and allowed lateness both insert older records, and a check that assumes the maximum only ever increases will produce confusing results at exactly the moment things get interesting (Watermarks).
The threshold is where the engineering happens. A threshold set from the pipeline's schedule produces an alert for every normal delay; a threshold set from the *consumer's deadline* produces alerts that mean something, and that difference is the whole design (The Freshness SLO).

Four things people mean by freshness

Ask three engineers on the same team how fresh a table is and you can get three different numbers, all correct, all measuring something different. The divergence is not pedantry: the four definitions agree when everything is healthy and separate precisely when something is wrong, which makes their disagreement one of the most useful diagnostic signals available.

The one a consumer means is almost always the first row: how recently did the thing being recorded actually happen. The one that is easiest to implement is the third, because the orchestrator already has it, and a great many platforms ship the easy one under the name of the useful one.

The practical recommendation is to compute at least two — event-time and arrival-time — and alert on their difference as well as on their absolute values. When arrival-time freshness is fine and event-time freshness is degrading, the pipeline is happily processing a source that has stopped producing, which is a class of incident that no single number reveals.

Definition	Computed as	Right when	Fails by
Event-time freshness	`now() - max(event_time)` over the newest closed period.	The consumer's question is about the world: how recent is the newest thing that happened.	Depending on a source timestamp that may be assigned before commit, or on a column that a broken cast has quietly nulled.
Arrival-time freshness	`now() - max(ingested_at)`, where `ingested_at` is stamped on receipt.	You need to know whether the ingestion path is moving, independently of whether the source is producing.	Being mistaken for the definition above. It is fresh whenever your pipeline is busy, including when it is busy processing nothing of value.
Run recency	`now() - last_successful_run_end`.	A quick sanity signal for a pipeline you do not otherwise instrument.	Reporting perfect freshness for a job that runs on schedule and writes zero rows every time (The Pipeline Succeeded. The Data Is Wrong.).
Physical modification time	Table or partition metadata from the store.	A near-free approximation for low-tier datasets where a query is not worth its cost.	Changing when nothing meaningful happened — a compaction, a re-clustering or a no-op merge all update it (File Compaction).

Per-dataset freshness against a per-dataset target

1-- One row per dataset. Never averaged: the roll-up is the count of
2-- datasets in breach, so one broken table cannot hide behind 399 good ones.
3with observed as (
4  select 'fct_orders'      as dataset,
5         max(order_ts)     as newest_event,
6         max(ingested_at)  as newest_arrival
7  from   fct_orders
8  where  order_ts >= current_date - 3       -- bound the scan, not the answer
9  union all
10  select 'dim_customer', max(valid_from), max(ingested_at)
11  from   dim_customer
12  where  valid_from >= current_date - 30
13)
14select
15  o.dataset,
16  t.target_minutes,
17  extract(epoch from (now() - o.newest_event))   / 60 as event_age_min,
18  extract(epoch from (now() - o.newest_arrival)) / 60 as arrival_age_min,
19  case
20    when not c.source_expected_now              then 'expected quiet'
21    when now() - o.newest_event
22         > make_interval(mins => t.target_minutes) then 'BREACH'
23    else 'ok'
24  end as state
25from observed o
26join dataset_targets  t on t.dataset = o.dataset
27join production_calendar c on c.dataset = o.dataset
28                          and c.as_of   = date_trunc('hour', now());

Two details carry the lesson. The join to production_calendar is what stops the check firing on a weekend the source never produces on. The two ages are kept separate so that a growing gap between them can be alerted on in its own right.

Event time, arrival time, and the period that is not finished yet

SIMPLIFIEDClock labels on a teaching timeline, with one late event standing in for a distribution of lateness. Real sources have a long tail, so "complete" is a confidence statement rather than a boolean, and the allowed-lateness setting is where you choose the confidence level (Late Events).

A freshness number computed against the newest *row* rather than the newest *complete period* is optimistic in exactly the situation where optimism is most expensive. A partition that is halfway through filling contains very recent records and is not a truthful answer to any question about that period.

The timeline below is a teaching clock, not a measurement. Four events happen inside the 09:00–10:00 hour; three arrive promptly and one arrives at 12:30. At 10:05 a naive check sees a two-minute-old record and reports excellent freshness, while the hour it belongs to is missing a quarter of its rows and will stay missing until the late arrival is folded in (Late-Arriving Data).

This is why the honest metric is the age of the newest period you are willing to call complete, and why the definition of complete belongs to the allowed-lateness policy rather than to the monitor. Once lateness is allowed, freshness genuinely gets worse by design, and that reduction is bought for completeness — a trade worth stating to consumers rather than absorbing silently (Watermarks).

Why "newest row" and "newest complete period" are different numbers

Hour 09 09:00–10:00Hour 10 10:00–11:00Hour 11 11:00–12:00watermark At 12:30 the watermark can only safely be placed at 09:00 if lateness of up to about three hours is allowed; without that, Hour 09 closed at 10:00 and `a3` has nowhere to go.

Event	Happened	Arrived	Lands in
a1	09:12	09:14	Hour 09 Normal path: arrives inside its own hour and is counted there.
a2	09:47	09:49	Hour 09 Also normal. At 10:05 this is the newest record in Hour 09.
a3	09:58	12:30	Hour 09 only if lateness is allowed The one that decides whether Hour 09 was ever complete. Without allowed lateness it is counted nowhere and the hour just looks quiet.
b1	10:03	10:05	Hour 10 At 10:05 a naive freshness check sees this two-minute-old row and reports the dataset as fresh — while Hour 09 is still incomplete.
b2	10:55	10:57	Hour 10

Freshness measured on the newest row is a property of arrivals. Freshness measured on the newest complete period is a property of answers, and only the second one is what a consumer is actually asking about.

The threshold, and the alert nobody reads

A freshness check has exactly one enemy and it is not the stale table. It is the false positive. Every alert that fires without an incident behind it spends a little of the attention the check depends on, and the balance is not replenished — six months of weekend alerts produce a muted channel, and the mute is permanent long after the underlying cause is fixed.

That makes the false-positive rate a design parameter with the same standing as the detection threshold. A check that detects every real stall and fires twice a week for nothing is strictly worse than a slightly looser check that fires only when something is wrong, because the second one will still be read next year.

The dominant source of false positives is not threshold tuning. It is periods where the source genuinely produced nothing: weekends, holidays, market closures, a partner who batches on Tuesdays. Modelling that calendar is more work than widening a threshold and it is the only approach that keeps both the sensitivity and the trust.

Where does the threshold come from?

This dataset should alert when it is how old — and derived from what?

The consumer's deadline, minus repair time

when A consumer has committed to a time they need the data by, and a re-run has a known duration.

cost Requires someone to state a deadline and someone to measure repair time. It is the only derivation that makes the alert mean something, and the hardest to get agreement on (The Freshness SLO).

A multiple of the schedule interval

when No consumer deadline exists yet and something is better than nothing.

cost Alerts about the pipeline rather than about the data, and silently loosens whenever the schedule gets finer. Treat it as a placeholder with an expiry.

An expected-arrival calendar

when The source produces on a known rhythm — weekdays, market hours, a partner's batch window.

cost A second artefact to maintain that will be wrong on the first unusual holiday. It removes the dominant false positive, which is usually worth it (Quality Alerting).

A learned band from the dataset's own history

when Arrival timing is variable but stable in distribution, and there are enough datasets that hand-setting thresholds does not scale.

cost Learns the outage it was trained through, and drifts quietly toward whatever the platform currently does rather than what it should do.

No alert, published number only

when A low-tier dataset where consumers can see the age next to the number and decide for themselves.

cost Relies on consumers looking. It is the honest option for most of a large platform and is rarely chosen because it feels like giving up.

How freshness alerting degrades, and what to do instead

Trigger	Symptom	Cause	Response
The source produces nothing over a weekend.	The freshness alert fires every Saturday morning.	The check has no model of when the source is expected to produce, so absence of data is indistinguishable from absence of delivery.	Add the production calendar and suppress by expectation rather than by widening the threshold. If the calendar is unknowable, alert on the gap between event-time and arrival-time freshness instead, which is quiet when the source is genuinely idle.
A backfill inserts older records.	Freshness appears to move backwards; alerts fire on a dataset that just got better.	The check assumes `max(event_time)` is monotonic, which is false whenever history can be amended.	Measure against the newest complete period rather than the newest row, and mark backfill windows so the check can ignore them (Planning a Backfill).
A cast nulls the timestamp column.	Freshness ages smoothly and slowly, looking like a gradual slowdown rather than a break.	`max()` over a column that is now all null returns the last good value, so the metric decays instead of failing.	Pair every freshness check with a null-rate check on the same column. The pair distinguishes "getting late" from "the clock broke" (Nullability & Defaults).
An engineer suppresses the alert during an incident.	Months later the dataset is stale and nothing fires.	Suppressions have no expiry, and the one person who knew about it has moved teams.	Make every suppression expire by default and surface active suppressions on the coverage report, so an unmonitored dataset is visible rather than invisible.
The monitor reads the pipeline's own status table.	During a real outage, no freshness alert fires at all.	The monitor and the monitored share a dependency, so they fail together.	Compute freshness by querying the dataset itself, from a scheduler that is not the pipeline's scheduler (Health Checks).

How to build it

Most important first.

Measure freshness per dataset, publish it per dataset, and never publish a platform average. If a roll-up is needed, use the count of datasets currently breaching their own target — a number that goes up when one table breaks.
Define it on event time where a business timestamp exists, and say in the dataset's documentation which column it is measured on. A freshness number whose definition is undocumented is a number two people will read two ways (Dataset Documentation).
Derive the threshold from the consumer's deadline minus the time needed to fix a problem, not from the schedule. "Finance needs yesterday by 08:00 and a re-run takes ninety minutes" gives a real threshold; "it runs hourly so alert at two hours" gives an alert about the pipeline.
Model the expected-empty case explicitly rather than widening the threshold. A calendar of expected production periods — weekdays only, market hours only — converts the most common false positive into a correct silence (Quality Alerting).
Track the freshness of what was *published*, not of what was computed. A dataset built successfully but not yet swapped in is not fresh from a consumer's point of view (Atomic Publish).
Show freshness where the data is consumed. A number on a monitoring dashboard is read by engineers; a "last updated" stamp next to the dashboard tile is read by the person about to make a decision (The Data Quality Dashboard).

What this actually promises

Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.

A passing freshness check guarantees a recent record exists in the dataset. It does not guarantee the period is complete, that older records were not dropped, or that any value is correct.
Event-time freshness guarantees a bound on how far behind the *world* the table is, assuming the source's timestamps are trustworthy — which for a system that assigns timestamps before commit, they are not entirely (Incremental Extraction).
Run-based freshness guarantees only that a process finished. It is the weakest of the four definitions and the one most often implemented, because it is the one the orchestrator hands you for free.
No freshness definition can distinguish "the source produced nothing" from "we failed to receive what the source produced". That distinction requires a second signal from the source itself, and its absence is the root of the false-positive problem (Reconciliation).

Can I trust it?

A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.

The check that would catch this

The check: per dataset, assert now() - max(event_time) is under the dataset's declared target, evaluated only during periods the calendar says the source is expected to produce. It catches stopped pipelines, stopped sources, removed tasks and unswapped publishes in one query.
It misses data that is fresh and wrong, which is most of what goes wrong. It also misses a partial period: a table containing one recent row and missing ninety percent of the hour passes freshness and fails completeness (Volume Anomalies).
And it misses the case where the timestamp column itself is broken. A cast that nulled event_time makes max() return the last good value forever, so the check reports a slowly ageing dataset rather than a broken one — pair it with a null-rate check on the same column (Distribution Tests).

Freshness

This lesson is about the measurement, and the measurement has a latency of its own: a check that runs every hour cannot detect a breach faster than an hour. For a dataset with a hard morning deadline, the check interval must be finer than the time needed to act.
Detection is only half the number a consumer experiences. Time-to-notice plus time-to-fix is the real staleness, and the first term is entirely under the monitoring layer's control.
Freshness bounds every downstream dataset: a model can never be fresher than its stalest input, and a mart is always at least one interval behind the model it derives from. Publishing per-dataset freshness makes that chain visible instead of surprising (Data Lineage).

When the schema or meaning changes

The column freshness is measured on is part of the dataset's contract. Renaming or retyping it breaks the monitor silently, and the monitor breaking silently is worse than the dataset breaking loudly (Data Contracts).
When a pipeline moves from daily to hourly, thresholds derived from the schedule become wrong in the loose direction and nobody notices, because looser thresholds do not fire. Re-derive from the consumer deadline instead, which does not move when the schedule does.
Adding allowed lateness changes freshness semantics: the newest complete period now closes later by design. That is a deliberate freshness reduction bought for completeness, and it needs to be re-declared to consumers rather than absorbed quietly (Late Events).

How to re-run this safely

After a stall, the recovery is a catch-up run over the missed periods. It should advance the watermark period by period rather than in one jump, so freshness recovers observably and a second failure is localised (The High-Water Mark).
Suppress the freshness alert for the affected dataset during a known catch-up, with an expiry. An indefinite suppression is how a dataset stops being monitored forever, and it is always created during an incident by someone who means to remove it.
Record the breach as data — dataset, period, duration, cause — so that the freshness SLO can be reported on rather than felt. A platform that cannot say how often it was late cannot argue for the investment to be less late (Pipeline SLOs).

What can go wrong

Failure modes

Freshness measured on a column the pipeline itself writes, which makes the number fresh whenever the pipeline runs regardless of what it produced.
A platform-level average that hides every individual stall — the failure this lesson exists to name.
A threshold widened after each false positive until it can no longer fire, with each individual widening entirely reasonable.
An alert that fires every weekend because the calendar of expected production was never modelled, and is therefore muted before it ever catches anything real (Alert Fatigue: The Page Nobody Reads).
A suppression created during an incident and never removed, so the dataset is silently unmonitored while appearing on the coverage report.
The monitor reading the same metadata table the pipeline writes, so both go quiet together.

Misreads

"Freshness is a platform property." It is a per-dataset property. Every average over datasets hides exactly the dataset that is broken, and the more datasets you have the better it hides it.
"The pipeline ran, so the data is fresh." The pipeline running is a statement about the pipeline. A run that wrote nothing leaves the table exactly as stale as it was (Pipeline Observability).
"We should alert whenever data is older than the schedule." That produces an alert for every normal variation. The threshold belongs to the consumer's deadline, not to the producer's cadence.
"A muted alert is a monitoring gap we will fix later." A muted alert is a decision that has already been taken. The fix is to reduce the false-positive rate until the alert deserves attention, not to un-mute it and hope (Quality Alerting).

Operating it

How you see it in production

Age of the newest complete record, per dataset, with the dataset's target drawn on the same axis. One chart per dataset, and a count of breaching datasets as the only roll-up.
Time-to-detect and time-to-recover per breach, retained as history. These two numbers are the argument for every subsequent investment in this area (Reading a Timeline: Observation Order Is Not Causal Order).
Divergence between event-time freshness and arrival-time freshness. When the two separate, the source is producing and you are not receiving — which is a different incident from "nothing is happening" (CDC Failure Modes and the Retention Deadline).
Alert-to-incident ratio per check. A freshness check that fired forty times and produced one incident is a check that is about to be muted.

What changes at 10x and 100x

At ten times the datasets, freshness definitions must come from the catalog automatically rather than being configured per table, or coverage becomes a function of who remembered.
At a hundred times, the breach list becomes the interface and individual charts stop being read. Grouping breaches by root cause through lineage is what keeps it actionable (Impact Analysis).
Finer schedules multiply check volume linearly and shrink the acceptable detection interval at the same time, so the monitoring cost of an hourly platform is more than twenty-four times that of a daily one.

What drives cost here

A freshness query is a max() on one column. It is close to free when that column is clustered or partitioned on, and a full scan when it is not — which makes the physical layout of the table a monitoring cost driver as well as a query one (Clustering and Sort Order).
Cost scales with datasets times check frequency. Checking four hundred tables every five minutes is a real workload; matching each check's interval to its dataset's cadence removes most of it for nothing.
Where the store exposes table metadata — last modification time, row counts — using it instead of a query trades precision for a near-zero cost, and is usually the right trade for low-tier datasets (Metadata: Technical, Operational and Business).

What this approach costs

Measuring on event time is the honest definition and requires a trustworthy business timestamp, which not every source provides. Falling back to arrival time is defensible if you say so in the documentation; it is dishonest if you call it freshness and let people assume.
A calendar of expected-empty periods removes the dominant false positive and is a second thing to maintain that will itself be wrong on the first unusual holiday.
Tight thresholds detect faster and page more. The right setting is the one where the alert still gets read in six months, which is usually looser than the one an engineer picks on the day they build it.

Where this applies

Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.

GENERALThe four definitions of freshness and the argument for measuring per dataset hold for any store that can answer a max() on a timestamp. What changes is whether the store exposes cheap table metadata that makes an approximate answer nearly free.
SOURCE-SPECIFICEvent-time freshness is only as good as the source's timestamp: a system that assigns created_at in application code before commit produces timestamps that can be older than the commit that carries them, while a change log carries the commit position itself and is exact (CDC vs Polling).
ORG-SPECIFICThresholds derived from consumer deadlines require someone to state a deadline. In organisations where no consumer will commit to one, freshness targets end up derived from the schedule and the alerting reverts to being about the pipeline.

Where the depth lives

This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.

Databasereplication

Backendhealth-checks scheduled-jobs

Observabilityalerting alert-fatigue slo sli error-budgets burn-rate-alerts incident-timeline

Domains that do not exist yet

— Distributed Systems owns why "now" is not a single value across machines, and therefore why a freshness number computed on one host can disagree with the same number computed on another.
— DevOps / Production Engineering owns the on-call rotation and escalation policy that a freshness page enters. The threshold decision here is only useful if it lands somewhere with an owner.