ObservabilityGENERALSCALE-SPECIFICORG-SPECIFIC

Volume Anomalies

Comparing today with the same weekday historically is the cheapest broad detector there is — and it misses every error that preserves row count, which is most value-level bugs.

What actually happens How to build it Can I trust it?

Who needs this, what one row is, and why the obvious build breaks

Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.

The question

Does the number of rows that arrived today look like the number that normally arrives on a day like today?

Who needs this

The on-call engineer, who wants one signal that fires on the widest possible range of ingestion and transformation faults for the least possible effort; and the analyst who would otherwise be the detector, noticing on Tuesday that Monday looked oddly quiet (Missing Rows).

What one row is

One row count per dataset per logical period, compared against the distribution of that dataset's counts for the same period-of-week historically. The comparison population is the design decision: comparing against yesterday detects almost nothing, and comparing against the mean of all days detects the weekend.

The obvious build

Assert that each daily load produces more than zero rows, or more than some fixed number an engineer picked when the pipeline was written. It is one line, it catches the total-outage case, and it is genuinely better than nothing.

Why it breaks

The threshold was set when the product was small. Volume tripled over two years, so the pipeline can now lose two thirds of a day and still pass (What Actually Drives Data Platform Cost).

How it breaks with real data

The threshold was set when the product was small. Volume tripled over two years, so the pipeline can now lose two thirds of a day and still pass (What Actually Drives Data Platform Cost).
Saturday has a quarter of Tuesday's volume, so a threshold that catches a bad Tuesday fires every Saturday, and a threshold that is quiet on Saturday cannot see a bad Tuesday.
A marketing campaign doubles genuine traffic. The anomaly detector reports an incident, an engineer spends an afternoon on it, and the alert loses a little more credibility (Correlation Is Not the Root Cause).
The check runs at 02:00 against a partition that is still filling, so it reports a shortfall every night and is duly moved to 06:00, where it now detects nothing until the morning (Atomic Publish).
A join fans out and the fact table gains rows. Volume goes *up*, which no lower-bound threshold examines, and revenue is overstated in perfect silence (Duplicate Rows).

Source→Ingestion→Raw→Transformation→Validation→Storage Model→Serving→Consumer→Observability

What is actually happening

Row count is a proxy for completeness that costs nothing to compute and correlates with a startling range of faults: partial loads, closed extract windows, filters that started matching, sources that stopped, casts that dropped rows, deduplication that started removing too much.
It works because most ways of breaking a pipeline change *how much* arrives. It fails for the same reason: any fault that changes values while leaving the count alone is invisible to it, and that class includes almost every business-logic error (Two Dashboards, Two Numbers).
The comparison population is where nearly all the engineering effort goes. Data volume has strong weekly seasonality, weaker monthly seasonality, a growth trend, and calendar effects. Comparing today with the same weekday over the last several weeks handles the first and third cheaply, which is why it is the default worth starting from.
A band beats a threshold. The useful statement is "today is outside the range this dataset's Tuesdays normally occupy", which requires a measure of spread. Using a median and an interquartile-style band rather than a mean and a standard deviation matters because the history contains the previous incidents, and a mean absorbs them (Percentiles: Which One, and How Many Users Is That?).
The check must be bounded on both sides. A lower bound catches loss; an upper bound catches duplication and fan-out, which are the failures that overstate a metric — and an overstated metric is the one people question least (Deduplication).

The cheapest broad detector there is

If a platform can afford exactly one check per dataset, it should be freshness. If it can afford two, the second is volume. Between them they detect the pipeline stopping, the source stopping, a partial load, a filter that started matching, a window that closed early and a join that fanned out — for the price of two aggregate queries.

What makes it work is the comparison, not the count. A count on its own means nothing; a count against the distribution of that dataset's own counts for the same day of week means a great deal, and the shift from a fixed number to a learned band is what turns a check that must be re-tuned every quarter into one that does not.

Weekly seasonality is the dominant structure in almost every business dataset, and handling it is nearly free. Trend is the second, and comparing against a recent window rather than all history handles that too. Monthly and annual effects — month-end, quarter-end, seasonal peaks — remain, and they are the reason a band should be wide enough to be trusted rather than tight enough to be clever.

Two ways to decide that today is unusual

A fixed threshold

Alert if the daily load produces fewer than fifty thousand rows. The number was chosen when the pipeline was written, lives in a config file, and is reviewed when it fires.

A band from the dataset's own history for the same weekday

Alert if the closed period's count falls outside the range this dataset's counts occupy for the same day of week over the preceding weeks, with incident periods excluded from that history, and with both a lower and an upper bound.

Data volume has weekly seasonality and a growth trend, so a fixed number is wrong in a different direction every quarter: it fires on Saturdays and it stops detecting real losses as the business grows. A relative band absorbs both without human tuning, and excluding known incidents stops the detector learning that outages are normal.

Today against this dataset's own Tuesdays

1-- Daily counts materialised once, so the baseline is cheap to re-read.
2with daily as (
3  select dataset, logical_period, rows_written
4  from   dataset_daily_counts
5  where  logical_period >= current_date - 60
6    and  logical_period not in (select period from known_incident_periods)
7),
8baseline as (
9  select
10    dataset,
11    extract(dow from logical_period) as dow,
12    percentile_cont(0.50) within group (order by rows_written) as p50,
13    percentile_cont(0.10) within group (order by rows_written) as p10,
14    percentile_cont(0.90) within group (order by rows_written) as p90
15  from   daily
16  where  logical_period < current_date - 1        -- exclude the day under test
17  group  by 1, 2
18)
19select
20  d.dataset,
21  d.logical_period,
22  d.rows_written,
23  b.p50 as typical_for_this_weekday,
24  case
25    when d.rows_written < b.p10 - (b.p90 - b.p10) then 'LOW  - possible loss'
26    when d.rows_written > b.p90 + (b.p90 - b.p10) then 'HIGH - possible duplication or fan-out'
27    else 'within band'
28  end as verdict
29from   daily d
30join   baseline b
31  on   b.dataset = d.dataset
32 and   b.dow     = extract(dow from d.logical_period)
33where  d.logical_period = current_date - 1;

Three decisions are doing the work: grouping the baseline by day of week, excluding known incident periods from the history, and bounding both directions. The band width shown is a starting shape to be widened per dataset from its own observed spread — it is not a recommendation to copy.

What volume cannot see

The blind spot is precise and worth stating exactly, because it is where the most damaging incidents live: any fault that changes values while preserving row count is invisible to this check. Since transformations mostly change values and rarely change counts, that covers the majority of business-logic errors.

The in-repo pipeline model makes the point cleanly. The fault that removes the refund subtraction from the revenue model leaves every row present, unique, well-typed and normally distributed; the row count is exactly right and the revenue is not. Only reconciliation against the source moves (Debugging a Data Incident).

The correct response is not to make the volume check cleverer. It is to accept that volume is a breadth signal and pair it with narrow, expensive checks on the small number of columns whose correctness anyone would actually notice — which is what the quality module is about (Data Quality).

Volume-family checks and their blind spots

Check	Expresses	Catches	Still misses
Row count for the closed period, banded against the same weekday	Today is a normal amount of data for a day like today.	Partial loads, stopped sources, closed extract windows, filters that started matching, fan-out joins when the upper bound is present.	Everything that preserves count. Also gradual loss, because the band moves with the decline (Missing Rows).
Rows out over rows in, per transformation	This step changes the number of rows by the factor it normally does.	Fan-out joins, deduplication that stopped working, a filter whose predicate changed meaning — and it localises to one step.	Any loss that happened before the first measured hop. The ratio is perfectly normal when the whole pipeline is fed half a day.
Distinct business keys per period	The number of real-world entities is normal, independently of how many rows represent them.	Duplication that inflates rows without adding entities, and loss of a whole segment even when total rows are backfilled by another segment growing.	Duplicates that arrive under new keys — a producer retry with a fresh event id is a new entity as far as this check can tell (Deduplication).
Per-category counts against their historical shares	The mix is normal, not just the total.	One country, tenant or channel disappearing while the total is held up by growth elsewhere — invisible to a total-count check.	A change that moves every category proportionally, and it grows expensive with cardinality (Cardinality: The Label That Took Down Monitoring).
Source-versus-serving count for a closed period	Everything the source recorded arrived.	Loss anywhere along the whole journey, in one query, without knowing where the pipeline is.	Duplicates that coincidentally offset losses, anything wrong identically at both ends, and any period that is not yet closed (Reconciliation).

The first four are cheap and internal; the last one is the only one that observes the ends of the journey against each other, and it is the one that requires access to the source. That access is usually the reason it does not exist.

Keeping the check alive for more than one quarter

SCALE-SPECIFICThese degradation paths assume enough datasets and enough alerts that individual tuning happens under time pressure. On a platform with five datasets an engineer holds the whole picture and can afford hand-tuned thresholds; the failure described here begins somewhere above the point where nobody remembers every check.

Volume checks die of noise. Each individual false positive produces a locally reasonable response — widen the band, move the schedule later, mute for the weekend — and the sequence ends with a check that is still listed on the coverage report and can no longer fire.

The failure rows below are the specific mechanisms, in roughly the order they arrive. Notice that only one of them is about the detector being wrong; the rest are about the detector being right and the *incident framing* being wrong, which is a communication problem wearing a monitoring problem's clothes.

The metric to watch is the ratio of alerts to confirmed incidents for this check. When it drifts past the point where an engineer expects the alert to be nothing, the check has stopped working, whatever its detection rate looks like on paper (Alert Fatigue: The Page Nobody Reads).

How a volume check degrades in its first year

Trigger	Symptom	Cause	Response
The check runs while the partition is still filling.	A shortfall alert every night at 02:00.	The check is scheduled on wall-clock time rather than on period close, so it measures an incomplete period.	Trigger the check on the publish event for the period, not on a clock. If that is impossible, add a close margin and measure the previous period instead of the current one (Atomic Publish).
A marketing campaign doubles genuine traffic.	A high-side alert, an afternoon of investigation, no fault found.	The detector has no channel through which business events reach it.	Annotate known events on the same timeline as the counts, so the alert arrives with its explanation attached. The alert is still correct; it just becomes cheap to close ("What Changed?" — Deploy Markers and the Invisible Deploys).
A model changes grain from one row per order to one row per line.	Every volume alert for the dataset fires, for weeks.	The baseline describes the old pipeline. The new counts are correct and incomparable.	Reset the baseline deliberately at the change, and record the model version with the count so the discontinuity is attributable rather than mysterious (Grain: What Does One Row Represent?).
A source sheds a small share of traffic every week.	No alert ever fires; six months later the totals are far below where they should be.	The baseline is a rolling window, so it tracks the decline.	Add a long-horizon comparison — this month against the same month last year — accepting that it is slow and noisy, and pair it with source-side reconciliation which does not use your own history as its reference (Reconciliation).
A previous outage is left in the baseline history.	The band is wide enough to swallow a repeat of the same outage.	Learned baselines learn whatever they are given, incidents included.	Maintain an explicit list of excluded periods and populate it from the incident record. It is the least glamorous line of code in the system and it preserves the detector's sensitivity (Data Incidents).

How to build it

Most important first.

Compare against the same weekday, over several weeks, using a median and a spread rather than a fixed number. This one change removes the two most common false positives — weekends and growth — without any tuning (Distribution Tests).
Bound both directions. The upper bound is the cheaper half to add and detects the class of fault that inflates numbers, which is the class nobody reports (Duplicate Rows).
Run the check only against closed periods. A partition still being written to is genuinely short, and a check that cannot tell those apart will be moved later and later until it detects nothing.
Check volume at every hop, not only at the end. A count at the raw layer, the staging layer and the serving layer localises a loss to one arrow instead of leaving a search across six systems (Debugging a Data Incident).
Annotate known events — campaigns, launches, migrations, backfills — so that a genuine business change is a recorded explanation rather than an unexplained alert. Borrow the pattern from deployment annotation ("What Changed?" — Deploy Markers and the Invisible Deploys).
Exclude known-bad history from the baseline. A month that included an outage teaches the detector that outages are normal, and the effect is permanent unless the period is excluded explicitly.

What this actually promises

Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.

A passing volume check guarantees the row count is within the band the dataset's recent history occupies for this period-of-week. Nothing more.
It does not guarantee the rows are the right rows: a load that dropped every European order and duplicated every American one can land squarely in the band.
It does not guarantee the period is complete, only that it is *typical*. A source that has been quietly under-delivering for a month establishes a new normal, and the check will defend it (Reconciliation).
It guarantees nothing whatsoever about values, types or meaning. Volume is a count, and correctness is not a count.

Can I trust it?

A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.

The check that would catch this

The check is itself the quality control: row count for the closed period against the median for the same weekday over the preceding weeks, with a band on both sides, evaluated per dataset and per hop.
It misses every fault that preserves count — a wrong join key that swaps values between rows, a currency that changed, a filter that replaced one category with another, a cast that nulled a column. That is the majority of value-level bugs, and it is why volume is a *breadth* signal rather than a correctness one.
It also misses gradual loss. A source shedding a small share of traffic each week never leaves the band, because the band moves with it (Distribution Tests).

Freshness

The check can only run once the period is closed, so its detection latency is at least one period plus whatever margin the close policy adds. For a daily dataset that is a day, which is why it complements rather than replaces freshness monitoring (Freshness Monitoring).
Running an intra-period version — "by 14:00 we normally have this many rows" — detects a stopped source within hours instead of the next morning, at the cost of a second baseline that has to be learned per hour of day.
A late-arriving batch makes a period's count non-final. Re-evaluating volume after allowed lateness expires produces a different and more truthful answer, and a check that only runs once will always evaluate the optimistic version (Late-Arriving Data).

When the schema or meaning changes

A schema change that adds a column does not change row count and is invisible here — which is the correct behaviour and worth knowing, because people expect volume checks to catch more than they do (Schema Evolution).
A change of grain does change row count, dramatically and legitimately. Moving a fact table from one row per order to one row per order line will fire every volume alert you have, and the response is to reset the baseline deliberately rather than to widen the band (Grain: What Does One Row Represent?).
When a pipeline changes its deduplication or filtering logic, the historical baseline stops describing the current pipeline. Record the model version alongside the count so a discontinuity can be attributed instead of investigated.

How to re-run this safely

After a confirmed loss, the repair is a bounded re-run of the affected periods, followed by a re-evaluation of the volume check for those periods — the second half is the part that gets skipped and is the only evidence the repair worked (Validating a Backfill Before You Publish).
Exclude the incident window from future baselines explicitly. Leaving it in teaches the detector to expect that shape, which quietly reduces sensitivity for months (What Backfills Break).
If the loss is unrecoverable — retention expired at the source — mark the affected periods in the dataset's documentation rather than leaving a hole that a future analyst will interpret as a business trend (Dataset Documentation).

What can go wrong

Failure modes

A fixed threshold set years ago, now so far below current volume that it cannot detect a catastrophic loss.
A baseline that includes previous incidents, so the detector has learned that outages are within normal range.
A lower bound with no upper bound, which is blind to every duplication and fan-out fault.
The check running against a period that is still filling, which produces a nightly false positive and ends with the check being scheduled so late that it is useless.
Alerting on a genuine business change with such confidence that people start ignoring the check — the detector is right about the data and wrong about the incident (Correlation Is Not the Root Cause).

Misreads

"Volume is normal, so the data is fine." Volume is a count. Every value in every row could be wrong and the count would not move (The Pipeline Succeeded. The Data Is Wrong.).
"The count is higher than usual, which is good." Duplication and fan-out both raise counts. An unexplained increase deserves the same investigation as an unexplained decrease and reliably receives less (Duplicate Rows).
"We compare with yesterday, which is simpler." Yesterday was a different weekday for a fifth of the week, and a gradual loss is invisible to a one-day comparison because each day resembles its predecessor.
"Anomaly detection will learn the right band." It will learn whatever the history contains, including the incidents. A learned baseline over unfiltered history is a detector trained to tolerate the failures you have already had.

Operating it

How you see it in production

Rows per dataset per logical period, plotted against the band derived from the same weekday historically. One chart that a human can read in a second.
Row counts at every hop on one axis, so a drop between two adjacent hops localises the fault immediately (Debugging a Data Incident).
Rows out over rows in per transformation, which detects fan-out that absolute counts hide (Pipeline Metrics).
Alert-to-incident ratio for this check specifically. It is the number that predicts whether the check will still be enabled next year.

What changes at 10x and 100x

At ten times the datasets, per-dataset thresholds must be derived rather than configured. A baseline computed from each dataset's own history scales; a number in a config file does not.
At a hundred times, the alert volume from a naive per-dataset check exceeds what anyone can read. Grouping by lineage root and reporting one incident per cause is the only thing that keeps it usable (Impact Analysis).
Finer periods reduce detection latency and increase noise, because per-hour counts are far more variable than per-day counts. Below a certain volume the hourly signal is mostly variance and the daily one is the only one that means anything.

What drives cost here

A count on a partitioned table reads partition metadata rather than data in most analytical stores, which makes this the cheapest broad check available. The cost appears when the count carries a predicate that prevents pruning (Partition Pruning).
The baseline query is the expensive half: several weeks of history per dataset, recomputed on every evaluation. Materialising daily counts once and reading from that table removes almost all of it (Incremental Processing).
Per-hop counts multiply the check count by the number of stages. That is usually worth it — the localisation it buys is measured in hours of an incident — but it should be a deliberate choice rather than a default applied to every dataset.

What this approach costs

Volume is the highest breadth-per-effort check in this domain and it buys no depth at all. Treating it as the quality strategy rather than as the first tripwire is the mistake it invites.
A tighter band detects smaller losses and fires more often on genuine business variation. The right width is the one where the check is still trusted after a season of real traffic, which is wider than it feels on the day you write it.
Per-hop checks localise faults fast and multiply the number of alerts a single incident produces. Without lineage-based grouping, better localisation makes the channel worse.

Where this applies

Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.

GENERALCounting rows per period and comparing against the same period-of-week is arithmetic, available in any store that answers SQL. What changes between stores is whether the count is served from metadata or requires reading data.
SCALE-SPECIFICBelow a few hundred rows a period, counts are dominated by variance and the band is so wide it detects only total outages; above a few million the signal is stable enough that a tight band is both safe and sensitive. The same check is nearly useless at one end and excellent at the other.
ORG-SPECIFICThe dominant false positive is genuine business change, which means the check's usefulness depends on whether marketing and product tell the data team about launches. Where they do not, the band has to be widened until the check detects much less.

Where the depth lives

This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.

Databaseaggregation-and-grouping window-functions

Observabilityalerting alert-fatigue percentiles averages-lie correlation-vs-causation deployment-markers cardinality

DSAsliding-window-fixed

Domains that do not exist yet

— DevOps / Production Engineering owns release annotation and change tracking. Most unexplained volume shifts have a deployment behind them, and correlating the two is the single fastest way to close a volume alert.