Data Quality
How do we know the data is correct enough to trust? Dimensions, tests, distribution checks, freshness and reconciliation — plus what every check still misses.
Every task green, every table populated, and the number still wrong. What "correct enough to trust" means, and why no single check establishes it.
Completeness, accuracy, freshness, uniqueness, validity, consistency — defined precisely, with how each is measured and which one is almost always asserted instead.
Assertions over rows and columns — not null, unique, non-negative, in a set, references valid — and the precise blind spot each one carries.
Every row is valid, every type is right, every key is unique — and today holds a small fraction of a normal day. The checks that compare data with its own history.
Expected latest data versus actual latest data — the cheapest check in the toolkit, two clocks that get confused, and the days it fires for no reason.
Count the rows and sum the measure at the source for a closed period, and compare with the serving table. The only check that observes both ends at once.
An alert nobody acts on trains people to ignore alerts. Severity from consumer impact, routing to the owner, and the difference between blocking a publish and sending a message.
One row per dataset — pipeline, freshness, completeness, status — and a hard rule that a green row is a statement about the checks you wrote, not about the data.
The team that produces a field owns whether it is correct. A data team can measure and report. Placing the whole obligation downstream guarantees it fails.