Data Quality Lab
Faults are injected; you write the checks. The exercise is not passing them — it is finding out what each check still misses, because a check whose blind spot nobody stated teaches false confidence, which is worse than no check at all.
The rows, the faults and the check outcomes come from the pipeline model in this repository. No figure here was measured, and no figure here is a threshold you should copy into a real platform: what a distribution check should tolerate is a property of your data, and anybody who hands you a number for it has not looked at it.
Quality is a portfolio, not a test suite. Every individual check is blind to something: a uniqueness check cannot see missing rows, a completeness check cannot see duplicates, a schema check cannot see a field whose meaning changed while its type did not, and none of them can see a transformation that is doing exactly what it was told when what it was told is wrong.
So the useful question is never “did the checks pass”. It is “which of the things that can go wrong would any of these checks have noticed” — and that question has a written answer only if somebody recorded each check's blind spot when they wrote it.
Data Quality Lab
Inject nulls, duplicates, a missing day and a volume drop, then choose the checks. Then look at what got through.
Turn on a fault, then turn on checks one at a time and watch which ones stay green. A fault that survives every check you can think of is the most instructive outcome available here.
The revenue model stops subtracting refunds.
| Check | Running? | Would it have fired? | Blind spot it keeps anyway |
|---|---|---|---|
Completeness Every order the source recorded for the period reached the serving table. | running | pass Every order in the source for this period is present. | Duplicates that coincidentally offset losses, and any period that is not yet closed. |
Uniqueness Each order id appears exactly once in the serving table. | running | pass Every order id appears exactly once. | A genuine duplicate that arrived under a new key — a producer retry with a fresh event id looks like a second order. |
Freshness The newest complete record is recent enough for the decisions this table drives. | running | pass Newest complete record is 12 simulated minutes old. | Data that is perfectly fresh and completely wrong. It also fires falsely on a period where the source genuinely produced nothing. |
Validity Every amount is non-null and parses as a number. | not run | pass Every amount is non-null and numeric. | A value that is well-typed and wrong — a price in the wrong currency passes every type check there is. |
Distribution The shape of the day resembles the days before it, per country and in total. | not run | pass Largest per-country share drift 0.8pp; total volume drift 0.0%. | Slow drift, and any error that preserves the shape while changing every value inside it. |
Reconciliation Revenue summed in the serving table equals revenue summed in the source for the same closed period. | not run | FAIL Serving table reports 1,010,654.50 against a source total of 934,498.90. | Anything wrong identically at both ends — a bug in logic shared by the extract and the model reconciles perfectly. |
- —Completeness — are all the rows that should exist here present? Absence produces no signal unless something counts.
- —Uniqueness — is the business key unique, as opposed to the row being unique? An id-based check passes on a producer retry.
- —Validity — do values conform to their type and domain? A failed cast that produces null passes every row-count check there is.
- —Freshness — is the newest data as new as the SLO says? Per dataset, because a platform average hides the table that stopped on Friday.
- —Distribution — has the shape moved? The only family that can catch a semantic change, and the hardest to set a threshold for.
- —Reconciliation — does an independent system agree? The only check that can catch a transformation that is confidently wrong.
- —A check written against the model that produced the data, by the person who produced it, testing the assumption that was already made.
- —A change that is correct in every dimension and wrong in meaning — the same column, now populated by a different upstream team.
- —Anything in a dataset nobody owns, because the check that is not maintained is deleted the third time it is noisy.
- —The period before the check existed, which is most of the history somebody is about to query.