ModelingGENERALENGINE-SPECIFICSOURCE-SPECIFIC

SCD Type 2 in Practice

valid_from, valid_to, is_current — the columns that preserve history, the join predicate every fact must use, and the interval bugs that produce numbers reconciling against nothing.

Who needs this, what one row is, and why the obvious build breaks

Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.

The question

How do you store every version of a dimension attribute, and how does a fact pick the one version that was true when it happened?

Who needs this

Finance re-running a closed month and expecting the same answer. An auditor comparing a filed figure to a live query. A commission calculation for a quarter that ended. All three need a number that does not move, and Type 2 is the mechanism that makes that possible — or, implemented wrongly, the mechanism that makes it confidently wrong.

What one row is

One row is one version of one entity over one time interval: (customer_id, valid_from). Not one customer. Every test, every count and every join in the model has to be re-read with that sentence in mind, and the number of bugs caused by forgetting it is the reason this lesson exists (Grain: What Does One Row Represent?).

The obvious build

Add valid_from, valid_to and is_current, close the old row and insert a new one whenever the source value differs. It is a straightforward merge, it obviously preserves history, and the first version of it usually works on the first day.

Why it breaks

The merge runs twice — a retry, a manual re-run, a backfill overlapping a scheduled run — and inserts a second identical version. Two rows now claim is_current = TRUE, and every fact joined to that customer is duplicated (Idempotent Data Pipelines).

How it breaks with real data
  • The merge runs twice — a retry, a manual re-run, a backfill overlapping a scheduled run — and inserts a second identical version. Two rows now claim is_current = TRUE, and every fact joined to that customer is duplicated (Idempotent Data Pipelines).
  • The join predicate uses BETWEEN valid_from AND valid_to. BETWEEN is inclusive at both ends, so an event at exactly the boundary instant matches two versions and that fact row is counted twice.
  • valid_to is left NULL for the current version, and the fact join uses event_time < valid_to. NULL comparisons are not true, so every fact belonging to the current version silently fails to match and disappears from the model (Missing Rows).
  • A change arrives late — the source reported on Thursday that the address changed on Monday. The new version opens on Thursday, so Tuesday's and Wednesday's facts are attributed to the old version, and the interval is a record of when you found out rather than when it happened (Late-Arriving Data).
  • The fact load resolves customer_key by joining on customer_id and is_current = TRUE, because it was written before versioning existed. Every historical fact is now keyed to today's version, so the dimension stores full history and the model reports Type 1 behaviour while paying Type 2 costs.
  • A full rebuild regenerates surrogate keys non-deterministically, and every fact in the warehouse now points at a different version than it did yesterday (Surrogate Keys).
SourceIngestionRawTransformationValidationStorage ModelServingConsumerObservability

What is actually happening

  • The dimension gains three columns. valid_from is the instant the version became effective, valid_to the instant it stopped, and is_current a convenience flag that is redundant with valid_to IS NULL and exists because filtering on a boolean is cheaper to write and to read.
  • Intervals are half-open: [valid_from, valid_to). The version is in force from valid_from inclusive up to but not including valid_to. The new version's valid_from equals the old version's valid_to exactly, so the timeline is covered with no gap and no overlap. This is the single most important implementation detail in the lesson, and using BETWEEN in the join breaks it (Merge Intervals).
  • The current version is marked by valid_to IS NULL or by a sentinel far-future timestamp such as 9999-12-31. The sentinel is uglier and considerably safer, because it makes every interval comparison a normal comparison rather than a three-valued-logic puzzle.
  • Each version gets its own surrogate key. That is what lets a fact reference a specific version without carrying a timestamp predicate of its own (Surrogate Keys).
  • The fact load resolves the key once, at load time, by finding the version whose interval contains the event's timestamp. After that, every analyst query is a plain equality join and is historically correct without anyone thinking about it.
  • Which timestamp bounds the interval is a choice with consequences. Detection time — when the pipeline saw the change — is easy and produces intervals that are records of your own observation. Effective time — when the change happened in the world — is what consumers actually mean and requires the source to tell you, which most do not. Bitemporal models keep both, and cost roughly what that sounds like (Event Time).

The rows, before and after

The whole mechanism is visible in six rows. Before the move, the customer has one version, open-ended. After the move, the old version is closed at the change instant and a new one opens at exactly the same instant, so the timeline is covered end to end with no gap and no overlap.

The boundary is the part to study. valid_to of the old row and valid_from of the new row are the *same value*, and the interval is half-open — the old version covers up to but not including that instant, the new one covers from that instant onward. An event at exactly 2026-03-14 09:12:00 belongs to exactly one version, and it belongs to the new one.

Get this wrong in either direction and it fails silently. Make both bounds inclusive and boundary events count twice. Leave a one-second gap between them and boundary events resolve to nothing and vanish from the model. Neither produces an error, and both change the totals.

Note also what the facts do: nothing. O-7781 was loaded in June 2025 with customer_key = 4470, and it keeps that key forever. The fact table is not touched by the change. That is the payoff — history is immutable because each fact holds a reference to the version that was true when it happened.

BEFORE the change -- dim_customer, one open version

+--------------+-------------+---------+---------+---------------------+---------------------+------------+
| customer_key | customer_id | country | segment | valid_from          | valid_to            | is_current |
+--------------+-------------+---------+---------+---------------------+---------------------+------------+
|         4470 | C-9912      | PL      | SMB     | 2024-01-08 00:00:00 | 9999-12-31 00:00:00 | TRUE       |
+--------------+-------------+---------+---------+---------------------+---------------------+------------+

AFTER the source reports country = DE, effective 2026-03-14 09:12:00

+--------------+-------------+---------+---------+---------------------+---------------------+------------+
| customer_key | customer_id | country | segment | valid_from          | valid_to            | is_current |
+--------------+-------------+---------+---------+---------------------+---------------------+------------+
|         4470 | C-9912      | PL      | SMB     | 2024-01-08 00:00:00 | 2026-03-14 09:12:00 | FALSE      |
|         4471 | C-9912      | DE      | SMB     | 2026-03-14 09:12:00 | 9999-12-31 00:00:00 | TRUE       |
+--------------+-------------+---------+---------+---------------------+---------------------+------------+
                                                   ^^^^^^^^^^^^^^^^^^^^^^
                                                   same instant, no gap, no overlap
                                                   interval is HALF-OPEN: [from, to)

  customer_id C-9912 now appears TWICE. It is no longer a key.
  The key is (customer_id, valid_from). The grain is one row per VERSION.

THE FACTS ARE UNTOUCHED

  fct_orders
  +----------+--------------+---------------------+---------+
  | order_id | customer_key | ordered_at          | revenue |
  +----------+--------------+---------------------+---------+
  | O-7781   |         4470 | 2025-06-02 14:31:00 |  500.00 |   -> PL forever
  | O-8420   |         4470 | 2026-03-14 08:00:00 |  310.00 |   -> PL (before)
  | O-8421   |         4471 | 2026-03-14 09:12:00 |  240.00 |   -> DE (at the boundary)
  | O-9003   |         4471 | 2026-04-11 10:05:00 |  180.00 |   -> DE
  +----------+--------------+---------------------+---------+

  2025 revenue by country still reports 500.00 in Poland, next year,
  and the year after. Nothing restates.

WHAT THE THREE BUGS LOOK LIKE

  BETWEEN valid_from AND valid_to      -> O-8421 matches BOTH rows: 240.00 counted twice
  valid_to = NULL with  ts < valid_to  -> NULL comparison is never true: O-8421 and
                                          O-9003 match NOTHING and leave the model
  two rows with is_current = TRUE      -> every fact for C-9912 duplicated

The predicate a fact must use

The interval logic appears in exactly one place in a well-built model: in the fact load, where the surrogate key is resolved. Every downstream query is then a plain equality join, and no analyst ever writes a validity predicate.

This is worth insisting on. Point-in-time correctness expressed as a predicate that every consumer must remember is a correctness property with a human in the loop, and humans forget. Expressed as a key resolved once at load time, it is a correctness property with a test on it.

The query below shows the resolution, the equality join that follows, and the three ways it is commonly written wrong. Read the wrong ones carefully — each is a single token different from the right one, and each changes the numbers rather than raising an error.

Resolving the version at load time, and the three ways it breaks
1-- CORRECT: half-open interval, sentinel high value, LEFT JOIN with fallback
2INSERT INTO fct_orders
3SELECT o.order_id,
4 COALESCE(c.customer_key, -1) AS customer_key, -- -1 = unknown member
5 d.date_key,
6 o.revenue
7FROM stg_orders o
8JOIN dim_date d ON d.date = CAST(o.ordered_at AS DATE)
9LEFT JOIN dim_customer c
10 ON c.customer_id = o.customer_id
11 AND o.ordered_at >= c.valid_from
12 AND o.ordered_at < c.valid_to; -- strict <, sentinel 9999-12-31
13
14-- Every consumer query is then a plain equality join, and is
15-- historically correct without anyone writing interval logic:
16SELECT c.country, SUM(f.revenue)
17FROM fct_orders f JOIN dim_customer c USING (customer_key)
18GROUP BY 1;
19
20
21-- WRONG 1: BETWEEN is inclusive at BOTH ends.
22 AND o.ordered_at BETWEEN c.valid_from AND c.valid_to
23-- An event at exactly the boundary instant matches the closing row AND
24-- the opening row. That order is counted twice. Every change that lands
25-- on a load boundary -- midnight, for a daily job -- hits this.
26
27-- WRONG 2: NULL for the open interval, compared with <.
28 AND o.ordered_at < c.valid_to -- valid_to IS NULL for current
29-- NULL comparisons are UNKNOWN, never TRUE. Every fact belonging to the
30-- CURRENT version fails to match and leaves the model. The newest data
31-- is the data that disappears, which looks like an ingestion problem.
32-- Fix: COALESCE(c.valid_to, TIMESTAMP '9999-12-31'), or use the sentinel.
33
34-- WRONG 3: resolving against current state instead of event time.
35 AND c.is_current -- the expensive one
36-- Every historical fact is keyed to TODAY'S version. The dimension holds
37-- perfect history that no fact ever references. Interval checks pass,
38-- key stability passes, referential integrity passes -- and the model
39-- behaves exactly like SCD Type 1 while paying SCD Type 2 costs.
40-- Symptom: closed periods keep changing. Nothing ever errors.
41
42
43-- WRONG 4, the query-side version: joining on the natural key.
44SELECT c.country, SUM(f.revenue)
45FROM fct_orders f
46JOIN dim_customer c ON c.customer_id = f.customer_id -- natural key!
47GROUP BY 1;
48-- Matches EVERY version. A customer with three versions contributes
49-- three times. This is why facts should not carry the natural key as a
50-- joinable column, or why it should be named so nobody joins on it.

Wrong 3 is the one to remember. It is the only bug here that leaves every structural check green, because the dimension is flawless — the defect is in which row the fact points at, and no test inside the dimension can see that (Surrogate Keys).

Interval integrity, as tests that block publication

A Type 2 dimension without these checks is a table with three extra columns. The checks are what make the three columns mean something, and they must run before publication rather than after, because every failure here produces silent arithmetic rather than an error.

The pattern of the failures is worth noticing: an overlap double-counts, a gap deletes, a missing current row makes an entity vanish from current-state queries, and a duplicate current row fans out. All four are changes in magnitude with no change in structure, which is exactly the failure class this domain exists to make visible.

Read the misses column on the last row especially. Interval integrity is bookkeeping about your own records; it cannot tell you whether those records describe what actually happened.

The four checks a Type 2 dimension needs, and what each cannot see
CheckExpressesCatchesStill misses
Exactly one row with is_current per natural keyEvery entity has precisely one present-tense description.A merge that inserted without closing (two current rows, fanning out every joined fact), and a merge that closed without inserting (zero current rows, the entity vanishing from current-state queries).Historical rows entirely. A dimension can have perfect current-row hygiene and completely broken history behind it. It also passes if is_current and valid_to IS NULL disagree, unless you assert their consistency separately.
No overlapping intervals per natural key — each row's valid_to equals the next row's valid_fromThe version timeline is a partition of time: complete, and without double cover.Overlaps, which count every fact in the overlapping window twice; and gaps, which make facts inside them resolve to nothing and quietly leave the model.Whether the boundaries are in the right *place*. A perfectly contiguous timeline built from detection timestamps is internally flawless and describes a history that did not happen (Event Time).
Surrogate key stability across a full rebuild of the dimensionA rebuild is safe for facts that already reference this dimension.Non-deterministic key generation — a ROW_NUMBER() or an identity column — which silently repoints every fact in the warehouse at a different version on the next rebuild.A generation rule that is deterministic but computed over different inputs in two places, producing two key spaces that are each internally stable. Only comparing the rules themselves finds that.
As-at spot check: sample facts from a closed period, resolve their attributes, compare with the source's change historyThe fact load resolved keys against event time, and the versions describe reality.A fact load joining on is_current, which no other check here can see; and boundaries that are systematically late because the source reports changes after they take effect.Anything the source's own change history is wrong about, and anything outside the sample. It is manual, slow, and the only check in this table that leaves the platform (Reconciliation).

The first three run on every load and block publication. The fourth is a periodic audit and is the only one that can tell you the model is describing reality rather than merely describing itself consistently.

Building the merge without corrupting it

ENGINE-SPECIFICThe atomicity of the merge stage depends entirely on the engine: a warehouse with a transactional MERGE closes and inserts in one visible step, while an append-only lake table without a transactional layer needs a write-then-swap or a table format that provides snapshot isolation, and without either the two-row window exists on every single run.

The load itself is where the failures are introduced, and the two properties that matter are atomicity and idempotency. Closing the old row and opening the new one must be one operation, or the dimension has an observable state where an entity has zero or two current rows. And a re-run must produce the same table, or every retry is a potential duplicate version.

The stage list below is the shape of a correct SCD2 load. Each stage says what it promises, which is what lets you reason about what a failure between two stages leaves behind. The detection stage is the subtle one: comparing only the versioned columns is what stops a Type 1 attribute change — an email correction — from creating a spurious version.

Notice that publication comes after validation, not before. Publishing then checking means the checks report an incident; checking then publishing means they prevent one. That ordering is the difference between a data platform and a data platform with tests (Contract Enforcement).

An SCD Type 2 dimension load
  1. 1
    Stage source snapshot

    Reads the current source state, or the day's change events, into a staging table with one row per natural key.

    guarantees One row per entity in the staging input, deduplicated by the source's own latest-change ordering.

    fails by Deduplicating by arrival order rather than by commit order, so an out-of-order update wins and the version records a state that was already superseded (CDC Ordering and Transaction Boundaries).

  2. 2
    Detect changes

    Compares staged values against the current dimension row, on the versioned columns only.

    guarantees A change is flagged only when a Type 2 attribute differs; Type 1 attribute differences are routed to an in-place update instead.

    fails by Comparing all columns, so an email or updated_at change creates a spurious version. Or comparing with <> on nullable columns, where NULL <> NULL is unknown and a change from a value to null is not detected (Nullability & Defaults).

  3. 3
    Assign version boundary

    Chooses the timestamp that closes the old version and opens the new one — effective date if the source supplies it, detection time otherwise.

    guarantees The new valid_from equals the old valid_to exactly, so the timeline is contiguous and half-open.

    fails by Using the load's wall-clock start time, so a rerun of yesterday's load stamps today's date and the boundary is wrong in a way no interval check can detect (Idempotent Data Pipelines).

  4. 4
    Generate surrogate key

    Computes the key for the new version deterministically from (natural_key, valid_from).

    guarantees The same version always receives the same key, on every run and every full rebuild.

    fails by A sequence or ROW_NUMBER(), which renumbers on rebuild and repoints every fact in the warehouse at a different version.

  5. 5
    Merge atomically

    Closes the superseded rows and inserts the new versions in a single transactional statement.

    guarantees No reader observes an entity with zero or two current rows.

    fails by Two separate statements. A failure between them leaves the dimension in exactly the state the first check exists to catch — and on an engine without transactional merge, that window exists on every run (Atomic Publish).

  6. 6
    Validate intervals

    Runs the current-row, overlap, gap and key-stability checks against the merged result.

    guarantees Only what the four checks assert. Notably not that the boundaries describe reality.

    fails by Running after publication, which turns a preventable defect into an announced incident (Data Tests).

  7. 7
    Publish

    Makes the validated dimension visible to fact loads and consumers.

    guarantees Consumers see a fully validated, internally consistent dimension, atomically.

    fails by Publishing while fact loads are already resolving keys against it, so some facts key against the old versions and some against the new within a single run (Task Dependencies).

  8. 8
    Load facts against it

    Resolves each fact's dimension keys by interval containment on the event timestamp.

    guarantees Each fact references the version that was in force when the event occurred — which is the entire point of everything above.

    fails by Joining on is_current, which converts the whole model to Type 1 behaviour while leaving every check green.

Read the failsBy column as a checklist for reviewing someone's SCD2 implementation. Six of the eight failures leave the pipeline green, and the last one leaves every data test green as well.

How to build it

Most important first.

  • Use half-open intervals with a far-future sentinel instead of NULL, and write the predicate as event_time >= valid_from AND event_time < valid_to. Never BETWEEN — it is inclusive at both ends and matches two versions at the boundary instant — and never a bare NULL bound, because a null comparison is never true and silently drops every fact belonging to the current version. Put the predicate in a macro so nobody types it by hand (SQL Transformations, Nullability & Defaults).
  • Make the merge idempotent: key it on (natural_key, valid_from) so a re-run updates rather than inserts, and re-running the same day's load produces a byte-identical table (Idempotent Data Pipelines).
  • Generate surrogate keys deterministically from (natural_key, valid_from) so a full rebuild reproduces the same keys and does not silently repoint every fact (Surrogate Keys).
  • Version only the attributes that need it. A Type 2 dimension with SCD1 columns alongside is normal and correct, and it keeps version count down (Slowly Changing Dimensions).
  • Test interval integrity on every load — exactly one current row per entity, no overlaps, no gaps — and block publication on failure. These are the checks that separate a Type 2 dimension from a table with three extra columns (Contract Enforcement).
  • Provide an is_current = TRUE view with the pre-versioning shape, so consumers who only want current state are not forced to learn interval logic and cannot get it wrong (Model Layering).

What this actually promises

Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.

  • Type 2 guarantees a reproducible historical answer for the versioned attributes — and only if three things hold: intervals do not overlap, the fact load resolved keys against event time, and surrogate keys are stable across rebuilds. Any one of the three failing silently removes the guarantee while leaving the schema intact.
  • It guarantees the interval boundaries reflect when the pipeline learned of the change, not when the change occurred, unless the source supplies an effective date. Those differ by however long the source took to tell you (CDC vs Polling).
  • It does not guarantee that a change was observed at all. Two changes between two batch runs collapse into one version, and the intermediate state is not recoverable from the dimension (Batch vs Streaming Ingestion).
  • is_current guarantees nothing on its own. It is derived state, and a merge that fails between closing the old row and inserting the new one leaves an entity with zero current rows — which an inner join treats as a customer who does not exist (Atomic Publish).

Can I trust it?

A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.

The check that would catch this
  • Check one — exactly one current row per entity: SELECT customer_id FROM dim_customer WHERE is_current GROUP BY 1 HAVING COUNT(*) <> 1. Zero rows or two rows are both failures. Two is a fan-out on every fact joined to that customer; zero makes the customer vanish from every current-state query. This must block publication, because both failures are silent and both inflate or deflate revenue without any job failing.
  • Check two — no overlapping intervals: for each entity ordered by valid_from, assert that each row's valid_to equals the next row's valid_from. Overlaps double-count every fact in the overlapping window; gaps make facts in the gap resolve to nothing and drop out of the model. A window function over the dimension expresses both in one query (Window Functions).
  • Check three — key stability across rebuild: rebuild the dimension into a scratch location and assert the surrogate key for each (natural_key, valid_from) is unchanged. This is the check nobody writes and the one that catches the worst failure in the module — a rebuild that silently repoints every fact in the warehouse at a different version (Validating a Backfill Before You Publish).
  • Check four — as-at spot check against the source: take a sample of facts from a past period, resolve their dimension attributes through the model, and compare with what the source's change history says was true at that instant. This is the only check that verifies the fact load resolved keys against event time rather than current state.
  • What every one of them misses: a dimension that is internally perfect and describes an interval that never happened. If the source reports changes late, valid_from is your detection time, and the model will confidently attribute two days of facts to the wrong version — with no overlap, no gap, exactly one current row, stable keys, and a completely reproducible wrong answer. Interval integrity is a check on your bookkeeping, not on reality, and only a source that supplies an effective date can close that gap (Event Time).
  • And none of them detects the most expensive failure of all: a fact load that joins on is_current = TRUE. The dimension passes every check, the history is all there, every interval is perfect — and every fact points at today's version. The model costs what Type 2 costs and behaves exactly like Type 1, and the only symptom is that historical reports keep changing.
Freshness
  • Version boundaries are as granular as the load interval. A daily merge cannot represent two changes on the same day; the second overwrites the first and the intermediate version never existed (Batch vs Streaming Ingestion).
  • A late-arriving change requires splitting an existing version retroactively and rekeying the facts that fall in the corrected interval. This is the most expensive routine operation in the module and the reason effective-dated sources are worth asking for (Late-Arriving Data).
  • The dimension must be loaded before the facts that reference it in the same run. A fact whose version has not been created yet either waits or lands on the unknown member (Task Dependencies).
When the schema or meaning changes
  • Adding a new Type 2 attribute increases the version rate from that point forward. History keeps the old version granularity, so a version boundary means something different before and after the change (Semantic Changes).
  • Demoting an attribute from Type 2 to Type 1 collapses versions and discards history irreversibly. It deserves the review a table drop gets (Breaking Schema Changes).
  • Changing the interval convention — NULL to sentinel, closed to half-open — invalidates every existing join predicate in the platform, and the queries that break do so by returning different numbers rather than by erroring (Impact Analysis).
  • A source that starts supplying effective dates lets you improve interval accuracy going forward. History remains detection-dated, so the two eras are not directly comparable and that should be documented rather than smoothed over.
How to re-run this safely
  • A duplicate-current incident is repaired by closing the spurious version and rebuilding every fact partition loaded while it existed. Lineage tells you which partitions; guessing does not (Lineage Debugging).
  • A gap is repaired by extending the preceding version's valid_to, then rekeying the facts that fell in the gap and were assigned to the unknown member.
  • A rebuild that shifted surrogate keys requires rebuilding the facts as well, which is why deterministic key generation is not a nicety here — it is what makes the dimension recoverable at all (Idempotent Data Pipelines).
  • Retroactively inserting a late change means splitting a version and rekeying facts inside the corrected window. Do it in a scratch location, validate the interval checks and the affected metric, then swap (Planning a Backfill).

What can go wrong

Failure modes
  • Two rows with is_current = TRUE, fanning out every fact joined to that entity.
  • BETWEEN in the join predicate, matching two versions at the exact boundary instant.
  • NULL in valid_to combined with a < comparison, silently dropping every fact belonging to the current version (Missing Rows).
  • A fact load joining on is_current, giving Type 1 behaviour at Type 2 cost — the failure that is invisible from inside the dimension.
  • Surrogate keys regenerated on rebuild, repointing every fact in the warehouse.
  • The mitigation failing: interval checks that run after publication rather than before it, so they report an incident instead of preventing one (Contract Enforcement).
Misreads
  • "is_current is the source of truth for the current version." It is derived state that a half-completed merge can leave wrong. valid_to IS NULL and is_current must agree, and a check should assert that they do.
  • "BETWEEN is fine, the boundary case is rare." It happens on every change that occurs exactly at a load boundary, which for daily loads at midnight is a substantial fraction of them.
  • "COUNT(DISTINCT customer_key) gives the number of customers." It gives the number of versions. Use the natural key, or filter to is_current (Grain: What Does One Row Represent?).
  • "We have SCD2, so our history is correct." Only if facts were keyed against event time. A fact load joining on is_current produces a dimension full of history that no fact ever references (Surrogate Keys).
  • "valid_from is when the change happened." It is when your pipeline learned of it, unless the source supplied an effective date. For a daily batch over a source that reports lazily, those can differ by days (Event Time).
Privacy, retention and access
  • Type 2 retains every previous value of every versioned attribute, which multiplies the personal data held and the period it is held for. That is a retention decision made implicitly by a modelling choice, and it should be made explicitly (Data Retention).
  • A deletion request has to reach every version, not just the current row, and every fact keyed to those versions. A deletion process written against is_current leaves the history intact and the request unfulfilled (Deletion Requests).
  • Versioning sensitive attributes creates a longitudinal record — where someone lived over time, how their status changed — that is more sensitive than any single value in it and needs column-level classification and access control (Row and Column Security).

Operating it

How you see it in production
  • Count of entities with a current-row count other than exactly one, per load. The alert threshold is zero (The Data Quality Dashboard).
  • Versions created per day per attribute. A step change means either a source behaviour change or a merge that stopped detecting equality correctly (Volume Anomalies).
  • Share of fact rows landing on the unknown member. A rise means version resolution is failing, usually because the dimension load is late or an interval has a gap (Pipeline Metrics).
  • Dimension row count against the engine's broadcast threshold, because a versioned dimension crossing it slows every query in the star with no schema change to blame (Broadcast Joins).
What changes at 10x and 100x
  • At 10x entities, the merge becomes the slowest part of the dimension load and usually needs partitioning or clustering on the natural key to stay reasonable (Clustering and Sort Order).
  • At 100x, or with high change rates, mini-dimensions become the standard answer: split the fast-changing attributes into their own small dimension the fact references directly, so the main dimension versions rarely (Dimension Tables).
  • A versioned dimension that outgrows broadcast changes every star query from a local join to a shuffle at once. That is a step change and it should be a monitored threshold, not a surprise (The Shuffle).
What drives cost here
  • Storage is entities times average versions per entity, which is small in absolute terms and can still cross a join-strategy threshold (Broadcast Joins).
  • The merge costs more than a replace: it reads the current dimension, compares, closes rows and inserts rows, and on copy-on-write table formats that rewrites files (Upserts and Merges).
  • The fact load pays an interval join per dimension per run instead of an equality join, which is more expensive at load time and removes the cost from every consumer query (When the Join Strategy Is the Bottleneck).
  • Retroactive corrections are the expensive operation: splitting a version and rekeying facts touches history rather than the current partition (What Backfills Break).
What this approach costs
  • Type 2 buys reproducible history and costs a grain change, a merge instead of a replace, an interval join at load time, three checks that must block publication, and a dimension that can outgrow broadcast. It is worth all of that for the attributes historical reports use and worth none of it for the rest.
  • A far-future sentinel buys freedom from null-comparison bugs and costs a value in the data that looks wrong to anyone reading it for the first time.
  • Deterministic surrogate keys buy rebuild safety and cost a generation rule that must be identical everywhere it is computed, including in the backfill code somebody writes a year later.

Modeling lab — one grain, ten questions

Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.

Modeling lab — one grain, ten questions
Pick the grain of the fact table. The questions do not change; what the table can honestly say about them does.
Fact grain
The transaction and its total. The individual products are gone.
Answered
3
With care
1
Confidently wrong
4
Unanswerable
2
4 of these questions get an answer at this grain that is wrong, and none of them raise an error. That is the whole difficulty: the unanswerable ones announce themselves, and these do not.
Business questionAt this grainWhy
What was revenue by country last month?
Wants: One order, or one order line — either works, provided the measure is additive at that grain and is not summed twice.
answeredThe measure is additive at this grain and each order is counted once.
What is the average order value?
Wants: One order. An average over lines answers a different question entirely.
answeredThe denominator is orders, which is exactly what one row is.
What was revenue by customer country at the time of each order?
Wants: One order, joined to the version of the customer that was current when the order was placed.
WRONG
no history
With a Type 1 dimension every historical order is attributed to the customer's current country. A customer moving from Poland to Germany silently rewrites last year's regional reports, and last month's report no longer reproduces.
What is net revenue after refunds?
Wants: One order, with refunds either netted into the measure or held as a separate signed fact at the same grain.
answeredRefunds net into the measure, or sit beside it as a signed fact at the same grain.
What was the total account balance on each day last year?
Wants: One account-day. A balance is a state, not an event, and cannot be reconstructed by summing transactions unless every transaction since account opening is retained.
WRONGSumming transactions per day gives the daily change in balance, not the balance. The chart has the right shape and the wrong y-axis.
What is month-three retention by signup cohort?
Wants: One user-month of activity, joined to the user's signup month.
WRONGUsers who were active but did not buy are invisible, so retention is understated by exactly the non-buyers.
What share of sessions ended in a purchase?
Wants: One session — which requires a session window over events, because no source system emits a session.
unanswerableNo source system emits a session. Without a session window over events there is no denominator to divide by.
Which products are most often bought together?
Wants: One order line, with the order key retained so lines can be grouped back into baskets.
unanswerableThe most instructive failure in this lab: the model is not wrong, it is at the wrong resolution, and no query can recover what was aggregated away.
What was yesterday's revenue, asked at 06:00 this morning?
Wants: One order, in a period that is not yet closed.
with careThe grain is right and the period is not closed. Orders that happened yesterday and arrive later today are still missing at 06:00.
What was global revenue, across markets that bill in different currencies?
Wants: One order, with both the transaction amount and the converted amount stored, plus the rate and the date the rate applied.
WRONG
no history
Converting at query time with today's rate makes every historical report change daily. Converting once with no record of the rate makes the number unreproducible. Both pass every type check there is.
answeredThe grain is the thing the question is about.
with careIt works, and there is one specific way to get it wrong.
WRONGIt returns a plausible number that is not the answer, and nothing raises.
unanswerableThe resolution needed was aggregated away. No query recovers it.
SIMPLIFIEDA single fact table against ten questions. A real model has several, and a question that one answers badly another may answer exactly — which is the argument for more than one fact table, not for a finer one.
Data pipeline visualizer
Data pipeline visualizer
Toggle a fault and watch it travel. The stage table localises a loss to an arrow rather than to a system; the checks say which signal would have fired; the two revenue figures say whether anybody would have looked.
faults
mitigations
true revenue
934,498.90 minor unitssim
dashboard shows
1,010,654.50 minor unitssim
published
yes
freshness lag
12 minsim
The dashboard is off by 76,155.60 minor unitssim — overstated. 1 of 6 checks fire, so somebody would have found out from a monitor rather than from a person.
stages — what one row means, and how many there are
StageOne row isRows inRows outΔ
Source database
One order, in its current state.4,000
4,000
Change capture
One committed change to one order.4,000
4,000
Event log
One delivered change record — possibly delivered more than once.4,000
4,000
Raw landing
One line in an immutable file, exactly as received.4,000
4,000
Transformation
One order, deduplicated and windowed.4,000
4,000
Serving table
One order, with measures and dimension keys.4,000
4,000
Dashboard
One number, with the grain now invisible.4,000
1
aggregated

Row counts are simulatedsim. The last row is where the grain disappears: one number, with nothing on the screen recording what one row of the source meant.

checks — and what each one is blind to
CheckResultWhat the model foundStill misses
Completeness
Every order the source recorded for the period reached the serving table.
passEvery order in the source for this period is present.Duplicates that coincidentally offset losses, and any period that is not yet closed.
Uniqueness
Each order id appears exactly once in the serving table.
passEvery order id appears exactly once.A genuine duplicate that arrived under a new key — a producer retry with a fresh event id looks like a second order.
Freshness
The newest complete record is recent enough for the decisions this table drives.
passNewest complete record is 12 simulated minutes old.Data that is perfectly fresh and completely wrong. It also fires falsely on a period where the source genuinely produced nothing.
Validity
Every amount is non-null and parses as a number.
passEvery amount is non-null and numeric.A value that is well-typed and wrong — a price in the wrong currency passes every type check there is.
Distribution
The shape of the day resembles the days before it, per country and in total.
passLargest per-country share drift 0.8pp; total volume drift 0.0%.Slow drift, and any error that preserves the shape while changing every value inside it.
Reconciliation
Revenue summed in the serving table equals revenue summed in the source for the same closed period.
FAILServing table reports 1,010,654.50 against a source total of 934,498.90.Anything wrong identically at both ends — a bug in logic shared by the extract and the model reconciles perfectly.
consequences, in the order they occur
  • 1Refunded orders were counted at their full value. Every row is present, unique, fresh and well-typed — and revenue is overstated.
partition load · straggler 2.34×sim the mean
US1,171 rows
DE678 rows
GB507 rows
FR437 rows
PL364 rows
ES350 rows
IT262 rows
NL231 rows
Fail the transform logicreconciliation movestry: Fix the transform logic

The revenue model stops subtracting refunds.

The code does exactly what it was told, and what it was told is wrong. Every row is present, unique, fresh, well-typed and normally distributed — and the number is too high.

SIMULATEDRow counts, revenue and check results all come from the row-level model in src/de/sim/pipeline.ts. Amounts are minor units in a model with no currency.

Where this applies

Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.

  • GENERALHalf-open intervals, one current row, and load-time key resolution are properties of interval arithmetic and hold in every engine. What varies is the merge syntax and whether the engine can express the merge atomically at all.
  • ENGINE-SPECIFICWhether closing the old row and inserting the new one is one atomic operation depends on the engine: a warehouse with transactional MERGE does it in one statement, while an append-oriented lake table without transactions can leave a window where an entity has zero current rows or two, which is exactly the state the checks exist to catch.
  • SOURCE-SPECIFICInterval accuracy is bounded by the source: a CDC stream with commit timestamps gives boundaries close to when the change committed, an effective-dated source gives boundaries that match the real world, and a nightly full extract gives boundaries that are simply the load time and cannot distinguish two changes on the same day.

Where the depth lives

This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.

Observabilityjoin-performance
Domains that do not exist yet
  • Distributed Systems owns the difference between the time an event occurred and the time it was observed, which is what decides whether an SCD2 validity interval describes the world or your own pipeline.
  • DevOps / Production Engineering owns the release process that should make a change to the interval convention a reviewed, staged migration rather than a commit that silently changes every number.