ModelingGENERALENGINE-SPECIFICSCALE-SPECIFIC

Surrogate Keys

A warehouse-generated key with no business meaning, because with history the natural key stops being unique and because source ids change underneath you.

Who needs this, what one row is, and why the obvious build breaks

Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.

The question

Why should a fact join to a warehouse-generated key rather than to the identifier the source system already provides?

Who needs this

Every join in the model, and every engineer who has to migrate a source system without breaking history. The key choice is invisible to analysts until the day it fails, at which point it is the only thing anyone talks about.

What one row is

One surrogate key value identifies one row of one dimension — which, in a versioned dimension, means one *version* of one entity, not the entity itself. That distinction is the entire reason surrogate keys exist and the thing most often misunderstood about them.

The obvious build

Join facts to dimensions on the natural key the source already provides — customer_id, sku, email. The values are meaningful, they are visible in both systems, and debugging a join is trivial because you can read the key.

Why it breaks

The dimension becomes versioned to preserve history. customer_id now appears once per version, so it is no longer unique, and every fact join to it fans out across all versions of that customer (SCD Type 2 in Practice).

How it breaks with real data
  • The dimension becomes versioned to preserve history. customer_id now appears once per version, so it is no longer unique, and every fact join to it fans out across all versions of that customer (SCD Type 2 in Practice).
  • The company migrates CRM systems and every customer is renumbered. Facts hold the old ids and the dimension holds the new ones; the join resolves nothing and years of history detach from their context in one deployment.
  • A customer is created in two source systems with different ids, then merged. Two natural keys now describe one entity, and no single value can join both halves of their history (Source of Truth).
  • sku is reused for a different product after the original is discontinued. Historical fact rows silently reattach to the new product's description, and the old product disappears from every report (Relationships, Keys and Constraints).
  • The natural key is a long composite — (tenant_id, region, external_ref) — and it is repeated on every row of a fact table with billions of rows, costing storage and shuffle bytes on every join (The Shuffle).
SourceIngestionRawTransformationValidationStorage ModelServingConsumerObservability

What is actually happening

  • A surrogate key is a value generated by the warehouse, with no meaning outside it, whose only job is to identify a dimension row uniquely and stably. Typically a monotonically increasing integer or a hash; the mechanism matters less than the property.
  • The decisive property is that a surrogate key identifies a row, not a business entity. In a versioned dimension there is one row per version, so customer_key 4471 means "customer C-9912 as they were between March and July", and that is exactly the level of precision a historical fact needs (SCD Type 2 in Practice).
  • The fact load resolves the surrogate key at load time, by looking up the dimension version that was current when the event occurred. That lookup happens once, during the load, rather than in every analyst's query — which is why point-in-time correctness ends up costing nothing at query time.
  • Surrogate keys also decouple the warehouse from source key churn. When the source renumbers, you update the natural key mapping in the dimension and every fact row is untouched, because facts never held the source id in the first place.
  • They are cheap in the fact table too: a four- or eight-byte integer replaces a long composite or a string, on billions of rows, in exactly the column that every join shuffles (Cardinality: The Label That Took Down Monitoring).
  • Hash-based surrogate keys — a deterministic hash of the natural key plus the version boundary — trade the need for a central sequence for the ability to compute the same key independently in parallel, which matters in distributed loads and makes rebuilds reproducible (Hash Table).

Why the natural key stops working

GENERALThe fan-out shown here is arithmetic rather than an engine behaviour and reproduces identically in every SQL dialect. What differs by engine is only whether a declared primary key on the dimension would have prevented the duplicate rows in the first place — in most analytical engines it would not, because the declaration is not enforced.

The case for surrogate keys is usually made abstractly, which is why it does not land. It is much clearer as two concrete events, both of which happen to every company eventually.

The first is versioning. The moment a dimension preserves history, the natural key appears more than once — once per version — and it is no longer a key at all. A fact joining on it matches every version of that entity, so a customer with three address changes multiplies every one of their orders by three.

The second is migration. Source systems get replaced, merged, and renumbered. A natural key is only as stable as the system that issues it, and that system's stability is not something the data platform controls. Facts that hold source ids are hostage to someone else's migration plan.

The layout below shows both failures against the same data. Read the middle block carefully: the join is legal, the SQL is unremarkable, and one customer's single order has become three rows of revenue.

dim_customer, versioned, joined on the NATURAL key
+-------------+---------+---------+------------+------------+
| customer_id | country | segment | valid_from | valid_to   |
+-------------+---------+---------+------------+------------+
| C-9912      | PL      | SMB     | 2024-01-01 | 2025-03-14 |
| C-9912      | DE      | SMB     | 2025-03-14 | 2026-01-20 |
| C-9912      | DE      | Enterp. | 2026-01-20 | NULL       |
+-------------+---------+---------+------------+------------+
   customer_id is NOT unique. It was never going to be.

fct_orders joined ON f.customer_id = d.customer_id:

  order O-7781, revenue 500.00, placed 2025-06-02
    -> matches version 1 (PL, SMB)        500.00
    -> matches version 2 (DE, SMB)        500.00
    -> matches version 3 (DE, Enterprise) 500.00
                                        ---------
    one order, reported revenue           1500.00

   No error. Row count tripled. Revenue tripled. Every version of
   every customer contributes once.

Same data, joined on the SURROGATE key resolved at load time:
+--------------+-------------+---------+---------+------------+------------+
| customer_key | customer_id | country | segment | valid_from | valid_to   |
+--------------+-------------+---------+---------+------------+------------+
|         4470 | C-9912      | PL      | SMB     | 2024-01-01 | 2025-03-14 |
|         4471 | C-9912      | DE      | SMB     | 2025-03-14 | 2026-01-20 |
|         4472 | C-9912      | DE      | Enterp. | 2026-01-20 | NULL       |
+--------------+-------------+---------+---------+------------+------------+

  order O-7781 was loaded with customer_key = 4471, chosen because
  2025-06-02 falls inside [2025-03-14, 2026-01-20).

    -> matches exactly one row            500.00

  And when the CRM migration renumbers C-9912 to CUST-000441209,
  you update ONE column in three dimension rows. No fact is touched.

Resolving the key at load time

The work that makes surrogate keys valuable happens once, in the fact load, and never again. The load takes the event, its timestamp, and the natural key of each dimension it references, and finds the dimension version that was valid at that instant.

That is the whole trick. Point-in-time correctness is expensive if every analyst has to express it — a BETWEEN valid_from AND valid_to predicate on every dimension in every query, which most people will forget and some will get subtly wrong. Resolving it once at load time moves the cost to a place where it can be tested and where it is paid once per row rather than once per query per row.

Two ways to generate the key are shown. A sequence is simple and requires a single writer. A deterministic hash of (natural_key, valid_from) requires no coordination, produces the same key on every rebuild, and is therefore what makes a full dimension rebuild safe for facts that already reference it.

Generating the key, and resolving it during the fact load
1-- 1. Generating the surrogate key deterministically.
2-- Same inputs -> same key, on every rebuild, in every worker.
3-- A sequence would also work, but only with a single writer, and a
4-- full rebuild would renumber everything.
5SELECT
6 ABS(HASH(customer_id || '|' || CAST(valid_from AS VARCHAR))) AS customer_key,
7 customer_id, country, segment, valid_from, valid_to, is_current
8FROM stg_customer_versions;
9
10-- ANTI-PATTERN, seen in production more often than it should be:
11-- ROW_NUMBER() OVER (ORDER BY customer_id, valid_from) AS customer_key
12-- It is unique, it looks fine, and the next full rebuild renumbers every
13-- row -- silently repointing every existing fact at a different version.
14
15-- 2. Resolving the key during the fact load: pick the version that was
16-- current WHEN THE EVENT HAPPENED, not the version that is current now.
17INSERT INTO fct_orders
18SELECT
19 o.order_id,
20 COALESCE(c.customer_key, -1) AS customer_key, -- -1 = unknown member
21 d.date_key,
22 o.quantity,
23 o.revenue
24FROM stg_orders o
25JOIN dim_date d
26 ON d.date = CAST(o.ordered_at AS DATE)
27LEFT JOIN dim_customer c
28 ON c.customer_id = o.customer_id
29 AND o.ordered_at >= c.valid_from
30 AND o.ordered_at < COALESCE(c.valid_to, TIMESTAMP '9999-12-31');
31 -- half-open interval: >= from, < to. Using BETWEEN here matches
32 -- two versions at the exact boundary instant and fans out.
33
34-- 3. Every analyst query is now free of point-in-time logic:
35SELECT c.country, SUM(f.revenue)
36FROM fct_orders f JOIN dim_customer c USING (customer_key)
37GROUP BY 1;
38-- ...and it is historically correct, because the key already encodes
39-- WHICH VERSION was in force ([[scd-type-2]]).

The COALESCE(..., -1) and the LEFT JOIN are load-bearing. An inner join here silently deletes any fact whose dimension context has not arrived yet, which converts a late dimension into missing revenue (Missing Rows).

What surrogate keys cost

It is worth being honest about the price, because the argument is usually presented as free and it is not. A surrogate key is a number that means nothing to anyone outside the warehouse. Every debugging conversation with a source-system engineer now requires a translation step, and every ad-hoc query needs one more join before a human can read the result.

They also introduce a class of failure that natural keys do not have: the keys themselves can be wrong. A regenerated key, a hash computed over a different column list in two models, or a version resolved against the wrong timestamp all produce a model that passes every structural test and attributes history incorrectly.

The decision below is genuinely a decision. Small models over stable sources with no history requirement do fine on natural keys, and adding surrogate keys there is machinery without a payoff. The moment either history or a source migration enters the picture, the calculus changes completely.

What the key choice actually moves
Bytes in the fact key column, scanned and shuffled

An eight-byte integer versus a composite string, on every row of the largest table, read by every query that joins. This is where the storage argument lives.

Fact-load lookup against each dimension

One join per dimension per run, added to the load in exchange for removing point-in-time logic from every consumer query. It grows with dimension count, not with fact volume.

Human debugging time

The cost nobody budgets. Every investigation needs a translation from a meaningless number to something a source-system engineer recognises, and it is paid on every incident.

Rework after a source migration

Near zero with surrogate keys — a column update in the dimension. Without them it is a rekey of every fact row, which is why this driver is small in one design and unbounded in the other.

Key generation compute

Hashing per dimension row is trivial, and dimensions are small. Almost always the least significant driver, and almost always the one that gets discussed first.

Relative weights, not measurements. The ordering is what transfers between platforms; the magnitudes do not, and this domain never publishes a price.

Relative weights for a typical star-schema load, shown to establish an ordering rather than as measurements. The teaching is the fourth row: its weight depends entirely on which design you picked, which is the only driver here that is a step function rather than a gradient.

Which key should facts join on?

Does this dimension preserve history, and how stable is the source's key space?

Natural key from the source

when The dimension is not versioned, the source key space is stable and single, and the model is small enough that join width is irrelevant.

cost Readable joins and no generation machinery. You are betting that history will never be versioned and the source will never be replaced — and losing that bet means rekeying every fact you have (Relationships, Keys and Constraints).

Sequence-generated surrogate key

when A single-writer load, a warehouse with cheap identity columns, and a team that will not do frequent full rebuilds.

cost A coordination point in the load, and a full rebuild that renumbers everything unless the mapping is persisted. Buys narrow facts and version-level precision.

Deterministic hash surrogate key

when Distributed loads, frequent full rebuilds, or any pipeline where reproducibility matters more than key readability.

cost Collision probability to reason about, and a generation rule that must be byte-identical everywhere it is computed. Buys rebuild safety and no coordination (Idempotent Data Pipelines).

Natural key stored alongside a surrogate key

when Almost always, in practice — the surrogate for joining, the natural key retained as an attribute for debugging and reconciliation.

cost Redundant bytes in the dimension, and a small risk that someone joins on the wrong one. Buys the ability to talk to source-system engineers without a lookup table.

How to build it

Most important first.

  • Give every dimension a surrogate key and make it the only thing facts join on. Keep the natural key in the dimension as an attribute, because it is what you need for debugging and for reconciliation against the source.
  • Generate the key deterministically if you can — a hash of (natural_key, valid_from) — so a full rebuild reproduces the same keys and does not orphan facts that were loaded earlier (Idempotent Data Pipelines).
  • Reserve a key for the unknown member — conventionally -1 or 0 — so facts with missing or not-yet-arrived context load and remain countable instead of being dropped by an inner join (Dimension Tables).
  • Resolve the key in the fact load against the dimension version valid at the event's timestamp, not against the current version. Getting this wrong is what turns an SCD2 dimension into an expensive Type 1 (SCD Type 2 in Practice).
  • Never expose surrogate keys to consumers as business identifiers. They are internal plumbing; a report showing customer_key = 4471 has leaked an implementation detail that means nothing to anyone (Schema Leakage).
  • Keep a mapping from natural key to surrogate key as a first-class artefact, because every debugging session and every source migration goes through it.

What this actually promises

Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.

  • A surrogate key guarantees uniqueness of a dimension row if the generation process guarantees it. A sequence does; a hash does up to collision probability; a ROW_NUMBER() recomputed on every full rebuild does not, and that last one is a real and common defect (Idempotent Data Pipelines).
  • It guarantees stability against source key churn, and nothing about source *value* churn. If the source changes what a customer id means rather than what it is, no key strategy helps (Semantic Changes).
  • It guarantees that a fact's dimension reference is a point-in-time reference, but only because the load resolved it that way. The key itself carries no timestamp and cannot be checked for correctness after the fact.
  • It does not guarantee referential integrity. Analytical engines usually do not enforce foreign keys, so a fact can hold a surrogate key that no dimension row has (Database Constraints).

Can I trust it?

A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.

The check that would catch this
  • Test uniqueness of the surrogate key in the dimension, on every load. It is the assumption every star join rests on, and it is one query (Data Tests).
  • Test that every surrogate key in a fact resolves to exactly one dimension row, and monitor the unresolved rate as a percentage over time rather than only alerting at zero (The Dimensions of Data Quality).
  • Test key stability across a full rebuild: rebuild the dimension into a scratch location and assert that the surrogate key for each (natural_key, valid_from) pair is unchanged. This catches the ROW_NUMBER() defect that only shows up after a rebuild, when every fact in the warehouse is pointing at the wrong version (Validating a Backfill Before You Publish).
  • What these miss: whether the key resolved to the *right version*. A fact keyed against the current dimension row instead of the one valid at event time passes uniqueness, passes referential integrity, resolves perfectly, and reports history under today's attributes. Only an as-at spot check against the source's change history finds it (SCD Type 2 in Practice).
Freshness
  • Surrogate key resolution couples the fact load to the dimension load: the fact cannot be keyed until the dimension version it needs exists. That ordering is a freshness cost paid on every run (Task Dependencies).
  • A fact arriving before its dimension row — a first order from a brand-new customer — must either wait, or land on the unknown member and be corrected later. Both are legitimate; the choice must be documented because it changes what "complete" means for the newest period (Late-Arriving Data).
  • Deterministic hash keys remove the ordering constraint for the key itself, since the key can be computed without reading the dimension. The dimension row still has to exist before the join resolves, so the constraint moves rather than disappearing.
When the schema or meaning changes
  • Adding versioning to a previously unversioned dimension changes the surrogate key from "one per entity" to "one per version". Facts loaded before the change point at entity-level keys that now mean the first version, and that reinterpretation needs a deliberate backfill decision (Planning a Backfill).
  • Changing the key generation mechanism — sequence to hash, or a different hash input — invalidates every existing fact reference. It is a full-model rebuild, not a schema change (Breaking Schema Changes).
  • A source migration is exactly the event surrogate keys exist for: update the natural-key attribute in the dimension, leave surrogate keys alone, and no fact row is touched (Expand and Contract Migrations).
How to re-run this safely
  • Deterministic keys make a dimension rebuild safe: the same inputs produce the same keys, so facts loaded earlier still resolve. Non-deterministic keys make a rebuild an incident, because every fact now points at a different row (Idempotent Data Pipelines).
  • If keys did shift, recovery means rebuilding the facts too, which requires the raw events and the dimension history to be re-derivable. This is the scenario that makes deterministic generation worth the extra care up front (Keeping Raw History: The Recovery Position and the Liability).
  • Keep the natural key on the fact as well, as a debugging column, if storage allows. It is redundant by design and it is what lets you re-derive the correct surrogate key after a key incident.

What can go wrong

Failure modes
  • Surrogate keys regenerated by ROW_NUMBER() on a full rebuild, silently reassigning every fact to a different dimension row.
  • A fact keyed against the current dimension version rather than the version valid at event time, converting an SCD2 model into a Type 1 model that costs SCD2 storage (SCD Type 2 in Practice).
  • Surrogate keys leaking into a report or an API, where they mean nothing and cannot be looked up by anyone outside the platform (Schema Leakage).
  • A hash collision, or — far more often — a hash computed over a different set of input columns in two places, producing two keys for one version.
  • The mitigation failing: an unknown member that absorbs unresolved keys so gracefully that a broken join is invisible until someone asks why Unknown is the largest segment.
Misreads
  • "Surrogate keys are premature optimisation." They are not an optimisation at all. They are what makes versioned history joinable, and retrofitting them after history exists means rekeying every fact.
  • "The surrogate key identifies the customer." It identifies a *version* of the customer in a versioned dimension. Counting distinct customer_key values counts versions, not customers, and that error inflates every unique-customer metric (Grain: What Does One Row Represent?).
  • "We can always join on the natural key if we need to." You can, and you will get every version of that entity, which fans out the fact and silently multiplies every measure (Grain: What Does One Row Represent?).
  • "A hash key means we never need a sequence." True, and it also means the key is a function of its inputs — change the input column list anywhere and you have silently created a second key space.

Operating it

How you see it in production
  • Distinct surrogate keys versus row count in each dimension, per load. They must be equal (Pipeline Metrics).
  • Share of fact rows on the unknown member, over time, per dimension. A step change is a key-space event (Volume Anomalies).
  • Key stability check output from each full rebuild, retained as a run artefact, so a key shift is attributable to a deployment rather than discovered weeks later (Data Lineage).
What changes at 10x and 100x
  • At 10x facts, the byte saving on the key column becomes a visible share of scan and shuffle cost.
  • At 100x, key generation strategy decides whether the load parallelises. A central sequence is a coordination point; a deterministic hash is not (Data Skew).
  • More dimensions means more lookups per fact load, and the load's runtime becomes a function of dimension count rather than of fact volume alone (Broadcast Joins).
What drives cost here
  • A narrow integer key on a billion-row fact costs far fewer bytes than a composite or string natural key, and those bytes are read on every scan and moved on every shuffle join (Scan Cost).
  • The lookup during the fact load costs a join per dimension per run, which is real compute added to every load in exchange for removing that work from every query (Compute Waste).
  • Deterministic hash keys cost CPU per row and remove the need for a coordinated sequence, which is usually the better trade in a distributed load (Distributed Data Processing).
What this approach costs
  • Surrogate keys cost an extra join to see any human-readable value, a key with no meaning outside the warehouse, and a debugging step where you must translate before you can talk to anyone from the source system. They buy versioning, source-migration immunity and narrower facts.
  • They also cost a lookup on every fact load and an ordering dependency between dimension and fact loads, which is real operational coupling.
  • Deterministic hash keys buy reproducibility and parallelism and cost a collision probability you must reason about and a generation rule that must be identical everywhere it is computed.

Modeling lab — one grain, ten questions

Change an input and watch which number moves — and which one does not. Everything here comes from a model in this repository, not from a measurement.

Modeling lab — one grain, ten questions
Pick the grain of the fact table. The questions do not change; what the table can honestly say about them does.
Fact grain
The transaction and its total. The individual products are gone.
Answered
3
With care
1
Confidently wrong
4
Unanswerable
2
4 of these questions get an answer at this grain that is wrong, and none of them raise an error. That is the whole difficulty: the unanswerable ones announce themselves, and these do not.
Business questionAt this grainWhy
What was revenue by country last month?
Wants: One order, or one order line — either works, provided the measure is additive at that grain and is not summed twice.
answeredThe measure is additive at this grain and each order is counted once.
What is the average order value?
Wants: One order. An average over lines answers a different question entirely.
answeredThe denominator is orders, which is exactly what one row is.
What was revenue by customer country at the time of each order?
Wants: One order, joined to the version of the customer that was current when the order was placed.
WRONG
no history
With a Type 1 dimension every historical order is attributed to the customer's current country. A customer moving from Poland to Germany silently rewrites last year's regional reports, and last month's report no longer reproduces.
What is net revenue after refunds?
Wants: One order, with refunds either netted into the measure or held as a separate signed fact at the same grain.
answeredRefunds net into the measure, or sit beside it as a signed fact at the same grain.
What was the total account balance on each day last year?
Wants: One account-day. A balance is a state, not an event, and cannot be reconstructed by summing transactions unless every transaction since account opening is retained.
WRONGSumming transactions per day gives the daily change in balance, not the balance. The chart has the right shape and the wrong y-axis.
What is month-three retention by signup cohort?
Wants: One user-month of activity, joined to the user's signup month.
WRONGUsers who were active but did not buy are invisible, so retention is understated by exactly the non-buyers.
What share of sessions ended in a purchase?
Wants: One session — which requires a session window over events, because no source system emits a session.
unanswerableNo source system emits a session. Without a session window over events there is no denominator to divide by.
Which products are most often bought together?
Wants: One order line, with the order key retained so lines can be grouped back into baskets.
unanswerableThe most instructive failure in this lab: the model is not wrong, it is at the wrong resolution, and no query can recover what was aggregated away.
What was yesterday's revenue, asked at 06:00 this morning?
Wants: One order, in a period that is not yet closed.
with careThe grain is right and the period is not closed. Orders that happened yesterday and arrive later today are still missing at 06:00.
What was global revenue, across markets that bill in different currencies?
Wants: One order, with both the transaction amount and the converted amount stored, plus the rate and the date the rate applied.
WRONG
no history
Converting at query time with today's rate makes every historical report change daily. Converting once with no record of the rate makes the number unreproducible. Both pass every type check there is.
answeredThe grain is the thing the question is about.
with careIt works, and there is one specific way to get it wrong.
WRONGIt returns a plausible number that is not the answer, and nothing raises.
unanswerableThe resolution needed was aggregated away. No query recovers it.
SIMPLIFIEDA single fact table against ten questions. A real model has several, and a question that one answers badly another may answer exactly — which is the argument for more than one fact table, not for a finer one.

Where this applies

Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.

  • GENERALThe argument — natural keys stop being unique once history is versioned, and source key spaces change — is independent of engine and applies to any model that preserves dimension history. Only the generation mechanism varies.
  • ENGINE-SPECIFICWhether a central sequence is practical depends on the engine: a single-node warehouse can hand out identity values cheaply, while a distributed load has no cheap global counter and generally prefers deterministic hashing, which changes the collision reasoning you have to do.
  • SCALE-SPECIFICBelow a few million fact rows the byte saving is irrelevant and a natural-key join is perfectly workable; the argument for surrogates at that scale is entirely about versioning and migrations, not storage. Above a billion rows the key width itself becomes a measurable share of scan and shuffle bytes.

Where the depth lives

This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.

Domains that do not exist yet
  • DevOps / Production Engineering owns the migration choreography — expand, dual-write, contract — that a source-system renumbering follows, and surrogate keys are what let the analytical side sit out that migration entirely.