AI DataGENERALSCALE-SPECIFICSIMPLIFIED

Feature Pipelines

Raw events to transformations to features to two consumers. The characteristic failure is one logical column computed by two pipelines, and it is a data-engineering failure with a data-engineering fix.

What actually happens How to build it Can I trust it?▶ Run the lab

Who needs this, what one row is, and why the obvious build breaks

Every lesson starts from the consumer, because designing from the source outward is this domain's characteristic mistake.

The question

The same feature is computed by a batch job for training and by a service at request time. What makes those two numbers differ, and how would you find out?

Who needs this

Two pipelines that must agree about one column: an offline job assembling historical rows, and a serving path computing the same values under a latency budget. Downstream of both is anyone comparing offline results with production behaviour and finding they do not match — a comparison that is only meaningful if the inputs were identical (Reconciliation).

What one row is

One row is one entity at one point in time: this customer, as they were at 10:03 on Tuesday. The point-in-time part is the whole difficulty. A row that carries today's values against last year's event is not a feature row, it is a join against current state wearing a timestamp (Slowly Changing Dimensions, Grain: What Does One Row Represent?).

The obvious build

Compute the features in SQL over the warehouse for the offline dataset, and re-implement the same logic in the service for the online path, because the warehouse cannot answer a point lookup inside a request. Two implementations, one specification, and for a first version this is both obvious and workable — the online path genuinely has different constraints (OLTP vs OLAP).

Why it breaks

A null-handling bug is fixed in the batch SQL — a missing value becomes zero rather than null — and nobody changes the serving code. The same customer now has two different values for the same feature depending on which pipeline is asked (Nullability & Defaults).

How it breaks with real data

A null-handling bug is fixed in the batch SQL — a missing value becomes zero rather than null — and nobody changes the serving code. The same customer now has two different values for the same feature depending on which pipeline is asked (Nullability & Defaults).
The batch job aggregates over a full calendar day, so a row describing an event at 10:03 includes activity from 14:00 the same day. Offline results improve for a reason that cannot exist in production, where 14:00 has not happened yet (Event Time).
The offline job joins to a customer dimension holding only current state, so historical rows carry today's segment. The feature is computed correctly and describes a world that did not exist at the time (SCD Type 2 in Practice).
Late-arriving events change a batch aggregate after the fact, so re-running the same offline job next month produces different rows for the same period. The dataset is not reproducible and nothing announces it (Late-Arriving Data).
The serving path reads a cached value refreshed hourly while the offline path assumes values as of the request instant. The two agree on the definition and differ by a staleness nobody wrote down (TTL and Expiry).
A timezone or unit difference — minor units versus major, local versus UTC day boundary — makes two implementations differ by a constant on a subset of rows, which is precisely the difference least likely to be noticed (Semantic Changes).

Source→Ingestion→Raw→Transformation→Validation→Storage Model→Serving→Consumer→Observability

What is actually happening

Training/serving skew is the same logical column computed by two pipelines, and that is a data-engineering failure with a familiar name: it is the dual-write problem in a different costume. Two writers, one meaning, no mechanism keeping them equal (The Dual Write Problem).
The reason there are two pipelines is legitimate and structural. Assembling millions of historical rows is a large scan; serving one entity inside a request is a point lookup. Those are opposite workloads and they belong in different stores (OLTP vs OLAP, Workload Isolation).
The mistake is concluding that two stores require two definitions. Two materialisations of one definition is fine; two definitions is the failure — and the difference between those two sentences is the entire lesson (The Metrics Layer).
Point-in-time correctness is an as-of join, and it is the same operation this domain already teaches for slowly changing dimensions: for each event, take the feature value whose validity interval contains the event time, not the value that is current now (SCD Type 2 in Practice, Stream Joins). A naive join against current state does not produce a slightly wrong row. It produces a row that contains information from after the moment it claims to describe, which makes any offline result computed from it optimistic in a way that cannot reproduce in production (Event vs Snapshot Modeling).
Late data is the second time dimension. A batch aggregate over a period keeps changing while events for that period keep arriving, so "the value of this feature for Tuesday" is only well defined once you say as of when you are asking (Late-Arriving Data, Watermarks).
The strongest available fix is not a better second implementation. It is to log the values actually served and use those logs as the offline dataset: the two pipelines then agree by construction, because there is only one computation and the other path reads its output (The Event Log). What a feature store is, in this domain's terms: a keyed low-latency serving table, a historical table supporting as-of retrieval, and one shared definition with metadata joining them. It is a materialisation pattern, not a new category of system (Data Marts).

One definition, two materialisations

SIMPLIFIEDDrawn with one computation job producing both materialisations. In practice a streaming path often produces the online values while a batch path produces the offline ones, which is acceptable exactly as long as both are generated from the same versioned definition — and is the single most common place where two definitions creep back in.

The diagram below has one property worth staring at: the definition appears once, and everything downstream of it is a materialisation. That is the target state, and the common state is two boxes labelled "compute features" on the two paths with nothing connecting them but an intention (The Transformation DAG).

Both stores are justified. A historical assembly is a large scan over columns; a request-time lookup is a keyed read under a latency budget. This domain has a whole module about why those workloads do not share storage well, and nothing about features changes that argument (OLTP vs OLAP).

The dashed edge is the one that removes the failure entirely where it is available. Logging the feature values that were actually served, and assembling training rows from those logs, means there is one computation and the second path reads its output — the same move as any other place this domain refuses to compute the same number twice (The Event Log, Agent Observability Data).

One definition, two materialisations, and the log that removes the disagreement

UserLLMAgentToolDataDecisionHumanGuardrail

Point-in-time correctness is an as-of join

GENERALThe as-of join is expressible in any engine, but the syntax and the cost differ sharply: some engines provide a native as-of or range join, others require a correlated subquery or a window function, and the physical cost depends on whether the feature table is sorted by entity and time. The correctness argument is identical everywhere; the query plan is not (Clustering and Sort Order).

The most damaging skew is not an implementation difference. It is a historical row containing information from after the moment it claims to describe, which makes every offline result computed from it optimistic in a way production cannot reproduce (Event Time).

The query below is deliberately ordinary. It is the same as-of join this domain already teaches for slowly changing dimensions: for each event, pick the feature row whose validity interval contains the event time. The only thing that makes it feel unfamiliar is the vocabulary attached to the output (SCD Type 2 in Practice).

The second query is the one people write first, and it is wrong in a way that produces better-looking results — which is the worst possible combination, because nothing about the output invites suspicion. Joining to a current-state table gives every historical row today's attributes, and the closer the model gets to the present, the more that looks like skill (Event vs Snapshot Modeling).

Assembling historical rows: the as-of join, and the join that leaks the future

1-- Feature values are stored with validity intervals, exactly like an SCD2
2-- dimension. One row per entity per period during which the value held.
3--   customer_features(customer_id, feature_set_version,
4--                     orders_7d, days_since_last_order,
5--                     valid_from, valid_to)
6 
7-- CORRECT: each event joins to the value that was valid when it happened.
8SELECT  e.event_id,
9        e.customer_id,
10        e.event_time,
11        f.orders_7d,
12        f.days_since_last_order,
13        f.feature_set_version          -- travels with the row, always
14FROM        labelled_event   e
15LEFT JOIN   customer_features f
16       ON   f.customer_id  = e.customer_id
17      AND   f.valid_from  <= e.event_time
18      AND   e.event_time   <  f.valid_to;   -- half-open: no double match
19 
20-- WRONG, and it looks better: current state joined onto historical events.
21SELECT  e.event_id, e.customer_id, e.event_time,
22        c.orders_7d                     -- as of now, not as of e.event_time
23FROM        labelled_event  e
24JOIN        customer_current c ON c.customer_id = e.customer_id;
25 
26-- ALSO WRONG, more subtly: a window that spans the event.
27-- Aggregating the whole calendar day of the event includes activity that had
28-- not happened when the decision was made.
29SELECT  e.event_id,
30        COUNT(*) AS orders_today       -- includes orders after e.event_time
31FROM        labelled_event e
32JOIN        orders         o
33       ON   o.customer_id = e.customer_id
34      AND   o.created_at >= DATE_TRUNC('day', e.event_time)
35      AND   o.created_at <  DATE_TRUNC('day', e.event_time) + INTERVAL '1 day'
36GROUP BY e.event_id;

Three queries, one difference: whether the row can see past its own timestamp. The half-open interval in the correct version matters as much as the join itself — inclusive bounds on both sides match two feature rows at a boundary and silently duplicate the event, which is an ordinary fan-out bug producing an extra row nobody counted (Duplicate Rows).

Where skew actually comes from

Every row below produces two different values for one logical column, and in every case both pipelines report success. That is the signature of this failure: no error, two numbers, and a discrepancy that surfaces weeks later as "it worked better offline" (The Pipeline Succeeded. The Data Is Wrong.).

Read the cause column and notice that only the first row is about code. The rest are about inputs — staleness, defaults, time semantics, late arrival — which is why "we use the same SQL on both sides" is not a defence and why parity has to be measured on values rather than argued from implementations.

The last row is the mitigation failing, and it is worth avoiding specifically because it destroys the only check that catches everything else: a parity check that recomputes with the wrong as-of time fires constantly, is wrong every time, and is muted within a week (Alert Fatigue: The Page Nobody Reads).

Six sources of training/serving skew

Trigger	Symptom	Cause	Response
A null-handling or rounding fix is applied to one implementation.	Offline and served values differ for a subset of entities, concentrated on the sparse ones.	Two hand-written implementations of one specification, kept equal by intention (The Dual Write Problem).	One versioned definition compiled to both paths, or log-and-reuse so only one computation exists (The Transformation DAG).
The serving path reads a value from a cache.	Values agree on definition and differ by a lag that varies with load.	A staleness the offline path has no model of (TTL and Expiry).	Make freshness part of the feature definition — the offline computation reproduces the same staleness, or the served value is computed live (The Freshness SLO).
A feature aggregates over the calendar day of the event.	Offline results are excellent and production results are not.	The window spans the event, so the row contains information from after the moment it describes (Event Time).	Windows must be strictly backward-looking from the event time, and the check for it is a test on the query, not a review (Data Tests).
The offline job joins to a current-state dimension.	Historical rows carry today's segment, region or tier.	The dimension holds only current state, so an as-of join is not expressible (Slowly Changing Dimensions).	Store feature and dimension values with validity intervals, and join on the interval containing the event time (SCD Type 2 in Practice).
Events for a period continue to arrive after it closes.	Re-running the same offline job next month produces different rows for the same period.	A window still receiving data, with no cut-off policy (Late-Arriving Data).	Declare a cut-off after which values are frozen, or declare explicitly that history is recomputed — and record which, on the dataset (Data Contracts).
A parity check is added after an incident.	It fires on every fresh feature, is investigated twice, and is muted.	Recomputation used the current timestamp instead of the request's as-of timestamp — the mitigation failing.	Parity must recompute at the served request's as-of time, from the logged inputs, with a stated tolerance per feature (Reconciliation).

Measuring parity, and what it will still miss

Skew is not a bug that gets fixed once. It is a drift between two materialisations that reappears every time either side changes, which makes it a monitoring problem rather than a review problem (Data Observability).

The checks below are cheap and mechanical, and their blind spots follow the usual pattern: they compare values and they cannot compare meanings. A feature computed identically on both paths and defined wrongly passes every one of them, which is why the definition still needs a human who knows the domain (Two Dashboards, Two Numbers).

Note which check finds unit and timezone differences. Point-wise parity on a traffic-weighted sample can easily miss a constant offset affecting a minority segment; the distribution comparison finds it immediately, because a systematic difference shows up in a mean long before it shows up in a sampled pair (Distribution Tests).

Continuous parity monitoring between offline and served features

Check	Expresses	Catches	Still misses
Sampled served requests recomputed through the offline path at the same as-of timestamp, compared value by value with a per-feature tolerance.	The two materialisations agree on the cases production actually sees.	Implementation drift after a one-sided fix, a cache the offline path does not model, a definition version mismatch between paths (The Dual Write Problem).	Segments that are rarely served. The sample follows traffic, so a feature wrong only for a small cohort passes comfortably (Data Skew).
Null and default rates per feature on both paths, compared over the same period.	Missing values are being handled the same way.	A default of zero on one side and null on the other, a join that silently drops entities, a lookup miss treated as absence (Nullability & Defaults).	Both paths defaulting identically and wrongly — a zero that should have been "unknown" is consistent, comparable and still misleading downstream.
Distribution comparison — range, mean per segment, quantiles — between offline values and served values.	The two populations look like the same measurement.	Unit differences, timezone boundary differences, and any constant offset affecting a subset (Distribution Tests).	Compensating differences that leave the distribution intact while individual entities are wrong, which is exactly what a per-entity key mismatch produces.
Age of the data behind each served value, measured on the serving path.	The served feature is as fresh as its definition claims.	A stale online store, a refresh job that stopped, a cache warming path that quietly fell behind (Freshness Monitoring).	Fresh values that are wrong. Freshness and correctness are independent, and a promptly-refreshed incorrect value is the hardest kind to doubt.
Distinct feature definition versions present in any assembled historical dataset.	The dataset was built from one definition of each feature.	A dataset spanning a definition change, a partial backfill, two teams' versions merged by a join (Semantic Changes).	A single version that was changed in place without bumping — version discipline is only as honest as the process that increments it.

Four of the five compare two computations of the same thing. That is the whole method for this failure class: skew cannot be established by reading two implementations and agreeing they look equivalent, only by comparing the values they actually produced (Reconciliation).

How to build it

Most important first.

Write the definition once and materialise it twice. Either one job computes values and publishes to both an offline table and an online store, or one transformation definition is compiled into both paths — never two hand-written implementations of one specification (The Transformation DAG).
Prefer log-and-reuse where the shape allows it: record the exact feature values used to serve each request, and build offline rows from those logs. Skew becomes structurally impossible for every feature handled this way (Agent Observability Data).
Make every feature row carry an entity key and an event timestamp, and assemble training rows with an as-of join. A feature table without a validity interval cannot answer a historical question correctly, only conveniently (SCD Type 2 in Practice). Record the definition version on every row. A feature whose computation changed is a different column that happens to share a name, and comparing across the change is a measurement error (Semantic Changes).
State a freshness SLO per feature and measure it on the serving path. "Orders in the last seven days" served from an hourly cache is a different feature from the same expression evaluated at request time, and the difference belongs in the contract (The Freshness SLO). Decide and document a late-data policy per feature: a cut-off after which the value is frozen, or an explicit statement that historical values are recomputed. Both are defensible; leaving it undecided is what makes offline datasets irreproducible (Late-Arriving Data).
Run a parity check in production: recompute a sample of served values through the offline path and compare, continuously rather than once at launch. Skew is not a bug you fix, it is a drift you monitor (Data Tests).
Backfill features with the same code that computes them going forward, from immutable raw events, so history and present are produced by one implementation (Backfills, Keeping Raw History: The Recovery Position and the Liability).

What this actually promises

Naming the guarantee you do not have is worth more than naming the one you do — everything downstream inherits the weakest promise in the chain.

One definition compiled to two paths guarantees the *expression* is the same. It does not guarantee the inputs are: different freshness, different late-data handling and different null defaults produce different values from identical logic (Data Contracts).
An as-of join guarantees a row describes the state at its event time only to the extent the underlying table records validity intervals. A dimension holding current state alone cannot support the operation at all (Slowly Changing Dimensions).
Logged-and-reused features guarantee exact agreement between what was served and what is trained on, and guarantee nothing for entities that were never served — there is no history before logging began (Missing Rows).
A batch feature value is a claim about a period as of a stated moment. Without that qualifier it is not reproducible, and re-running the job later is expected to produce something different (Reprocessing vs Retrying).
Nothing guarantees that a feature means what its name says. days_since_last_order computed from order creation and from order settlement are both correct, differently, and no type system distinguishes them (Semantic Changes).

Can I trust it?

A green pipeline is evidence that code ran. These four fields are the evidence that the data is right.

The check that would catch this

The check that earns its place first is a parity check: sample served requests, recompute the same features through the offline path with the same as-of timestamp, and compare value by value. Anything beyond a stated tolerance is skew, and it should page (Reconciliation).
It misses skew on entities or segments that are rarely served, because the sample follows production traffic. A feature that is wrong only for a small cohort passes a parity check comfortably (Data Skew).
Pair it with a distribution comparison between offline feature values and served values over the same period — null rates, ranges, means per segment. It catches systematic differences the point-wise sample misses, and it is the check that finds a unit or timezone difference (Distribution Tests).
Both miss a feature that is computed identically on both paths and defined incorrectly. Only someone who knows the domain catches "days since last order" counting cancelled orders (Two Dashboards, Two Numbers).

Freshness

Every served feature has a freshness that is part of its definition rather than an implementation detail. A cached seven-day count and a live seven-day count are different columns, and only one of them can be reproduced offline without knowing the cache policy (TTL and Expiry).
The offline path usually assumes values as of the event instant, which is the most optimistic possible assumption and rarely what the serving path actually did. Reconciling that assumption with reality is where most quiet skew lives.
Late-arriving events mean recent periods keep changing for a while. A feature computed over a window that is still receiving data is provisional, and offline datasets built from provisional windows are not reproducible (Late-Arriving Data).

When the schema or meaning changes

Changing a feature's computation is a semantic change to a column that keeps its name. Version it, write the version on the rows, and treat values before and after as different columns for every comparison (Semantic Changes, Breaking Schema Changes).
Adding a feature is additive and applies only forward unless it can be backfilled from raw events. Whether it can is decided by whether the raw events were retained, which is a decision made long before anyone wanted the feature (Keeping Raw History: The Recovery Position and the Liability).
A change to the underlying source — a new order status, a redefined timestamp — changes feature values with no schema change anywhere, which is the ordinary silent breakage this domain is built around (Data Contracts).
Removing a feature is a breaking change to any consumer reading it, and consumers of a feature table are frequently not enumerable without lineage (Impact Analysis).

How to re-run this safely

Features are derived, so they are rebuildable from raw events with the same code — provided the events were retained and the transformation is deterministic in the strict sense: no now(), no current-state joins, no dependence on when the job runs (Reprocessing vs Retrying, Determinism: Same Input, Same Output?).
Values that were only ever computed inside a serving path and never logged are not recoverable. That asymmetry is a strong argument for logging served values even when they are also computed offline (The Event Log).
A backfill of corrected features must write to a new location and be validated against the old before publishing, exactly like any other backfill — and the validation should include a parity check against what was actually served (Planning a Backfill, Validating a Backfill Before You Publish).

What can go wrong

Failure modes

Two implementations of one definition drifting apart after a fix is applied to only one (The Dual Write Problem).
A join against current state in the offline path, so historical rows carry today's attributes (Slowly Changing Dimensions). A window that includes events occurring after the moment the row claims to describe (Event Time).
Different null and default handling between paths, which produces a difference concentrated exactly on the sparse entities (Nullability & Defaults).
A stale cache on the serving path that the offline path has no model of (TTL and Expiry).
Late-arriving events silently changing historical feature values, making offline datasets irreproducible (Late-Arriving Data).
The mitigation failing: a parity check that recomputes offline values with the *current* timestamp rather than the request's as-of timestamp, which reports skew on every fresh feature and is muted within a week (Alert Fatigue: The Page Nobody Reads).

Misreads

"Training/serving skew is a modelling problem." It is two pipelines computing one column and disagreeing. Everything about diagnosing and fixing it — definitions, versions, as-of joins, parity checks — is data engineering (The Dual Write Problem).
"We use the same SQL, so there is no skew." Same expression, different inputs: a cache on one side, a different late-data policy on the other, a different null default. Skew is a property of the values, not of the source code (Reconciliation).
"Point-in-time correctness means adding a timestamp column." It means joining to the value that was valid at that timestamp. A timestamp beside a current-state join is a row that documents its own incorrectness (SCD Type 2 in Practice).
"A feature store removes skew." It provides one place to define and two places to materialise. If two teams define the same concept differently inside it, it stores the disagreement faithfully (The Metrics Layer).
"The offline numbers were better, so the pipeline is fine." Offline numbers computed from rows containing information from after the event are expected to be better, and cannot be reproduced by any serving path (Event Time).

Privacy, retention and access

Features are derived from personal data and frequently remain personal data: a vector of behavioural counts keyed by customer identifies that customer as surely as their email does (Data Classification, PII in Pipelines).
Deletion has to reach the offline feature table, the online store, any logged served values, and any dataset assembled from them. Four places, and the online store is the one most likely to be treated as a cache and forgotten (Deletion Requests).
Features can encode attributes nobody intended to use — a postcode-derived aggregate carries more than location. That is a classification question at definition time, not a review after the fact (Data Minimization).

Operating it

How you see it in production

Parity: sampled served values versus recomputed offline values, per feature, as a continuous signal rather than a launch-day test (Reconciliation).
Null rate per feature on both paths, side by side. Divergence in null rate is the earliest and cheapest skew signal there is (The Dimensions of Data Quality).
Served feature freshness — the age of the underlying data behind each served value — measured on the serving path rather than assumed from the cache configuration (Freshness Monitoring).
Distribution of each feature over time on both paths, so a definition change or an upstream source change appears as a step rather than as a mysterious behaviour change (Distribution Tests, "What Changed?" — Deploy Markers and the Invisible Deploys).
Late-event volume per feature window, because it is the direct measure of how provisional recent values are (Volume Anomalies).
Definition version present on every row, and a count of distinct versions in any dataset assembled for offline use. More than one without an explicit reason is a bug (Data Tests).

What changes at 10x and 100x

At ten times the entities, the as-of join becomes the expensive part of building any historical dataset, and layout — partition by date, cluster by entity — turns it from a full scan into a bounded one (Partitioning, Partition Pruning).
At a hundred times, offline assembly becomes incremental: compute new feature rows as events arrive rather than recomputing history, with all the watermark and late-data questions that come with it (Incremental Processing, The High-Water Mark).
Entity cardinality drives the online store rather than data volume: hot keys and skew on the serving path behave like any other keyed workload (Hot Keys: When Aggregate Metrics Hide a Saturated Node, Data Skew).
Feature count scales the governance problem. Fifty features with owners and definitions are manageable; five hundred computed by several teams reproduce the metric-definition problem exactly, with the same fix (The Metrics Layer, Data Discovery).

What drives cost here

The dominant drivers are the ordinary ones: bytes scanned to assemble historical rows, bytes retained for feature history, and repeated computation when features are recomputed rather than materialised once and read twice (What Actually Drives Data Platform Cost).
As-of joins are expensive by nature — they are range joins over validity intervals — and their cost is driven by entity cardinality and by how many historical rows are being assembled. Partitioning and sorting by entity and time is what keeps them tractable (Clustering and Sort Order).
Serving cost is dominated by point-lookup volume and by keeping the online store warm, which is a completely different shape from the offline scan. Sizing one from the other is a common and expensive mistake (OLTP Workloads).
Logging served feature values costs retained bytes proportional to traffic and removes an entire class of recomputation. It is usually the cheaper side of the trade (Storage Lifecycle).

What this approach costs

One definition compiled to two paths costs a build step and a constrained expression language, and buys agreement by construction. Two hand-written implementations are faster to write and are a standing commitment to keeping two codebases equal forever.
Logging served values removes skew for everything it covers and costs retained bytes proportional to traffic, plus a bootstrap problem: nothing exists before logging started, so the first dataset still has to be assembled offline.
Freezing feature values after a late-data cut-off makes datasets reproducible and makes them slightly wrong for entities whose events arrived late. Recomputing keeps them accurate and makes every offline result unreproducible. Pick one deliberately and write it down.

Dataset review questions

This lesson uses the shared review exercise.

The questions this domain asks of every dataset. Answer each one for the data this lesson is about — a question you cannot answer is the finding.

What does one row of this table mean? Say it in a sentence with no "and" in it.Which decision does this dataset drive, and what does being wrong cost that decision?Where did each column come from, and which system is the authority for it?How fresh is it, how fresh does the consumer believe it is, and who wrote that number down?If this pipeline runs twice over the same range, is the result the same?Which check would fail if this were silently wrong — and if the answer is none, what would you add?What happens to this table when the upstream schema changes without warning?Who is paged when it breaks, and who finds out first — them or the consumer?

0 of 8 answered.

Where this applies

Almost nothing here is universal. These labels say what each claim is specific to, and where a different engine, format, warehouse or scale would differ.

GENERALThat two materialisations of one definition are fine and two definitions are not, and that historical rows need an as-of join, hold for any stack and any consumer of features. What differs is which store serves the online path and how the definition is shared between the two, not the failure being prevented.
SCALE-SPECIFICFor a small entity count and relaxed latency, the serving path can query the same warehouse the offline path uses, and skew is structurally impossible because there is one pipeline. The second store — and with it the entire skew problem — appears only when a point lookup inside a request stops being affordable against analytical storage.
SIMPLIFIEDDrawn as one offline path and one online path over one feature set. Real deployments commonly have streaming features with second-level freshness beside batch features refreshed daily, which multiplies the freshness statements each feature needs rather than changing what parity means.

Where the depth lives

This domain teaches how data moves and how you know it arrived intact. It hands the rest off by name.

Databasedenormalization index-types caching-patterns window-functions joins

Backenddual-write-problem cache-aside cache-ttl search-index-sync eventual-consistency-in-backends transactional-outbox

API Designconsistency-expectations

Securitydata-classification least-privilege

Observabilitycache-performance-signals hot-keys cardinality deployment-markers alert-fatigue

Cloudmanaged-databases object-storage

Concurrencydeterminism

Domains that do not exist yet

— Distributed Systems owns why two materialisations of one definition can never be kept in step by writing to both — the dual-write problem, and the log-based patterns that replace it with a single write everything else reads.
— DevOps / Production Engineering owns the rollout of a feature definition change across two paths that deploy on different cadences, and the window in between where one side is computing the new definition and the other is not.