Dataset Construction
Raw data becomes a dataset through filtering, joining, labelling and feature creation. Each stage is a decision, and each decision can introduce bias or leakage that no model can undo.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The model is trained on "the data". Which pipeline produced that data, what did each stage decide, and where could the answer or a bias have entered?
A subscription business wants to know which customers will cancel next month so the retention team can call them first. An analyst has already built "the churn table" in the warehouse; the ML team is asked to train on it.
The table has features and a label column. Train on it, evaluate on a held-out slice, and ship the model that scores best. The analyst already did the hard part.
The label came from the *current* status, and the usage features were computed up to the same day. A cancelled customer has zero usage in the last 30 days because they left; the model learns that low usage predicts churn, which is the answer restated, and validation looks superb.
- The label came from the *current* status, and the usage features were computed up to the same day. A cancelled customer has zero usage in the last 30 days because they left; the model learns that low usage predicts churn, which is the answer restated, and validation looks superb.
- The filter to "customers with at least one login" removed the customers who never onboarded — who churn at the highest rate — so the deployed model has never seen the population the retention team most wants to call.
- Joining billing on
customer_idduplicated every customer with two payment methods; those rows landed in both train and test, and the metric measured memorisation of duplicates. - None of this raised an error. The pipeline ran, the schema validated, and the model was promoted on a number that described nothing the product would ever see.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether an active subscriber cancels within the 30 days following a snapshot date. The label is the presence of a cancellation event inside that window.
- The decision downstream is a ranked call list for a team that can reach a few hundred customers a week, so the output is a probability that a cutoff turns into a queue.
- Raw sources: a subscriptions table with status changes, a product event stream, a billing table and a support-ticket table, each owned by a different team with its own timestamps and its own idea of what a "customer" is.
- The churn table was built by filtering to customers with at least one login, joining the four sources on
customer_id, labelling from the current subscription status, and computing usage features over "the last 30 days" relative to the day the table was built. - One row is one customer as of the day the analyst ran the query. There is no snapshot date column.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A dataset is produced by a chain: raw data → filtering → joining → labelling → feature creation → dataset. Each arrow is a decision. Filtering chooses the population; joining chooses which entity is a row and what can be duplicated; labelling chooses when and how the answer is observed; feature creation chooses which moment in time each column describes.
- Bias enters when a stage systematically excludes or over-represents part of the population the model will meet in production. Leakage enters when a stage lets information that will not exist at prediction time — the future, the label, or a derivative of it — into a feature.
- The model cannot see the pipeline. It sees rows. Whatever the pipeline decided becomes, from the model's point of view, a fact about the world, and the offline metric faithfully measures how well the model learned that fact.
Five arrows, five decisions
The churn table looks like a fact. It is the output of a pipeline, and each stage made a choice the analyst may not remember making. The filter chose the population. The join chose the grain and what could be duplicated. The label chose which moment counts as the answer. The features chose which moment each column describes.
Draw the chain and annotate what each arrow can do to the model. The point is not that any stage is wrong; it is that each one can be, and the model has no way to tell.
- 1Raw data
Subscriptions, product events, billing and support tickets, each with its own timestamps and its own notion of a customer.
fails by Sources that already reflect a previous model's decisions, or that only record customers who reached some step.
- 2Filtering
Choose which entities become rows — active customers, customers with a login, customers in supported countries.
fails by Selection bias: the excluded segment is exactly the one the product will score, or the one that churns most.
- 3Joining
Attach the sources to one row per customer on a shared key.
fails by Duplicated entities from one-to-many joins; a current-value dimension table that describes the future for a past row.
- 4Labelling
Decide what event, in what window after the snapshot, counts as the positive class.
fails by A label taken from the current status, so it reflects events after the features were computed.
- 5Feature creation
Aggregate behaviour up to the snapshot date into columns.
fails by A window that ends after the snapshot; an aggregate over "the last 30 days" relative to build time rather than to the row.
- 6Dataset
Rows, one per customer-snapshot, with features and a label.
fails by Looking exactly like a clean table regardless of which of the above went wrong.
The last row is the lesson. A leaking or biased dataset has the same schema, the same row count and the same validation-friendly shape as a correct one.
Where the answer got in
The churn table's label is the current subscription status. Its usage features are computed over the 30 days before the table was built. For a customer who cancelled three weeks ago, those 30 days are mostly after the cancellation, and their usage is zero because they are gone — not because low usage foreshadows leaving.
The model learns that near-zero recent usage is churn, and it is right, on this table. In production it scores active customers whose recent usage is genuinely recent, and the pattern it learned describes people who have already left. The retention team calls the wrong people and the validation number stays excellent.
looks like A sensible engagement aggregate: count of product events in the 30 days before the row was created.
why it leaks The row's label was read from the current status, so for churned customers most of the 30-day window falls after they left. The feature encodes the consequence of the label rather than a cause of it.
fix Add a snapshot date to every row. Compute events_last_30d over the 30 days before the snapshot and the label from the 30 days after it, and rebuild the table for many snapshot dates.
The population the model has never met
The filter "at least one login" was there to remove junk accounts. It also removed everyone who paid and never onboarded, who cancel at the highest rate of any segment. The model was never shown them, and it scores them as if they were ordinary quiet users.
This is why the pipeline needs to report what it removed, not just what it kept. The training population is an assumption about production, and it needs a monitor like any other.
Every segment of customers the model scores in production is represented in the training set in roughly the proportion it appears in production, or the exclusion is deliberate and applied at serving time too.
holds when The pipeline's filters are mirrored in the serving path, and a row-count-by-segment report is compared against production traffic on every rebuild.
breaks when A filter removes a segment for data-hygiene reasons that is later scored anyway; the product changes who reaches the funnel step the filter keyed on; a new market launches with customers unlike any in the table.
respond Do not retrain on the same pipeline. Change the filter or add the serving-side exclusion, then rebuild, then retrain.
1WITH base AS (2 SELECT customer_id, country, has_logged_in3 FROM customers_as_of('2026-06-01')4),5after_login_filter AS (6 SELECT * FROM base WHERE has_logged_in7)8SELECT9 country,10 COUNT(*) AS before_filter,11 COUNT(*) FILTER (WHERE has_logged_in) AS after_filter,12 1.0 - COUNT(*) FILTER (WHERE has_logged_in) / COUNT(*) AS removed_share13FROM base14GROUP BY country15ORDER BY removed_share DESC;The interesting output is the top of the list: a country where the filter removes most customers is a market the model will not understand, and that is decided before a single weight is trained.
How to build it
Most important first.
- Write the dataset pipeline as code you own, with a snapshot date as an explicit parameter, so every row says *as of when* it describes the entity (What Is One Example?).
- Construct the label from events inside a window that starts after the snapshot, and compute every feature only from events before it (Label Construction, Temporal Leakage).
- Record what every filter removed — row counts before and after, by segment — and compare the surviving population to the production population the model will score (Selection Bias).
- Assert uniqueness of the row key after every join, and decide deliberately how a duplicated entity is handled (Entity Leakage).
- Version the dataset with the code and parameters that produced it, so a metric can be traced back to the exact pipeline (Dataset Versioning).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Row counts at every stage of the pipeline and the distribution of key segments before and after each filter. This is the number that reveals a bias; the validation metric cannot.
- The share of features whose computation window ends at or before the snapshot date — ideally all of them, checked by a test rather than by reading SQL.
- Validation AUC on this dataset tells you how learnable *this dataset* is. It says nothing about whether the dataset resembles production.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The population that survives the pipeline's filters is the population the model scores in production, segment by segment, and any deliberate exclusion is applied identically at serving time.
- Every feature in a row is computed from information that existed at that row's snapshot date, and the label is computed from events strictly after it.
- Each row key is unique, or duplication is handled by an explicit rule that the split respects.
- Offline: run the pipeline for a past snapshot date and confirm no feature value changes when events after that date are deleted from the sources.
- Online: compare the segment distribution of scored production customers against the training population weekly; a segment the model never saw should be visible before its scores are trusted.
- Over time: re-run the row-count-per-stage report on every rebuild and alert when a filter starts removing a materially different share of rows.
What can go wrong
- The snapshot parameter is added, but one feature is still computed from a slowly-changing dimension table that holds only the current value, so it quietly describes the future (Point-in-Time Correctness).
- The population check is done once, at build time; a product change later moves the onboarding funnel and the training population no longer matches who the model scores.
- The join is fixed, but an upstream team changes how
customer_idis assigned, and duplication comes back without any schema change to catch it.
- Owning the dataset pipeline as code means the ML team now maintains SQL and orchestration that an analyst used to run by hand, and it becomes a pipeline with its own failures (Pipeline Reliability).
- A snapshot-based dataset with a label window cannot use the most recent month of data, because those labels have not matured yet; the training set is always somewhat stale.
- Population checks and uniqueness assertions fail on legitimate changes too, and every false alarm erodes the habit of reading them.
- "The analyst's table is the source of truth, so training on it is safe." It is the source of truth for a dashboard that describes the present. A training set must describe the past as it was known at the time.
- "Validation was excellent, so the pipeline is fine." A leaking pipeline produces an excellent validation number by construction. The metric cannot detect the problem it is caused by.
- "We will fix the bias in the model with weights." Reweighting can correct a known, measured skew. It cannot invent the customers the filter removed.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat every stage of dataset construction can bias or leak follows from the fact that the model only sees rows, so it holds for any task, data modality and model family.
- DATA-SPECIFICThe snapshot-and-window structure is specific to entity-over-time data such as customers, accounts or machines; an image classification dataset has no snapshot date, but its filtering and labelling stages carry the same selection and label-quality risks.
Where the depth lives
This domain teaches the model and hands the rest off by name.