Dataset Construction
Filtering, joining, labelling and feature creation each introduce bias or leakage. What one training example represents, and how sampling decides what the model can learn.
Raw data becomes a dataset through filtering, joining, labelling and feature creation. Each stage is a decision, and each decision can introduce bias or leakage that no model can undo.
One row is one user, or one transaction, or one user-day, or one query-document pair. Choosing the grain decides the snapshot date, the label window, and what counts as a duplicate.
Random, stratified, temporal and group-based sampling each preserve a different property of the population. Which property matters depends on what the model will meet in production.
The dataset only contains the cases that reached the step where the label was recorded. Approved loans have repayment labels; declined ones do not. The model learns about the selected, and is deployed on everyone.
The table contains the customers, companies or machines that are still here. The ones that failed were deleted, archived or never joined, and the model learns what survivors look like.
When positives are one in a thousand, always predicting negative is almost perfectly accurate and completely useless. Imbalance decides the metric, the split, the threshold, and whether the probabilities can be trusted.
The label is the thing the model is trained to reproduce. Noisy, delayed, drifting, disputed or machine-generated labels put a ceiling on everything downstream, and the ceiling is invisible in the metric.