TreesFRAMEWORK-SPECIFICSIMPLIFIEDDATA-SPECIFIC

XGBoost and LightGBM as Implementations

Second-order gradients, a regularised objective, histogram binning, leaf-wise growth, native missing-value and categorical handling — what the fast implementations add to boosting, at the level of the mechanism and never the API.

Target & data What to measure What must stay true

The problem, the obvious approach, and why it breaks

Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.

The question

What do the production boosting libraries do that the textbook algorithm does not, and which of those choices changes how the model overfits, handles missing values or scales?

The problem

A demand-forecasting team moved from a hand-rolled boosting loop to a production library and got a model that is faster, better on validation and different in ways they cannot explain — it handles missing values without imputation, it overfits harder on their smallest series, and its trees look nothing like the balanced ones they were used to.

The obvious approach

A library is a faster version of the same algorithm. Swap it in, keep the hyperparameters, expect the same model in less time.

Why it breaks

The library grows trees leaf-wise — always splitting the leaf with the largest gain — rather than level by level. With a depth limit that was tuned for level-wise trees, the leaf-wise model grows a few very deep branches into the small series and memorises them. Same nominal depth, different model, different overfitting.

How it breaks — usually after the offline metric looked fine

The library grows trees leaf-wise — always splitting the leaf with the largest gain — rather than level by level. With a depth limit that was tuned for level-wise trees, the leaf-wise model grows a few very deep branches into the small series and memorises them. Same nominal depth, different model, different overfitting.
Missing values are no longer imputed; the library learns a default direction per split. The team's imputed-zero for missing lags had been silently encoding "new SKU" and the model had used it; the native handling learns something else, and the SKU-launch forecasts change.
Histogram binning quantises each feature into a fixed number of bins before the split search. A price feature with a few dozen meaningful levels is fine; a high-resolution lag feature loses the fine cuts, and the change is invisible unless the binning is understood.

Problem→Target→Data→Representation→Split→Model→Training→Evaluation→Validation→Deployment→Inference→Monitoring→Drift→Retraining

What is being predicted, and from what data

This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.

Target

Predict daily units per SKU per store two weeks ahead. The label is the observed sales count, which is censored by stock-outs — a zero can mean no demand or no stock.
The decision is a replenishment order; the cost is asymmetric between overstock and stock-out, and the model's output feeds an inventory policy rather than being the decision itself.

Data

One example is one SKU-store-day with lagged sales, calendar features, price, promotion flags, and weather. Tens of millions of rows; many features missing for many rows — a new SKU has no lags, a small store has no weather station.
Series vary enormously in volume: a few thousand rows for a slow SKU at a small store, millions for a fast one at a flagship.

How it actually works

Precisely enough to predict its behaviour — not a framework API.

Second-order gradients. Textbook boosting fits each tree to the negative gradient. The production objective takes a second-order Taylor expansion of the loss per row — gradient g_i and Hessian h_i — and chooses split gains and leaf values in closed form from their sums: leaf value −Σg / (Σh + λ), gain proportional to (Σg)² / (Σh + λ). The Hessian acts as a per-row weight and makes the step a Newton step rather than a gradient step, which converges in fewer rounds and behaves more consistently across losses.
Regularised objective. The objective includes λ on leaf values (an L2 term), α (L1), and γ per leaf — a minimum gain to split at all. These are the regularisation of the tree structure written into the same formula that scores splits, so depth limits and minimum child weight become terms of one objective rather than separate stopping rules (Regularisation).
Histogram binning and leaf-wise growth. Each feature is pre-binned into a few hundred buckets and the split search runs over bins, not values — far fewer candidates, cache-friendly, and the basis for parallel and distributed training. Growth is leaf-wise (best-first): expand the leaf with the largest gain anywhere in the tree, subject to a leaf-count limit, rather than completing each level. Leaf-wise finds gain faster and overfits small data faster; the leaf-count limit and minimum rows per leaf are the controls. Missing values are handled by learning, at each split, which branch missing rows should take. Categoricals are handled natively in some implementations by searching over partitions of the category set, ordered by gradient statistics, rather than one-hot.

The objective, with the second derivative in it

Textbook boosting fits the tree to −g, the negative gradient. The production objective expands the loss to second order around the current prediction: Σ [g_i f(x_i) + ½ h_i f(x_i)²] + Ω(f), with Ω the regularisation on the tree. For a fixed tree structure this is a quadratic in each leaf's value, so the optimal leaf value and the gain of any split have closed forms in terms of the sums of g and h over the rows in the leaf.

That closed form is what makes the implementation fast and consistent: no inner optimisation per leaf, the same formula for every differentiable loss, and the Hessian weighting rows so that a row the model is already confident about contributes little. λ appears in the denominator of both formulas, which is where "regularised objective" becomes concrete — a leaf with few rows or small Hessian sum is shrunk toward zero.

Leaf value and split gain from gradient and Hessian sums

1def leaf_value(g_sum, h_sum, lam):
2    # optimal constant for a leaf under the second-order objective
3    return -g_sum / (h_sum + lam)
4 
5def split_gain(gL, hL, gR, hR, lam, gamma):
6    # improvement from splitting a node into left/right, minus the per-leaf penalty
7    def score(g, h):
8        return g * g / (h + lam)
9    return 0.5 * (score(gL, hL) + score(gR, hR) - score(gL + gR, hL + hR)) - gamma
10 
11# for log loss on a row with label y and current probability p:
12#   g = p - y          (first derivative)
13#   h = p * (1 - p)    (second derivative: small when the model is already sure)

Read h = p(1 − p). A row the model already scores near 0 or 1 has a tiny Hessian and barely moves the leaf value; a row near 0.5 dominates. That is a per-row weighting the textbook gradient step does not have, and it is why the library converges in fewer rounds and why min_child_weight — a floor on Σh in a leaf — is a regulariser on *uncertain* rows rather than on row count.

Bins, leaves and the direction missing values go

Before any split search, each feature is quantised into a few hundred bins by quantiles. The search then runs over bin boundaries, which turns a sort-per-feature-per-node into a histogram-per-node and makes the whole thing cache-friendly and parallel. The cost is resolution: a cut that falls between two values in the same bin cannot be made. For most features nothing is lost; for a feature whose signal lives in fine differences, the bin count is a hyperparameter.

Leaf-wise growth changes the shape of the tree. Level-wise completes each depth before starting the next and produces balanced trees; leaf-wise expands whichever leaf, anywhere, offers the largest gain, and produces lopsided trees that go deep where the gain is. On large data that is more model per tree; on a small series it is a branch that follows a handful of rows to the bottom. Missing values get their own rule: at each split, the rows with a missing value are tried on the left and on the right, and whichever gives more gain becomes the default direction, learned per split from training-time missingness.

What each implementation choice can do in production

Trigger	Symptom	Cause	Response
Histogram binning	Fine cuts on a lag feature vanish; forecasts coarsen for high-frequency SKUs	Meaningful thresholds fell inside a single bin	Raise the bin count for that feature or transform it so the cuts land on bin edges
Leaf-wise growth	Small series overfit; aggregate validation improves	Deep branches isolate a few rows in low-volume slices	Cap leaves and set minimum rows per leaf; evaluate per volume tier
Learned missing direction	New-SKU forecasts shift after the switch; sensor outages score oddly	Missingness meant "new" in training and "outage" in serving	Make the meaning explicit as a feature; monitor null patterns at the boundary
Native categorical partitions	Rare families memorised; great training curve	Partition search isolated small category groups	Minimum rows per partition; group rare levels; permutation-check the feature

Same depth setting, two growth strategies

Level-wise, max_depth = 6

Every branch reaches depth six; at most 64 leaves, roughly balanced. A slow SKU's rows share leaves with similar rows from other series.

Leaf-wise, num_leaves = 64

Sixty-four leaves placed wherever gain is largest; some branches are depth two, some depth fifteen. A slow SKU with a few sharply different days can get its own deep branch.

Neither is better in general. Leaf-wise finds more gain per leaf on data that supports deep branches and memorises small slices on data that does not. The two settings that look equivalent — 2⁶ leaves — are different models, and the team's "same hyperparameters" were not the same.

What the black box assumes about the data

The library fitted three things the textbook loop did not: bin edges per feature, a default direction per split for missing values, and category partitions. All three are artefacts of the training data's distribution and all three are inside the model, where a schema check does not see them. They are the assumptions the model now carries.

The cheapest of these to break is missingness. The model learned that a missing weather value goes left because in training, missing weather meant a small store; the day the weather feed fails for every store, every row goes left, and the forecasts move for a reason no feature-distribution monitor will name unless it watches null rates.

must stay trueMissing means what it meant

The pattern of missing values at serving time has the same causes and the same relationship to the label as it had in the training table, because the model learned a per-split routing for missing rows from that relationship.

holds when Missingness is structural (new SKUs have no lags, small stores have no weather) and the structure is unchanged; feeds are as reliable in production as they were in the history.

breaks when An upstream feed fails or is paused, a new integration back-fills a previously missing column, or the training table was built from a cleaned source with fewer nulls than the live one.

how you would know Per-feature null rate at the serving boundary compared to the training table, alerting on a step change; forecast error on the high-null slice against its validation counterpart (Data & Feature Tests).

respond Treat a null-rate step as a serving incident, not a model one — fix or fall back on the feed — and add an explicit missingness feature before the next retrain so the semantics are visible.

How to build it

Most important first.

Retune when changing implementation. The knobs have the same names and different meanings: max_depth under leaf-wise growth is not the depth limit it was under level-wise, and a leaf-count limit is the primary control (Hyperparameters).
Decide what missing means before letting the library decide: if a missing lag means "new SKU", make that an explicit feature so the model learns it deliberately rather than via a default direction (Missing Data).
Check bin resolution against features that carry fine structure; raise the bin count for those or transform them so the meaningful cuts survive binning.
Treat native categorical handling as a strong but overfit-prone tool on high-cardinality features: it will find gain in the category set by search, and minimum rows per category-partition is the guard (Categorical Encoding).

What to measure

Which number actually maps to the decision — and which numbers look relevant and are not.

Validation error under a time-based, series-grouped split, per volume tier — the small series are where leaf-wise growth overfits and the aggregate number hides it (Evaluation Slices).
Forecast change on new-SKU rows before and after the switch: the missing-value semantics changed and this slice is where it shows.
Training time and memory as promotion criteria. The library's speed is why it was adopted; measure it against the retraining cadence, not in the abstract.

What must stay true after deployment

The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.

Assumptions

Missingness has the same meaning at serving time as in training, since the model learned a per-split default direction for it rather than a value.
The feature distributions stay inside the bin edges computed at training time; values beyond the last bin edge fall into the edge bin and carry no distinction.
The category vocabulary is stable enough that native categorical partitions remain meaningful; a new category value takes a default path the model never chose deliberately.

How to verify — offline, online, and over time

Offline: per-volume-tier validation error; a null-pattern comparison between training and serving inputs; a check of which features are bin-limited.
Online: forecast error on the new-SKU slice and on rows with unusual null patterns, against the corresponding validation slices.
Over time: re-examine bin edges and categorical vocabularies at each retrain; both are fitted artefacts that drift with the data.

What can go wrong

Failure modes in production

The default direction for missing values is learned from training rows where missingness meant one thing (small store, no weather) and applied at serving time where it means another (sensor outage) — a skew that no schema check catches (Train / Serve Skew).
Leaf-wise growth on the smallest series produces branches that memorise a handful of days; the aggregate validation number improves because the large series dominate it.
A categorical with thousands of SKU-family values is handled natively, the partition search finds gain by isolating rare families, and the model memorises them exactly as one-hot would have — with a better-looking training curve.

What the recommended approach costs

Speed and scale come from binning and best-first growth, both of which change the model that is fitted; the fast library is not the textbook algorithm run faster.
Native missing-value handling removes an imputation step from the pipeline and moves its semantics inside the model, where they are harder to inspect and easier to skew.
The regularised objective gives more knobs, each of which is a validation-tuned hyperparameter with interactions the team must learn; the defaults are tuned for benchmark data.

Misreads

"XGBoost and LightGBM are the same thing." They implement the same objective with different defaults for growth (level-wise vs leaf-wise historically), binning and categorical handling; the same settings produce different models.
"The library handles missing values, so we no longer need to think about them." It learns a direction per split from training-time missingness. If missingness means something different in production, it is a skew the library cannot see.
"Faster training means we can just add more rounds." Rounds are capacity; the round count is still chosen by early stopping on validation, however cheap each round has become.

Where this applies

ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.

FRAMEWORK-SPECIFICWhich growth strategy is the default, whether categoricals are handled natively, and how missing values are routed differ between XGBoost, LightGBM and CatBoost and between versions of each; the mechanisms described here are stable, the defaults are not, and this lesson stays at the mechanism.
SIMPLIFIEDThe objective is presented with the leaf-value and gain formulas and without the full derivation, the sparsity-aware split search, or the distributed training design; the team's before/after behaviour is illustrative of the mechanisms, not a measurement.
DATA-SPECIFICLeaf-wise growth is a clear win on large datasets where the deepest branches are well supported and a clear hazard on small ones where they isolate a few rows; the same setting is right and wrong depending on rows per leaf.

Where the depth lives

This domain teaches the model and hands the rest off by name.

Data Engineeringdata-quality

Computer Architecturecache-fundamentals