Your data has a time dimension. How do you split it, and how do you cross-validate?
Answer it out loud before you open anything. The value of the flags below is in comparing them to what you actually said — including whether you asked about the data before naming a model.
An energy company forecasts hourly load per region 24 hours ahead. Three years of hourly data exist. The team's first model used five-fold cross-validation with shuffled folds and reported an error far better than the incumbent statistical forecast; the incumbent team says the comparison is unfair.
React to this
Say what you would question, what you would trust, and what you would need to know first.
Evaluation summary — hourly load model v1 (illustrative) data: 2021-01 .. 2023-12, hourly, 9 regions split: KFold(n_splits=5, shuffle=True, random_state=42) features: hour, dow, temp_forecast, load_lag_1h, load_lag_24h, load_lag_168h, rolling_mean_24h MAE: ours 41 MW incumbent 118 MW (-65%) Incumbent team: "our forecast is issued at 09:00 for the next day; yours saw the hour before."
What it is really testing
Whether the candidate knows that a random split on temporal data lets the model see the future, and whether they can describe a validation scheme that respects time — including the subtleties: a gap for the horizon, multiple folds that move forward, and the fact that the last fold is the most representative and the least reliable.