Follow One Prediction
A user opens a SaaS dashboard on a Tuesday morning. Somewhere in the page load a model decides whether to show them a retention offer. Fourteen stops from the click to the row this request becomes in next quarter's training set.
At every stop the interesting question is not what happens but what this stage assumes about the stages either side of it. The feature cache assumes a freshness window; the artifact assumes a preprocessing contract; the threshold assumes the validation distribution it was chosen on; the feedback row assumes the outcome was observed rather than caused. The layer that looks like plumbing — the cache, the batching queue, the config value that holds the threshold — is usually where the model was actually decided, which is why every stop has a layer deeper.
User opens the product
A user on an annual plan, eleven months in, opens the dashboard on a Tuesday morning. Nothing about the page suggests a decision is being made about them; the retention offer, if it appears, will look like a banner.
- — The prediction timestamp, which fixes what "available at prediction time" means for every feature.
- — That this is a request-time decision, which is the choice that made online inference necessary — or, as the deeper text suggests, the choice that should have been questioned.
- — The users most at risk do not open the product, so the system never scores them and the retention offer reaches only the engaged.
- — The request is a bot, a support agent impersonating the user, or an integration — none of which the training data contained.
Take it further
What the prediction-time rule at stop one is protecting against — and what the offline number does when it is broken.
ML Failure Simulator →Any stop on this walk can fail silently. Read the signals and name the cause.
Capstone →Design the churn system this walk runs through, then absorb the failures nobody put in the brief.