Serving & Train/Serve Skew
Identical weights can fail if the features differ. Skew, feature stores as optional infrastructure, point-in-time correctness, freshness, latency breakdown and fallbacks.
The weights are identical in training and production. The features are not. A model can be exactly right about inputs it will never see again.
A feature store is optional infrastructure that makes one feature definition serve both training and low-latency inference, with lineage attached. It is one answer to skew, not a prerequisite for ML.
A training example at time T may only use information that existed at T. The as-of join is how you build that, and the offline store exists to make it cheap.
Features update in seconds, minutes, hours or days. The model was trained on values of a particular age, and the serving architecture must deliver the same age or the model is reading a different signal.
A prediction request is parsing, feature fetch, preprocessing, model compute, postprocessing and network. The model is rarely the slow part for tabular systems, and almost always is for large networks.
Throughput is predictions per unit time; latency is how long one waits. Batching raises the first by spending the second, and queue depth — not CPU — is the signal that says you are running out of both.
When the model or its features are unavailable, the system must return something defined: the previous model, a rule, a cached score, a default ranking, or an explicit "no prediction". Which one is a product decision.