ML System Design
The questions to ask before drawing boxes, then recommendation, fraud, churn and search ranking designed end to end — and the boundary with Agentic Engineering.
Sources → data platform → features → training → registry → serving → application → monitoring. Eight boxes, five owning teams, and four interfaces that decide whether the system can be reasoned about at all.
Ten questions — target, latency, batch or online, freshness, volume, label delay, model size, fallback, retraining, cost — each of which decides a part of the architecture before any model is chosen.
Events → candidate generation → features and embeddings → ranking → serving → feedback. A two-stage latency budget, a loop in which the model writes its own training data, and an offline metric that measures agreement with the previous policy.
Rare positives, a strict online latency budget, asymmetric costs, an adversary who adapts to the model, and labels that arrive ninety days late. Every constraint in the domain at once.
A weekly call list for a team of fixed capacity. Batch scoring, a threshold that is a queue size, labels a month late, explanations the callers can use — and a demonstration of why the online endpoint someone will propose is unnecessary.
Query → candidate retrieval → ranking model → results. Lexical and embedding retrieval, learning-to-rank from click labels that carry position bias, NDCG-style evaluation at concept level, a latency budget per stage, and interleaving for the online test.
ML Engineering owns the model: training, fine-tuning, evaluation, embeddings, inference, serving, drift, MLOps. Agentic Engineering owns what is built on top: prompting, RAG, tools, memory, agent architecture, agent evals. The line is where the depth lives, not where the LLM is.