Would a GPU make our inference faster, and what is the risk of quantizing the model to INT8?
Answer it out loud before you open anything. The value of the flags below is in comparing them to what you actually said — including whether you asked about the data before naming a model.
A team serves a gradient-boosted risk model and a small transformer for text classification from the same CPU fleet. Latency complaints are about the transformer at p99. An infrastructure proposal moves both to GPU instances at roughly triple the cost and separately proposes INT8 quantization "for free speed".
React to this
Say what you would question, what you would trust, and what you would need to know first.
Serving fleet snapshot (illustrative) model type p50 p99 req/s cpu% risk_gbm_v12 boosted trees 0.4ms 1.1ms 820 18 intent_tx_v3 transformer 66M 38ms 310ms 26 71 Proposal A: move both to GPU instances, ~3x fleet cost Proposal B: INT8 quantize intent_tx_v3, "no accuracy impact expected" Note: p99 for intent_tx_v3 spikes at the top of each hour with the cron-driven backfill.
What it is really testing
Whether the candidate knows what a GPU is good at — large, parallel, batchable matrix work — and can therefore predict which of the two models benefits, and whether they understand quantization as a trade of precision for speed and memory whose cost must be measured on the model's actual metric, on slices.