MAPE and Its Caveats
Mean absolute percentage error reads naturally and fails badly: undefined at zero, dominated by small actuals, and asymmetric between over- and under-forecasting. On a demand forecast it blows up on exactly the low-volume items nobody was worried about.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Your demand forecast reports a MAPE that looks poor, but the high-volume products — the ones that matter for revenue — are forecast well. Where is the number coming from, and what is it actually rewarding?
A supply-chain team reports forecast accuracy as MAPE because finance wants a percentage. The number is dominated by a long tail of products that sell one or two units a week, and the model that scores best on it is one that consistently forecasts low. The team is being asked why their "accuracy" is bad while the warehouse is running fine.
Report MAPE. It is a percentage, everyone understands it, it is scale-free so products of different volume can be pooled, and finance already tracks it.
A SKU-week with an actual of zero has an undefined percentage error. The implementation either drops it — silently removing the rows where under-forecasting a stock-out matters — or adds a small constant, which turns the error into an arbitrary large number.
- A SKU-week with an actual of zero has an undefined percentage error. The implementation either drops it — silently removing the rows where under-forecasting a stock-out matters — or adds a small constant, which turns the error into an arbitrary large number.
- The tail SKUs contribute enormous percentage errors on tiny quantities. A miss of one unit on an actual of one is a hundred percent; a miss of a hundred units on an actual of ten thousand is one percent. MAPE ranks the first as a hundred times worse.
- MAPE penalises over-forecasting more than under-forecasting: the denominator is the actual, so forecasting high on a small actual is unbounded while forecasting zero on any actual is capped at a hundred percent. A model that minimises MAPE learns to forecast low, and the warehouse runs out of the very things it was forecasting.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict weekly unit demand per SKU. The label is recorded sales, which is zero in many SKU-weeks — not because demand was zero but because the item was slow, out of stock, or not on the shelf.
- The forecast drives a purchase order, so the cost of a miss is asymmetric: an over-forecast holds inventory, an under-forecast loses a sale and possibly the customer.
- One example is one SKU-week. Thousands of SKUs, most of them slow: a large fraction of rows have actual sales of zero, one or two.
- The high-volume SKUs — a few hundred — account for most revenue. They are well-behaved series with small percentage errors. The tail SKUs are noisy by nature; a forecast of two against an actual of one is a hundred percent error.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- MAPE is the mean over rows of
|actual − forecast| / |actual|. Every row is normalised by its own actual, so each row's influence is inversely proportional to its size. The metric is not scale-free; it is scale-inverted. - The asymmetry comes from the same denominator. For an actual of ten, forecasting twenty is a hundred percent error and forecasting zero is also a hundred percent — but forecasting thirty is two hundred, and there is no under-forecast that can exceed a hundred. The metric is bounded on one side and unbounded on the other, and an optimiser goes to the bounded side.
- Zero actuals make the row undefined. Common fixes — drop, add epsilon, use symmetric MAPE — each change what is measured, and none of them recovers the information that the model under-forecast an item that then sold out.
Where the number comes from
Split the MAPE by volume band and the report explains itself. The top band — the SKUs that carry the revenue — has a small percentage error. The bottom band, where actuals are one or two units, has errors of a hundred percent and above on nearly every row, because there is no forecast of a one-unit item that is not a large percentage off. The average of a few hundred small numbers and several thousand large ones is a large number.
The forecast was never bad where it mattered. The metric was computing an average over rows weighted by the reciprocal of their size, which is the opposite of the weighting the business has.
1import numpy as np2 3def mape(y, f):4 m = y != 0 # rows with zero actuals are undefined5 return np.mean(np.abs(y[m] - f[m]) / np.abs(y[m])), (~m).sum()6 7def wape(y, f):8 return np.sum(np.abs(y - f)) / np.sum(np.abs(y)) # defined; head-weighted9 10# one head SKU, many tail SKUs, all forecast reasonably11y = np.concatenate([[10_000], np.ones(500)])12f = np.concatenate([[10_100], np.full(500, 2.0)]) # 1% off on the head, 1 unit on the tail13 14print(mape(y, f)) # dominated by 500 rows of 100% error, plus a count of dropped zeros15print(wape(y, f)) # dominated by the head SKU, where the units areThe second return value of mape is the number of rows silently removed. Print it. Those are the stock-out weeks, and the metric was computed without them.
The direction the metric pushes
A model tuned to minimise MAPE discovers that under-forecasting is cheap. Forecast zero and the row costs a hundred percent, whatever the actual was; forecast high on a small actual and the row costs far more. Across a long tail, the cheapest policy is to forecast low on everything uncertain. The model is not malfunctioning; it is minimising what it was asked to minimise.
In a warehouse the expensive miss is usually the under-forecast — the empty shelf, the lost sale. So the metric's accidental asymmetry points in the opposite direction to the business cost, and improving the metric makes the business worse.
looks like A standard percentage error, tracked in a dashboard, improving quarter on quarter after the model was tuned on it.
why it leaks It does not leak; it *steers*. The bounded under-forecast side and the unbounded over-forecast side make forecasting low the cheapest policy on uncertain rows, so the optimiser drifts toward under-forecasting regardless of what the business wants.
fix Evaluate with a metric whose asymmetry matches the business cost — a weighted APE or a quantile loss with the under-forecast penalised — and monitor stock-out rate as the online outcome.
A metric with an aggregation level
Every forecast metric is computed at some grain, and the grain decides what the number means. At SKU-week, zeros and tail noise dominate. At category-month, the zeros vanish and the percentage error is small and stable — and if purchase orders are placed per category, that is the honest level. If they are placed per SKU, the category number is describing a decision nobody makes.
The assumption behind any reported MAPE is that the grain and the weighting match the decision. When they do not, the number is true and irrelevant.
The forecast is evaluated at the level of aggregation and with the row weighting at which the purchase decision is made, so a change in the metric corresponds to a change in the cost of the orders placed.
holds when Orders are placed per SKU and the metric weights each SKU by its volume or its miss cost; or orders are placed per category and the metric is computed at category level.
breaks when A dashboard reports unweighted SKU-level MAPE for decisions made per category, or category-level error for shelf allocation made per SKU; or the catalogue's tail grows and the unweighted number drifts with it.
respond Change the report before changing the model. Recompute at the decision grain with the decision weighting, and only then judge whether the forecast needs work.
How to build it
Most important first.
- Match the metric to the aggregation the decision uses. If purchase orders are placed per SKU, weight errors by the cost of the SKU's miss; if they are planned per category, evaluate at category level where the zeros disappear (Choosing a Regression Metric).
- Prefer a weighted absolute percentage error — the sum of absolute errors over the sum of actuals — when a percentage is required. It is dominated by the high-volume items, which is where the money is, and it is defined when individual actuals are zero.
- Report MAE in units per volume band. The tail SKUs get their own number and stop polluting the head; the head number is the one finance should be looking at.
- If under- and over-forecasting cost differently, say so in the metric: a quantile loss with the asymmetry the business actually has, rather than a percentage that has an accidental asymmetry in the other direction (Forecast Evaluation).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Weighted APE at the level the purchase order is placed, so the number is dominated by the items whose misses cost money.
- Stock-out rate and holding cost on the delayed actuals — the two business outcomes the forecast is supposed to move, in opposite directions.
- Do not report unweighted MAPE across a catalogue with a long tail. It measures the tail, and it rewards forecasting low.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The evaluation is not dominated by rows with tiny actuals — the volume distribution of the evaluation set matches the volume distribution the business cares about.
- Zero and near-zero actuals mean low demand, not censored demand. Where they mean stock-outs, the metric is computed on a label that is wrong in exactly the direction the metric cannot see.
- The direction of the metric's asymmetry does not conflict with the direction of the business cost. MAPE's bias toward under-forecasting must not be optimised in a business where the under-forecast is the expensive miss.
- Offline: compute MAPE per volume band. If the overall number is driven by the lowest band, the report is describing the tail. Compare model rankings under MAPE and under a weighted metric; a flip is the asymmetry at work.
- Online: track stock-out rate and days of inventory alongside the forecast metric. A forecast metric improving while stock-outs rise is the low bias being rewarded.
- Over time: audit which rows the evaluation drops or clips for zero actuals, and count them; a growing count is censoring growing.
What can go wrong
- The team switches to a weighted metric and the tail is now invisible. A new product that the model forecasts at zero for months never registers on any report (Cold Start).
- Zero actuals are dropped from the evaluation. Stock-outs — the rows where the model under-forecast and the shelf emptied — are precisely the ones removed, so the metric is computed on the weeks the forecast did not fail.
- The model is tuned to minimise MAPE and learns the low bias. Recorded sales fall because the shelf is empty, the actuals fall, and next quarter's MAPE improves (Feedback Loops).
- Weighted metrics hide the tail, which is where new products and slow movers live; a separate report for the tail is required and is one more thing to maintain.
- A cost-weighted or quantile loss is honest about the asymmetry and opaque to finance, who asked for a percentage.
- Evaluating at the aggregation level of the decision means the model may be right in aggregate and wrong per SKU, which is fine for the purchase order and not for shelf allocation.
- "MAPE is scale-free, so we can pool all products into one number." Each row is normalised by its own actual, so small actuals dominate the pooled number. It is scale-inverted, not scale-free.
- "Our forecast accuracy is poor." The unweighted MAPE is poor. Compute it on the SKUs that carry the revenue and the answer is usually different — the number was describing rows that do not matter to the decision.
- "Use symmetric MAPE, it fixes the asymmetry." sMAPE changes the asymmetry rather than removing it — it is still undefined when both actual and forecast are zero, and it penalises under-forecasts more than MAPE does. It is a different metric, not a corrected one.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe denominator behaviour — undefined at zero, dominated by small actuals, bounded for under-forecasts and unbounded for over-forecasts — follows from the formula and applies to any forecast, in any domain.
- DATA-SPECIFICOn a catalogue of uniformly high-volume items with no zeros, MAPE behaves reasonably and reads well; the caveats become failures as the volume distribution acquires a long tail, which is most retail, most parts inventories and most web traffic.
- SIMPLIFIEDThe percentage errors quoted are for the shape of the argument — the point is the ratio between a one-unit miss on a one-unit actual and a hundred-unit miss on ten thousand, not the specific figures.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Operations research — safety stock, service level and the newsvendor model are where the asymmetric cost of a forecast miss is formalised; this lesson stops at the point where the metric should encode that asymmetry.