Why we do not trust one number
Every model so far - baselines, ETS, ARIMA, Prophet, gradient boosting - reported a single holdout MAPE. That number is a start, but it is one draw from a noisy process: land on a quiet fortnight and your model looks brilliant; land on a promo spike and it looks broken. Tonight we replace the single guess with a distribution of scores across many folds, add metrics that survive zeros and let you compare across series, and finally check that our prediction intervals are honest. This is the session that decides which model actually ships.
Rolling-origin cross-validation 6 min live
The fix for a noisy holdout is to use many of them, honestly. Rolling-origin cross-validation (also called time-series CV or walk-forward) picks an origin, trains on everything before it, forecasts the next H steps, scores that fold, then rolls the origin forward and repeats. Average the fold errors and you get a stable score with a spread that tells you how much to trust it. The train window can expand (keep all history) or slide (fixed length) - but it never, ever includes the future.
LiveExpanding vs sliding window, and how many folds3 min▶
Two dials, one rule:
- Expanding window keeps all history before the origin - best when the series is stable and more data helps.
- Sliding window keeps a fixed-length recent history - best when old behaviour is no longer relevant (a regime change).
- Forecast at your real horizon. Each test fold is exactly H steps, because that is the decision you actually make.
- Enough folds to see spread. A handful of folds already turns one number into a mean plus a range - which is the whole point.
Three metrics, and honest intervals 5 min live
One metric never tells the whole story. Use three: MAPE (percentage error, intuitive but breaks near zeros), RMSE (in the series' own units, punishes big misses hardest), and MASE - the mean absolute error scaled by the in-sample naive error. MASE is the honest one: MASE < 1 means you beat naive, MASE > 1 means you did not, and it compares cleanly across series of different scales. Then, separately, check the coverage of your interval: an 80% band should contain about 80% of the actuals - no more, no less.
| Metric | What it says | Watch out |
|---|---|---|
| MAPE | Average error as a % of the actual - easy to explain to a business. | Explodes when actuals are near zero; punishes over- and under-forecast unevenly. |
| RMSE | Typical error in the series' own units; large errors dominate. | Not comparable across series of different scale; sensitive to outliers. |
| MASE | Error relative to the naive baseline. < 1 beats naive; scale-free. | Needs the in-sample naive error as denominator - use the seasonal version for seasonal data. |
Self-studyWhy MASE is the metric to lead with2 min read▶
MAPE is what you show a business; MASE is what you defend in a model review. Its denominator is the in-sample error of the naive forecast, so it bakes the baseline straight into the score:
- MASE = 1 - you tied the naive baseline. No value added.
- MASE < 1 - you beat naive. The smaller, the better.
- MASE > 1 - a complex model losing to "same weekday last week". Do not ship it.
This closes the loop on b1's promise. "Beat the baseline or do not ship" stops being a slogan the moment MASE is on the dashboard - a single number that says, unarguably, whether the complexity earned its keep.
Implement MAPE, RMSE, and MASE from scratch ★ 12 min · everyone builds
Three short functions, no black boxes. We score the seasonal-naive baseline and one model (a 7-day moving average) on the Lumen holdout so you can read all three numbers side by side - and watch MASE tell you plainly whether the model beat naive.
Rebuild the Lumen daily series and the time split from b1 (train / test, horizon H).
Write MAPE, RMSE, and MASE as pure functions - MASE scaled by the in-sample seasonal-naive error (m=7).
Score the seasonal-naive baseline and one model; print all three metrics for each and compare.
A rolling-origin backtest loop ★ 8 min · build your own
Now turn one number into a distribution. We slide the origin forward, refit a simple model on the expanding history, forecast H steps, score the fold, and report the mean MAPE plus its spread across folds. This is the b3 live forecaster's single MAPE, made honest - the same score, averaged over many origins instead of one.
Choose fold origins that each leave H future days to score against.
For each origin: train on everything before it, forecast seasonal-naive H steps, record the fold MAPE.
Report mean MAPE and standard deviation across folds - the headline and the trust range.
The spread is the point. A model with mean MAPE 9% +/- 2% is a very different bet from 9% +/- 8%. Swap the seasonal-naive step for ETS, ARIMA, Prophet, or your gradient-boosting model and rank them by mean backtested MASE, not by whoever won a single lucky holdout. That ranking is the one you defend in a review - and the honest version of the ladder the b3 live forecaster hints at.
Try it yourself - this week ◐ 30-45 min total
- Score your own model from an earlier session with all three metrics on a fixed holdout. Note where MAPE and MASE disagree, and why.
- Run the rolling-origin backtest with 6-8 folds. Report mean MAPE and its spread, and compare it to your single-holdout number - how far off was the one-shot score?
- Rank two models by mean backtested MASE. State which you would ship and defend it in one sentence.
- Check interval coverage: what fraction of test actuals fell inside your 80% band? If it is far from 80%, your interval is mis-sized.
- Optional: read FPP3 chapter 5 (Hyndman & Athanasopoulos) on evaluation and time-series cross-validation.
Sources covered (cited)
Taught from the forecasting canon on evaluation. This session covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Why prefer rolling-origin CV over a single holdout?
A single split can flatter or punish a model by luck. Rolling-origin marches the origin forward, so you get a mean error and a spread that says how much to trust it.
2 · What does MASE < 1 mean?
MASE scales your error by the in-sample naive error. Below 1 means you beat naive; above 1 means a complex model lost to the baseline and should not ship.
3 · An 80% prediction interval should contain roughly what share of actuals?
Coverage checks interval honesty. If only 50% of actuals land inside an 80% band, the interval is overconfident; a point forecast without a well-calibrated interval is false confidence.