Why measurement is a leadership job
A forecast that is never scored is just an opinion in a chart. The most expensive forecasting mistakes are not bad models - they are good-looking numbers nobody checked against reality. Your job is not to run the metric; it is to insist the metric exists, ask what it says, and refuse the false comfort of a single line with no error bar. Four questions do most of the work: how wrong is it, on average? was it tested on the future? does it beat the obvious baseline? and does it admit its own uncertainty?
How wrong is it? Three ways to say it 8 min live
Every forecast has error: the gap between what it predicted and what actually happened. There is no single "the accuracy" - there are a few standard ways to summarise those gaps, and each answers a slightly different question. You do not need the formulas. You need to know which number your team is quoting, and what it hides.
LiveMAPE, RMSE, MASE - what each one is really telling you4 min▶
Three names come up in almost every forecasting review. Here is the plain-English version of each - and the trap in each.
- MAPE (mean absolute percentage error) - the average gap expressed as a percentage. "We are off by about 8% on a typical day." Intuitive and unit-free, which is why executives love it. The trap: it explodes near zero. On a day with almost no sessions, being off by a handful looks like a 400% error, so MAPE lies on low-volume or spiky series.
- RMSE (root mean squared error) - the typical size of a miss, in the same unit as the data (sessions, dollars, units). The trap: it squares the gaps first, so one huge miss dominates the score. That is a feature when a big miss is what actually hurts you (a stockout, a blown capacity plan), and a bug when you only care about the everyday case.
- MASE (mean absolute scaled error) - the honest referee. It scores your forecast relative to the naive baseline. Below 1 means you beat naive; above 1 means you are worse than "just repeat last week". Why leaders should ask for it: it is the one number that answers "is the model earning its keep?" in a single glance.
Self-studyWhy "8% error" means nothing without context2 min read▶
A percentage on its own is not good or bad. An 8% MAPE is excellent for volatile daily web traffic and embarrassing for stable monthly revenue. Three questions turn a bare number into a judgement:
- Compared to what? 8% only means something next to the baseline's error (Part 2). If naive also scores 8%, your model added nothing.
- At what horizon? Error almost always grows the further out you forecast. A 3% next-day error and a 3% next-quarter error are very different achievements.
- On which days? An average hides the pattern. A model that is superb on weekdays and hopeless on promo days may need different handling, not a better single number.
Lumen's dashboard quotes "6% MAPE" in green. The leader who asks the right questions learns that the seasonal-naive baseline also scores 6%, the number is measured one day ahead but the planning decision is 30 days out, and most of the error is concentrated on campaign launch days. The headline was true and almost useless. Measurement is only as honest as the questions around it.
Test it on the future it never saw 9 min live
You cannot judge a forecast on the data it was trained on - that is like grading students on questions they already had the answers to. A model can memorise the past perfectly and still be useless tomorrow. The only fair test is to hide some of the future, forecast it, then reveal it and score. Do it once and luck can flatter or punish you. Do it many times, marching the window forward, and you get an honest average. This repeated test is called backtesting, or rolling-origin evaluation.
LiveBeat the naive baseline - the single most important check3 min▶
Before you celebrate any forecast, ask one question: does it beat naive? The naive forecast is "tomorrow equals today"; the seasonal-naive is "next Tuesday equals last Tuesday". These cost nothing and require no model - and they are shockingly hard to beat.
- The baseline is the bar, not the enemy. If a complex model cannot beat seasonal-naive on a backtest, its complexity is not earning its place. Retire it, or keep the baseline.
- This is MASE in one sentence. MASE below 1 = you beat naive; above 1 = the naive line was better. It is the same check, made into a number.
- The baseline is also your safety net. When a fancy model breaks in production, seasonal-naive is the fallback that keeps the business running.
The M-competitions taught the field humility. Across the largest public forecasting bake-offs (M4, 2018; M5, 2020), simple methods and combinations of them were repeatedly hard to beat, and elaborate models often lost to a well-chosen baseline. The leadership takeaway: demand that every model prove itself against naive on a backtest before it ships. Complexity must earn its keep.
Self-studyOne holdout vs many folds - why repetition matters2 min read▶
A single test window can mislead. If the one month you held out happened to be calm, every model looks great; if it contained a spike, every model looks terrible. Neither tells you how the forecast behaves on average.
- More folds, more trust. Scoring across many rolling windows averages out the luck of any single period.
- Match the test horizon to the decision. If you plan 14 days out, each fold should forecast 14 days out. Testing one day ahead and planning a month ahead is measuring the wrong thing.
- Never tune on the final test. Choosing a model by peeking at the very data you will report on is a form of cheating (leakage). We return to this in Session 6.
A point forecast alone is false confidence 6 min live
"We expect 1,200 sessions next Tuesday" sounds precise, and that is exactly the problem. The future is a range, not a number. A responsible forecast reports a prediction interval - "1,200, and 80% likely to land between 1,050 and 1,350". The interval is not the model hedging; it is the model being honest about how much it does not know.
LiveCoverage - the promise an interval has to keep3 min▶
An 80% prediction interval makes a testable promise: over many forecasts, the truth should fall inside it about 80% of the time. That property is called coverage, and you can check it.
- Too narrow (say 50% coverage) - the model is overconfident. Plans built on it will be blindsided far more often than expected.
- Too wide (say 99% coverage) - technically safe, but so vague it is useless for a decision. An interval from zero to infinity is always right and never helpful.
- Intervals widen with the horizon. Next week is fairly knowable; next quarter much less. If someone shows you a far-out forecast with a razor-thin band, be suspicious.
Try it yourself - this week ◐ 20-30 min total
- Pick one forecast your team or business actually relies on (traffic, demand, cash-flow, headcount).
- Ask, and write down the answers: What is its MAPE (or RMSE)? If nobody knows, that is your finding.
- Does it beat the naive baseline? If it has never been compared to "repeat last week", ask for that comparison.
- Does it report a prediction interval? If it is a single number, ask what the range of likely outcomes is.
- Write one sentence on the decision this forecast drives, and whether the current accuracy is good enough for that decision.
- Optional: skim the evaluation chapter of FPP3 (Hyndman & Athanasopoulos, otexts.com/fpp3) - the open textbook behind this track.
Sources covered (cited)
Taught from the forecasting canon and the published empirical evidence. This session covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Why can a forecast look great in review and still fail in production?
A model can memorise the past. The only fair test is on hidden future data - a backtest - ideally repeated across many rolling folds.
2 · What does MASE below 1 tell a leader?
MASE scores the forecast against naive. Below 1 beats naive; above 1 is worse than just repeating last week. It is the "does it earn its keep" number.
3 · Why insist on a prediction interval, not just a single number?
A point forecast hides uncertainty. An interval states it, and its coverage is testable - too narrow means overconfident, too wide means useless.