Bias, error and the track record

Every number on the widget comes from one of these three. Write them down in this form, because small changes in wording produce different numbers.

Bias keeps the sign, so overs and unders cancel. It answers "which way does this forecast lean?" Error drops the sign, so nothing cancels. It answers "how far off is a typical quarter?" Coverage ignores the point forecast and checks only the range, so it answers "when this forecast says 'somewhere between', can I plan on that?"

Both bias and error are scaled by actuals in total, not quarter by quarter. A tiny quarter with a huge percentage miss would otherwise dominate the average, which is the main reason per-quarter percentage averages behave badly on lumpy sales numbers.

On the bench. The open pool is 3,465 real Olist leads first contacted March to May 2018 and not won by 1 June 2018. Each replay draws 300 of them; draw 0 is "this quarter", the one Analyst session 4 forecast. Which leads really closed in the 91 days to 30 August 2018 is real. Stages, amounts and which rep owns a deal are constructed with seed 20181011, so every reload prints the same numbers.

On your own data. You do not need replays: you have past quarters. For each one you need two frozen things, the forecast as it was submitted and the pipeline snapshot it was made from (the as-of join from Analyst session 1), plus what really closed by quarter end. Eight quarters is a thin track record; twelve is better. Replays stand in for that history here because the public rows cover only one usable quarter.

What replays cannot do. Every replay shares the same market, the same season and the same buyers' behaviour. Real quarters differ in all three, so a real track record will usually look worse than a replayed one.

Rep commit: +33.9% bias, 48.2% error. It leans high, and most of its error is that lean. A steady lean is the most fixable fault a forecast can have, as long as it stays steady.

Stage-weighted: +62.7% bias, 68.2% error. It leans hardest of the three. Almost all of its error is lean, which says the stage table (Kettle's own, from the prior quarter) is set too generously for the deals now in the pipe.

History-based: +18.6% bias, 38.8% error. The smallest of both, and still not small. Over half its error is noise rather than lean, which is what you expect from about seven wins a quarter: a couple of deals arriving a week early or late moves the actual a long way.

The call ratio is a rep's total commit over twenty quarters divided by what that rep really closed. It is the rep's bias plus one. Above 1 over-calls, below 1 under-calls.

Aiko, 0.98. Unbiased in total, 68.0% error per quarter. As a single-quarter call her number is close to useless, yet she is the rep most managers would hold up as "accurate". Ben, 4.60. He commits from stage 4, a stage below the others, and over-calls more than four-fold; error 360.1%. Chiara, 1.45. Mildly high, 69.7% error. Dev, 0.00. He commits only once a contract is signed, so his commit is always zero and his error always 100.0%. That is an under-caller, often called a sandbagger, and his zero tells the BD lead nothing about what he will close.

History-based learns conversion from earlier cohorts. In the real Olist rows the May 2018 cohort converted 10.0%, against 13 to 14% for January to April. A history-based forecast assumes the next quarter's leads behave like the last ones, so when a weaker cohort arrives it over-calls. That is the honest limit of the method: it cannot see a change it has not yet lived through. The remedy is not a cleverer formula; it is watching the lean quarter by quarter and asking why when it moves.

A systematic lean is a pattern in how a person reads deals, so it is a coaching conversation for the BD lead, not something data should quietly correct. If an analyst divides Ben's commit by 4.60 every quarter, three things go wrong: Ben never learns, nobody notices when he changes, and the forecast now depends on a private adjustment no one signed off. Show the call ratio to the BD lead and to Ben, agree what "commit" means, and keep scoring. Who owns that conversation, and who owns the method, is the subject of Business session 3, Who owns the forecast.

Dev's case is different. A zero is not a lean you can scale, because zero times anything is zero. The fix is a definition: commit means a deal the rep expects to sign this quarter, not one already signed.

Rep commit's range runs from the commit to commit plus best case, so it is two rep judgements stacked. Stage-weighted and history-based ranges are modelled: the point forecast plus and minus 1.2816 standard deviations, a normal approximation of a sum of yes-or-no deals, each with its own win probability. That is the standard 80% band under a normal curve, and it is labelled as modelled on the widget.

The approximation is rough here. This quarter, 7 deals won out of 300. A sum of seven or so wins is lumpy: one large deal closing or slipping moves the actual by more than the smooth bell curve expects. So even an honest modelled range should be read as "about 80%", and coverage is the check on whether "about" is close enough.

Twenty quarters is a small sample, so 16 of 20 is the centre of a spread, not a pass mark. Constructed arithmetic for an honest 80% range: the count of covered quarters has a standard deviation of the square root of 20 x 0.8 x 0.2 = 3.2, which is 1.79. So 14 to 18 covered is ordinary; 17 is comfortably inside.

10 of 20 is more than three standard deviations below 16. An 80% range that covers half the time is either too narrow, centred in the wrong place, or both. With stage-weighted's +62.7% bias, the centre is the larger problem. 4 of 20 for rep commit is not a calibration slip at all: the range starts at the commit, and the commit sits above the actual in most quarters.

Coverage that is too high is a fault too. A range covering 20 of 20 every time is usually so wide it does not help anyone plan.

"Happy ears" is the sales habit of hearing a yes the buyer never said. On the bench, the checkbox moves 30% of Kettle's deals below stage 5 up one stage (constructed). Then:

• Rep commit bias rises from +33.9% to +121.5%; coverage falls from 4 to 0 of 20.
• Stage-weighted bias rises from +62.7% to +134.0%; coverage falls from 10 to 3 of 20.
• History-based is unchanged: it reads each deal's origin and age, never its stage.
• Reps: call ratios move to Aiko 1.69, Ben 8.15, Chiara 2.18, Dev 0.00. Dev does not move, because he still commits only signed deals.

History-based's immunity is also its blind spot. It cannot be inflated by a rep's optimism, and for the same reason it cannot see a real late-stage signal either. Analyst session 6 puts that trade-off on one bench.

Forecasting and revenue-intelligence vendors and analyst firms publish surveys on how few sales forecasts land within some band of the actual. Leaders disagree about what they are worth. Some use them as an outside anchor: if most teams miss widely, a 38.8% error is not a crisis. Others treat them as marketing, because the method and sample are usually not published and the publisher sells the fix. Both readings are reasonable, and no survey figure is quoted here for that reason. Your own twenty quarters, scored with the definitions above, are a better benchmark than any survey for the one decision that matters: which forecast your BD lead should believe next quarter.