The forecast honesty bench

Error this quarter. Forecast minus what really closed, in millions of yen and as a percentage of what closed. Positive means the forecast was too high.

Bias over 20 replays. Add up every replay's forecast minus actual, divide by the total actual. Near zero means the method does not lean either way, even if each single quarter misses.

Error over 20 replays. The average size of the miss, sign ignored, over the average actual. A method can have small bias and large error: it overshoots one quarter and undershoots the next, and the two cancel in the bias. Analyst session 5 met a rep like that.

Coverage. In how many of the 20 replays the method's range contained the actual. An 80 percent band that covers 17 of 20 is roughly honest about its own uncertainty; one that covers 10 is too narrow or aimed at the wrong place.

Real Olist B2B funnel rows (Brazilian marketplace seller acquisition) stand in for Kettle's history. For each lead the bench knows its origin, its first-contact date and whether and when it really won. The history-based method learns its rates only from leads first contacted in January and February 2018, as they could be seen on 1 June: 2,169 leads, 268 won by then. The open pool it forecasts is a different set of leads, so the score is out of sample.

Olist has no stage history and no lost deals, so stages were constructed from each lead's real outcome with deliberate noise: deals that really closed soon tend to sit at later stages, but not always, and some deals that never closed sit at stage 4 or 5. Amounts are drawn from a skewed distribution with a median of 360,000 yen. The quarter's 300 deals are a seeded draw. Everything constructed is labelled on the widget.

Immune. History-based never reads a field a rep can edit. On rung 4 it prints the same 5.01 as on rung 3, with the same bias and coverage, by construction.

Not free. The same blindness means it ignores real signal. A buyer who really did ask for a contract last week is, to history-based, just a 40-day-old paid-search lead. In Kettle's constructed stages, the deals at stage 5 closed far more often than the deals at stage 1: the stage carries information. A team with honest stages and an event log can do better than either method alone, for example by fitting history-based rates within each stage. That needs the stage history from Analyst session 1.

Honest nuance. On these rows, history-based also leans high over the replays (bias +18.6 percent), because recent leads converted less than the training months. Immune to happy ears is not the same as unbiased.

Commit is drawn at a stage line, so inflating stages pushes deals across it. On rung 4 the rep-commit total rises from 7.86 to 10.67m yen, +82.1 percent against what closed, and its coverage over twenty replays falls to 0 of 20. The rep who commits only signed deals, Dev, is untouched, because nothing below signature counts for him; he under-calls every quarter instead.

A point forecast says what is most likely. A range says how wrong the forecaster expects to be. The BD lead commits to a number; the data team publishes the range around it and, from Analyst session 5, how often past ranges covered what happened. That is the ownership split from Business session 3.

The 80 percent bands here come from a normal approximation of the sum of many yes-or-no deals, each with its own probability and amount. With seven wins in a quarter, the real distribution is lumpy and the normal shape is only rough, which is one reason coverage under the whale falls to 13 of 20 even for history-based. Say "80 percent band, modelled" whenever you show one.

When one deal is a third of the quarter, it deserves its own line: the forecast with it, the forecast without it, and the date its buyer last did something. Leaders can then decide whether to commit to the whale, which is their call, while the data team keeps the method honest for the rest. Kettle's whale is constructed; the shape, one deal carrying a quarter, is very common in small B2B teams.