Small data, honest uncertainty

Ask what rate could have produced one redemption in eighteen and the honest answer is: a lot of them. A true rate of 0.05 would do it routinely. A true rate of 0.192 does it a good share of the time. A true rate of 0.30 would do it rarely, and 0.50 almost never. Lay those plausibilities out along the axis and you have drawn the posterior: a curve that peaks at the observed 0.056, falls away slowly to the right, and is still clearly above zero at 0.192, where Northgate actually is.

That curve is the whole result. The 0.056 on the slide is one point on it, the peak, and the peak of a wide shape is not a summary of the shape. A point estimate from n = 18 is not wrong; it is meaningless without its spread, in the same way that "the flight is at 3" is meaningless without knowing whether the airline means 3 o'clock or sometime in the afternoon. The spread is the information. Eighteen observations bought a shape, and the shape is wide because eighteen is small.

• Posterior. The distribution over the rate after the data are in. Tonight, under a flat prior, it is almost entirely the data's own shape.
• Flat prior. Beta(1, 1), two pseudo-observations. As close to "no belief" as a prior gets. Session 1 showed it is not nothing; tonight it is used precisely because it claims so little.
• Posterior mean. The balance point of the shape. Northgate's is 0.100 under the flat prior, well above the observed 0.056, because the shape has a long right tail.
• 89 percent interval. The range holding 89 percent of the posterior. Northgate: 0.020 to 0.222.
• Equal-tailed. How the interval is cut: 5.5 percent of the posterior left outside on each side. Every interval tonight, and every Halloway interval on the pages that follow, is 89 percent equal-tailed, and says so.
• Credible interval. The Bayesian interval. It supports the sentence "there is an 89 percent probability the rate is in here".
• Confidence interval. The frequentist interval. A different sentence about a different thing, owned by learn-statistics-with-phoebe b7, and contrasted with the credible interval in exactly one card tonight.
• Width. The interval's top minus its bottom. Northgate 0.202, Westfield 0.037. The number that does the arguing when a point cannot.

Three things are true of Northgate's interval at once, and a leader has to hold all three. It is honest: it contains the true rate, 0.192, which the point estimate missed by 0.136. It is useless for a decision: a rate somewhere between 0.020 and 0.222 covers "close the outlet" and "best in the chain" in one breath, and no budget can be built on it. And it is correct: eighteen observations under a prior that claims almost nothing do not support anything narrower, and any narrower interval would have been bought with a belief rather than with data.

The instinct in the room is to fix the third property by fixing the first two, to pull the interval in until it says something. Session 1 showed the price of doing that with an asserted prior: an interval of 0.068 to 0.159 that read like knowledge and excluded the truth. Session 3 shows the legitimate way, letting the other eleven outlets narrow it. Tonight the discipline is simpler and comes first: say the width before you try to shrink it. An outlet was nearly closed because nobody in the room knew the width was 0.202.

"There is an 89 percent probability the rate is between 0.020 and 0.222." That is a statement about the rate, and it is the sentence almost everybody believes a confidence interval is making. A credible interval is allowed to make it, because the posterior is a probability distribution over the rate itself and the interval is simply where 89 percent of it sits.

The frequentist confidence interval is a different sentence about a different thing. It describes the reliability of the procedure that produced the interval across many repeated samples, not the probability that this particular rate lies in this particular range, and reading it as the Bayesian sentence is one of the most common misreadings in applied work. learn-statistics-with-phoebe b7 owns that interval and the reasoning behind it; nothing here re-derives it. What matters tonight is narrower: when you hear the sentence "89 percent probability the rate is between", ask which kind of interval is being described, because only one of them earns those words.

Eighty-nine is not sacred. It is the level the standard Bayesian tooling ships with: ArviZ 1.x sets its default interval probability, the setting called stats.ci_prob, to 0.89, and its default interval kind, stats.ci_kind, to "eti", the equal-tailed interval. The previous defaults were 0.94 and the highest-density interval. The maintainers' stated reason for 0.89 is that it "produces intervals with lower variability, leading to more stable summaries": at the extremes of a distribution the estimated edge moves around a lot from one run to the next, and a slightly less ambitious level is steadier. Equal-tailed means 5.5 percent of the posterior is cut from each end, which is what every Halloway interval means and every page says.

The number matters less than the habit. There is a documented case, in a third party's notebooks, of intervals that silently changed kind when the library was upgraded: the code still ran, the numbers still printed, the prose still said "highest density", and the intervals were now equal-tailed. Nobody had done anything wrong; a default had moved under them. The defence is to state kind and level with every interval, so that "89 percent equal-tailed" travels with the numbers and a reader can tell when it stops matching. An interval with no kind and no level attached is not yet a result. It is a pair of numbers waiting for someone to ask.

Westfield's interval, 0.124 to 0.161, is 0.037 wide. Northgate's is 0.202 wide. Fifty times the data produced an interval about a fifth as wide, which is a great deal, and a long way from a point. The posterior mean of 0.142 is still 0.008 below the true 0.150, and the interval is what tells you that gap is unremarkable: 0.150 sits well inside it. Without the interval, a finance team comparing 0.142 to last year's 0.150 might have called it a decline. With it, the honest reading is that 900 vouchers cannot tell those two numbers apart.

That is the general lesson about volume. More data narrows the shape, and each extra voucher narrows it by less than the one before. It never collapses the shape to a dot, and the width at any given volume is a fact about the data, not a fault in the analysis. A slide that shows Westfield as 14.2 percent and Northgate as 5.6 percent is showing two dots of wildly different reliability at identical size. The interval is the only thing on the page that knows the difference.

Replace "the rate is 5.6 percent" with "there is an 89 percent probability the rate is between 2.0 and 22.2 percent" and stop talking. The width does the arguing for you. Nobody hearing that sentence proposes a closure, and nobody hearing it proposes a bonus either; they propose getting more data or borrowing some, which is the right next move and the subject of session 3. The point estimate invited a decision the evidence could not support. The interval declines the invitation on your behalf.

Two cautions. First, an interval you cannot describe is not one you should present: say the level and the kind, "89 percent, equal-tailed", or someone will assume 95 and be wrong about what you claimed. Second, this is one card, not the method. The full craft of saying uncertainty to executives, the interval, the probability of a difference, the stake attached to being wrong, is practitioner session 9's whole 45 minutes. Whether a stated 89 percent turns out to be honest when scored across many such statements is a separate question again, and learn-decision-intelligence-with-phoebe owns it. Tonight's move is the first and smallest: never show a small-data point without its width.