Saying it to executives

Here is Northgate under the flat prior, session 2's first row, said the way it should be said. "The best estimate of Northgate's redemption rate is 0.100. There is an 89 percent probability it lies between 0.020 and 0.222 (equal-tailed). Eighteen vouchers cannot pin it tighter than that." Three sentences, four jobs. The estimate is the posterior mean, not the observed 0.056, because the observed rate is one redemption in eighteen and the posterior mean is what that evidence supports once the two pseudo-observations of the flat prior are counted. The interval carries its level, 89 percent, and its kind, equal-tailed, because either word missing makes the range unreproducible. The stake is the last sentence: the width is the news, and the reader is told so rather than left to notice.

The fourth part, the probability of the thing they care about, is absent at Northgate on purpose. No decision hangs on Northgate alone tonight, so there is no "the thing they care about" to put a probability on. When there is one, as at Westfield, it goes in as its own sentence with its own stake. A sentence that includes a probability nobody asked for is padding; a sentence that omits the one they did ask for is the real failure, and it is the more common of the two.

• The estimate. The posterior mean, stated as "the best estimate" and never alone. Northgate 0.100 under the flat prior; 0.124 under partial pooling from session 6. Which one you report depends on the prior line below, and the reader is told which.
• The interval, with its level and its kind. "There is an 89 percent probability it lies between 0.020 and 0.222 (equal-tailed)." The level is 89 because that is what ArviZ 1 reports by default; the kind is equal-tailed because that is how ArviZ 1 cuts it, at the 5.5 and 94.5 percent quantiles. Both words, every time.
• The probability of the thing they care about. Only when a decision hangs on the number. At Westfield it is P(B > A) = 0.975 at n = 100 per arm. It is never the interval restated; it is the probability of the action being right.
• The stake if wrong. At Westfield, the expected cost of choosing B is 0.0005 in redemption rate. At Northgate, where there is no decision, the stake is what the data cannot do: eighteen vouchers cannot pin the rate tighter than 0.020 to 0.222. A probability without a stake is not a decision, session 7's whole point, and the sentence inherits it.

The flat-prior sentence for Northgate is true and nearly useless: 0.020 to 0.222 would support closing the outlet or doubling its budget with equal ease, and the sentence is honest about that in its last line. Session 6 gives a better estimate. Partial pooling across the twelve outlets learned kappa = 116, weights Northgate's own month at 0.13, and puts the outlet at 0.124. The pooled sentence starts differently: "Under partial pooling across Halloway's twelve outlets, the best estimate of Northgate's redemption rate is 0.124, leaning 0.87 on the other eleven." Then its interval, cut the same way, with the same two words. Then the same closing stake, because eighteen vouchers are still eighteen vouchers.

Here is what neither sentence can promise. Northgate's true rate is 0.192. The flat interval contains it, the pooled estimate halved the error from 0.136 to 0.068 and is still under it. An 89 percent interval promises that 89 percent of the posterior's mass sits inside the range; it does not promise the truth is there, and a sentence that says "the rate is between 0.020 and 0.222" has dropped the word probability and made a promise the arithmetic never made. Say "there is an 89 percent probability". Whether your 89 percent intervals turn out to contain the truth 89 percent of the time, measured across many such sentences, is a calibration question, and learn-decision-intelligence-with-phoebe sessions 02 and 04 own it. The sentence's job is to be the thing that audit can be run on.

Until ArviZ 1 the default summary interval was a 94 percent HDI, the narrowest range holding 94 percent of the posterior. ArviZ 1 changed both halves at once: stats.ci_prob is now 0.89 and stats.ci_kind is now "eti", an equal-tailed interval cut at the 5.5 and 94.5 percent quantiles. The maintainers' stated reason is that 0.89 produces intervals with lower variability and therefore more stable summaries. The argument for the number is theirs to make. The consequence for the sentence is not optional.

A third-party repository documented two notebooks whose intervals silently became equal-tailed on upgrade while the prose beside them still said HDI. The code ran, the numbers changed, the words did not. A sentence that says "the interval is 0.020 to 0.222" cannot be reproduced, because a reader does not know whether to cut 89 or 94, equal-tailed or highest-density, and each choice gives a different range from the same posterior. A sentence that says "89 percent, equal-tailed" can be reproduced by anyone with the draws. That is the whole reason both words are in the template, and it is why the keyword in ArviZ 1 is now prob, set explicitly, rather than a default trusted.

The Northgate interval is a credible interval, and it supports the sentence "there is an 89 percent probability the rate is in here", because the posterior is a distribution over the rate and the interval holds 89 percent of it. A confidence interval is a different object: a statement about the procedure that produced it, across repeated samples, and it does not support that sentence about this rate. Executives hear the Bayesian sentence whichever interval they are shown, which is exactly why the Bayesian one has to be labelled and the other one has to be handed on. The frequentist machinery, what a confidence interval does claim and how it is built, is learn-statistics-with-phoebe session b7, and nothing about it is derived here. The contrast is the only thing tonight needs.

At n = 100 per arm, Westfield's test reads A 0.100, B 0.200. P(B > A) is 0.975, the expected loss of choosing B is 0.0005 in redemption rate, and the rule fixed before the experiment, stop when the expected loss of the chosen arm falls below 0.001, says stop and choose B. The executive sentence carries the probability and the stake together: "There is a 97.5 percent probability B is better; if we are wrong, the expected cost is 0.0005 in redemption rate." The semicolon is doing real work. Split into two sentences, the first one gets quoted alone. Session 7 showed why that matters: at n = 50 the probability was already 0.915 and the rule said keep going, because the expected loss was 0.0029, three times the threshold. A probability alone would have stopped fifty customers too early.

Then the honesty clause, in the same paragraph and never in an appendix. Across 200 seeded replays of the same experiment, stopping at n = 100 by the same rule, 23 would have stopped and 2 of those 200 would have stopped on the wrong arm. Write it as a count: "Run 200 times, this rule stopped on the wrong arm 2 times in 200." It is a 1 percent wrong-stop rate at that sample size, it is cheap, and it is not free. A reader who gets the 0.975 and the 0.0005 without the 2 in 200 has been told the price of the decision and not the odds of having bought the wrong thing. The reason looking early and often would inflate that count is learn-experimentation-with-phoebe session b6's argument. The paragraph's job is to report the measured rate and stop there.

• Report a point estimate without its interval. "Northgate is at 0.100" is the sentence that flagged the outlet for closure. The interval was 0.020 to 0.222 and would have ended the conversation.
• Report a probability without its stake. "97.5 percent probability B is better" is half a decision. The other half is 0.0005, and at n = 50 the half you would have quoted was 0.915 with a stake of 0.0029 behind it, which the rule rejected.
• Say "significant". The word belongs to a procedure this sentence did not run and would be read as a verdict from it. Say the probability and the stake instead; they are what you have.
• Say the prior "washed out". It was outvoted, and a hundred votes take a long time to outvote. The Wrong prior, Beta(40, 60), worth 100 pseudo-observations, still moves Westfield's estimate by 0.026 at n = 900: posterior mean 0.167 against 0.142 under the flat prior. Session 2's number, and the reason the prior disclosure line below exists.
• Promise there is no peeking problem. The expected-loss rule controls the cost of a wrong decision, not the frequency of one, and the 2 in 200 is the frequency. State the count and hand the argument to learn-experimentation-with-phoebe b6. A paragraph that says "we could stop whenever we liked" has made a claim the bench measured and contradicted.

Every executive paragraph carries one more line, usually first: which prior, worth how many pseudo-observations, and why. "Flat, Beta(1, 1), worth 2 pseudo-observations, because we had no defensible history for a new outlet." Or "Informed, Beta(12, 88), worth 100 pseudo-observations, from the chain-wide history." Or "Learned by partial pooling across the twelve outlets, kappa 116, which weights Northgate's own month at 0.13." Two numbers and a provenance. The line exists because the estimate depends on it: Northgate reads 0.100, 0.088, 0.110 or 0.347 under session 2's four priors, and an executive who has not been told which is which has been handed a number with a hidden author. The argument about how a prior should be chosen, the three positions and the shared ground of a prior predictive check, is leader session 5's, and the disclosure line is where its conclusion lands in one clause.

The line also catches the prior that should never have shipped. The Wrong prior, Beta(40, 60), predicts roughly 40 percent redemption. Halloway has never observed above 0.225 at any outlet. A prior predictive check, session 8, fails it before a single voucher is counted, and a disclosure line that says "Beta(40, 60), worth 100 pseudo-observations" fails it in the reader's head for the same reason. What the line cannot do is tell you whether the intervals it introduces are honest over many forecasts. That measurement, calibration over a track record, is learn-decision-intelligence-with-phoebe sessions 02 and 04.