Commissioning a Bayesian answer

Session 1 made the point that a prior nobody can express as a pseudo-count has not been chosen, it has been defaulted. Tonight extends that to the whole answer. The interval level is a default: the tooling shipped 94 percent HDI until 2026 and ships 89 percent equal-tailed now, and an analyst who never thought about it will report whichever their installation happens to produce. The decision rule is a default: "the probability looked high" is what people fall back on when nobody wrote a threshold down, and session 4 showed that 0.915 looked high at n = 50 while the expected loss said keep going. The check is a default too, in the sense that the default is not to run one.

None of these defaults is malicious and most of them are survivable. The problem is that a default cannot be argued with, because there is nothing on the page to argue against. A brief converts each default into a sentence somebody signed. That is the whole job. It does not make the answer right; it makes the answer contestable, which for a leader is the property that matters.

• The prior, as a pseudo-count and a source. "Beta(12, 88), a hundred pseudo-observations, from the chain-wide history." Or "learned across the twelve outlets, kappa 116." Two numbers and a provenance. Session 1 owns the pseudo-count; session 3 owns the learned version.
• The interval, kind and level. "89 percent equal-tailed." Both words matter. The level tells a reader how much of the posterior the range holds; the kind tells them how it was cut, equally from both tails rather than as the narrowest range. Session 2 owns the sentence the interval makes.
• The decision rule, if a decision hangs on it. "Stop when the expected loss of the chosen arm falls below 0.001." Or a region of practical equivalence, Kruschke's rule, that lets the answer be "no meaningful difference" rather than "not yet". Session 4 owns the rule and the reason 0.915 was not enough.
• The check that was run. A prior predictive simulation before the data, to see whether the belief could produce data like the data you expect; a posterior predictive one after, to see whether the fitted model could produce data like the data you got. Session 5 owns the prior predictive check as the shared ground the three prior positions agree on.

A prior is a statement about what you believed before you looked. If it is written after the eighteen customers are in, it is no longer that. It is a conclusion about those eighteen customers, dressed as a belief, and the arithmetic will happily blend the data with a restatement of the data and hand back a narrow interval that has counted the evidence twice. Nobody does this on purpose. It happens when the analyst opens the notebook, sees 1 in 18, thinks "that seems low for Northgate", and reaches for a prior centred a little higher than they would have yesterday. The brief stops it by making the prior a line somebody signed on a date.

The stopping rule has the same problem with a different costume. A threshold written before the experiment, "stop when the expected loss falls below 0.001", is a rule. A threshold written at n = 50, when P(B > A) has just reached 0.915 and the team is impatient, is a decision to stop with a number attached afterwards. Session 4 showed that the pre-written rule said keep going at 0.915 because the loss was still 0.0029, and stopped at n = 100 at 0.0005. It also showed that the rule, run two hundred times, stopped on the wrong arm twice. That is the measured cost of the rule, and it is only a fair cost if the rule was fixed before anyone saw a result. The argument about why looking early and often inflates that cost belongs to learn-experimentation-with-phoebe, and this brief simply insists that the rule be dated.

You do not need to run any of this. You need to know it exists, so that when an analyst answers "which version and which default" you can tell September 2026 from a notebook copied in 2023. Four facts, as of 18 September 2026.

• PyMC is at 6.3.2, released 8 September 2026. PyMC 6.0.0 shipped on 13 May 2026 and needs Python 3.12 or later. "pymc3" is a dead import. If a notebook says pymc3, it predates the current major version by years, and its intervals were cut under the old default.
• ArviZ is at 1.3.0, and is three libraries under one name: arviz-base, arviz-stats, arviz-plots. The old results container, InferenceData, no longer exists; the current one is an xarray DataTree. An answer that talks about InferenceData as current is describing the previous generation of the tool.
• The interval default changed with ArviZ 1. The level moved from 94 percent to 89 percent, and the kind moved from HDI, the narrowest range, to equal-tailed, the 5.5 and 94.5 percent quantiles. The maintainers' stated reason is that 0.89 produces intervals with lower variability and therefore more stable summaries. A third-party repository documented two notebooks whose intervals silently became equal-tailed on upgrade while the prose still said HDI. That is the whole reason line 2 of the brief exists.
• The sampler default changed too. Under PyMC 6, nutpie is the default sampler when installed, with 400 tuning steps against 1000 under PyMC's own sampler. You will never need this number. You need to know that "the defaults" is not a fixed thing, so that "we used the defaults" is not an answer to "what did you use".

The question these facts let you ask is short: "Which PyMC and ArviZ versions, and is that interval 89 percent equal-tailed?" An analyst working in the current tooling answers in one breath. One who hesitates has given you the information you needed.

Under the sensible prior the brief reads: "Beta(12, 88), a hundred pseudo-observations, from the chain-wide history, because Halloway outlets have sat near 0.12 for years." Written that way, the weak point is visible before a single customer is counted. A hundred against eighteen means the history carries 0.85 of the answer, and someone can ask whether Northgate, the newest site in a different kind of neighbourhood, is a chain-wide outlet. The interval, 0.068 to 0.159, is precise and misses the truth by 0.033. That is not a failure of the arithmetic. It is the bet the brief described, lost.

Under the flat prior the brief reads: "Beta(1, 1), two pseudo-observations, because we did not want to assume anything." The interval, 0.020 to 0.222, contains the truth and would support closing the outlet or doubling its budget with equal ease. Under the learned prior the brief reads: "partial pooling across the twelve outlets, learned kappa 116, which weights Northgate's own month at 0.13, because eleven other outlets are the best available evidence about what a twelfth is like." The estimate is 0.124, the error falls from 0.136 to 0.068, and the outlet is not closed. Three briefs, each honest, each arguable, and the argument is only possible because each one wrote its first line down.

Session 3 was careful to say both halves of the pooling result, and the brief has to be as careful. Northgate under partial pooling is 0.124 against a truth of 0.192. The method halved the error; it did not remove it. A report that says "the estimate improved" is true. A report that says "Northgate is now well understood" is not, and the second sentence is the one that gets written when the first is not required.

So a fifth line, informal but required: what the answer still cannot tell you. For Northgate it is that eighteen customers remain eighteen customers, that the estimate leans 0.87 on the other eleven outlets, and that a genuinely unusual outlet will be under-estimated by exactly the method that saves the ordinary ones. Whether the 89 percent intervals your team produces turn out to contain the truth 89 percent of the time, measured over many such numbers, is a different question and belongs to learn-decision-intelligence-with-phoebe. The brief's job is narrower: to make sure the sentence that gets acted on is the one the analysis supports.