Where the memo stands
Default: do not launch. Criteria: launch if expected net gain clears $4k. Estimates: intervals from a measured estimator. Forecast: base-rated, updated on real evidence. Information verdict: the $5k study is worth at most $3.6k here - decide with what you have. What remains is the last mile where good decisions still die: the telling.
★ What you walk out with tonight. The evidence-backed way to state uncertainty without bleeding trust. Four named ways your decision metric will be gamed, with the pre-commitments that survive them. And a finished memo you can run on Monday.
LiveWhy the telling is a discipline, not a garnish3 min▶
Every earlier session produced something honest and uncomfortable: intervals instead of points, base rates instead of stories, a study priced below its sticker. The last temptation is to sand all that off for the executive summary - "the campaign will work" reads so much cleaner than "70 percent, interval attached".
The sanding feels safe because of a widely believed fear: admit uncertainty and they will trust you less. Tonight's first result is that the fear, tested properly at scale, is mostly wrong - and the way it is wrong tells you exactly HOW to write the memo's last section.
Mango Lane's last big forecast was delivered as "we project $210k" - no interval, though the analyst had one. It came in at $174k. The next quarter's forecast, interval and all, was met with "why should we believe your range". The trust lost was not from the uncertainty; it was from having hidden it the time before.
Numeric ranges barely cost trust. Verbal hedges do.
8 min liveThe question "does admitting uncertainty reduce trust" has a proper answer: a research program led by Spiegelhalter's Cambridge group tested it across experiments totaling thousands of participants, published in PNAS in 2020. The answer has a shape, and the shape is the writing advice.
LiveThe finding, stated with its own calibration4 min▶
The PNAS experiments had people read identical facts and numbers delivered with no uncertainty, with a numeric range, or with a verbal hedge. Communicating uncertainty as a numeric range produced at most a small decrease in the perceived reliability of the numbers and little to no loss of trust in the source. The vaguer verbal form - "some uncertainty around..." - hurt more, on both counts. A follow-up at larger scale held the pattern: only in a combined sample above ten thousand did a small negative effect on trust in the numbers even appear, with the source's trustworthiness unaffected.
Held to this course's own standard, the claim is NOT "uncertainty never reduces trust" - it is: precisely quantified uncertainty carries minimal trust penalty; imprecise verbal uncertainty is what erodes it. Which converts directly into how the memo's last section gets written: every uncertain quantity as a range, no adjective doing a number's job.
The deeper reason this works. A range demonstrates you know WHERE your knowledge ends - which reads as competence. "Some uncertainty" demonstrates only that you have not located the edge. Audiences punish the second because it deserves punishing.
Self-studyThe memo's communication section, row by row3 min▶
| Row | What it contains | Form |
|---|---|---|
| The decision | Launch / hold / kill, in the first sentence | Declarative - the reader never hunts for it |
| The number | Expected value WITH its 90% interval | Numeric range, per the PNAS rule |
| The flip | What future evidence reopens this decision | Session 1's criteria, carried forward as a standing trigger |
| The track record | The estimator's calibration number | One line: "intervals from an estimator measured at N%" |
| The judge | The metric that will score this decision, pre-committed | Part 2's Goodhart-proofed definition, frozen tonight |
Five rows, no adjectives where numbers belong. A memo written like this is refutable, which is precisely what makes it trustworthy.
Goodhart-proofing the decision metric
6 min liveThe memo pre-commits a metric that will decide, next quarter, whether the launch was right. The moment a measure becomes that kind of target, it starts bending - and the bending has exactly four named shapes, catalogued by Manheim and Garrabrant. Naming them in advance is the vaccine.
LiveWhy pre-commitment is the only defense that scales3 min▶
Every Goodhart variant exploits a definition finalized AFTER incentives attached. The regressional drift gets explained as impact because nobody froze a holdout comparison in advance; the adversarial farming passes review because the assignment rule was never written down. The common fix is boring and total: the memo freezes the metric, the comparison group, the operating range and the assignment rule tonight - while nobody's bonus depends on the answer.
This is session 1's ordering law, returned at the far end: criteria before data drove the decision; metric-definition before outcomes judges it. The discipline is the same muscle both times.
The cleanest tell that a metric was defined after the fact: it has exclusions. "Retention, excluding the enterprise accounts, excluding March" is a metric that lost an argument with reality and was renegotiated. A pre-committed metric has no scars.
Finish the memo, deliver the verdict
★ 20 min · everyone buildsThe voucher decision, closed end to end - then your own. Everything below uses sections your memo already carries; tonight is assembly and delivery, which is harder than it sounds and shorter than you fear.
Write the voucher verdict's first sentence. The pieces: default was no-launch; criteria demanded $4k expected net gain; the calibrated estimates put expected gain at $5.2k with a 90% interval of $1k to $9k; the study was priced at most $3.6k against its $5k cost. Verdict: "Launch, without the study - expected net gain $5.2k (90% interval $1k to $9k), clearing our pre-set $4k bar." One sentence, decision first, range attached.
Add the flip-trigger and the track record. "Reopen if month-one retained revenue runs below the interval's floor" - the criteria living on as a standing trigger. Then the metadata line: "estimates from an estimator measured at N percent on claimed-90 intervals" - your session 2 number, still doing work.
Freeze the judge. Write the metric block with all four vaccines: retained revenue per flagged customer (causal-proof), against the randomized holdout (regressional-proof and adversarial-proof), valid for spend between $8k and $16k (extremal-proof). Date it. This paragraph is what next quarter's argument will be settled by, which is why it gets written this quarter.
Read the finished memo aloud, timed. Under two minutes or it is carrying decoration. Every sentence either states the decision, a number with its range, a trigger, or a pre-commitment. Anything else - throat-clearing, methodology tourism, hedging adjectives - goes.
Now close YOUR memo - the live decision you have carried since session 1. Verdict sentence, flip-trigger, track-record line, frozen judge. If a section will not close because an estimate is missing, that is the memo working: it just told you the one thing worth finding out before Monday.
The sentence the whole course compresses into. A decision is well-made when the default was stated before the data, the estimates carry their calibration, the forecast started from the base rate, the analysis was worth its price, and the telling would survive its own audit. Everything else is throughput.
After the course
45 min- Deliver one real memo this week - the one you closed tonight. Two minutes, decision first, ranges attached. Note what questions it kills versus what your old format used to invite.
- Install the calibration log as a team ritual: every estimate anyone commits to gets logged as an interval and scored on arrival. One spreadsheet, zero tooling, compounding returns.
- Audit one existing team metric against the four Goodharts. Write which variant it is currently most exposed to and the one-line vaccine. Most metrics fail at least two.
- Re-run the equivalent-bet test and the six-word question in the same meeting. They compose: what would change your mind, and would you bet at those odds? Together they are most of this course in fourteen words.
- Optional deep read: Spiegelhalter's The Art of Uncertainty - the book-length version of tonight's Part 1, from the researcher whose group produced the trust evidence.
Three questions
Clear all three for the final passport stamp.
1 · What did the PNAS trust experiments actually find about communicating uncertainty?
The finding has a shape, and the shape is the advice: "90% interval $1k to $9k" keeps trust nearly intact; "some uncertainty around the impact" erodes it. Never flatten this to "uncertainty never reduces trust" - the honest claim is that PRECISELY QUANTIFIED uncertainty carries minimal penalty.
2 · The campaign team proposes judging the launch by voucher redemption rate, measured on the flagged segment, definition finalized at quarter-end. How many Goodhart variants is this exposed to?
Causal (clicks are not loyalty), regressional (extreme scores drift back on their own), extremal (the linear relationship breaks outside normal spend), adversarial (steer vouchers to stayers and the number shines) - and bad faith is not required, only incentives. The vaccine is one paragraph, frozen tonight: retained revenue, randomized holdout, capped range, dated.
3 · The finished memo says "Launch: expected net gain $5.2k, 90% interval $1k to $9k." Month one lands at $1.4k. A stakeholder says the memo was wrong. What is the calibrated response?
This is why the interval was published: it pre-defines which outcomes count as surprise. Landing in the low tail is not error, it is the 90 doing its job - and an estimator whose intervals never catch uncomfortable realizations was overconfident, not accurate. The memo's authority comes precisely from having stated this before knowing.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map.
Session 6 cheat sheet · pin this
Session 1 reordered the meeting: default and criteria before data. Session 2 measured your 90 and made it trainable. Session 3 anchored forecasts on base rates and taught the chimp line with its own author's correction. Session 4 turned uncertainty into promises with guarantees. Session 5 priced information and cancelled a study. Tonight the memo shipped with its uncertainty stated and its judge frozen. The voucher launched; the analysis that was never going to change anyone's mind did not - and both of those, finally, were decisions.