Where the leader track has taken you
Five sessions ago the running decision was should Cadence, an AI meeting-notes product, build a lecture-notes product for higher education next year? a1 showed which parts of research AI actually compresses. a2 gave your team an evidence standard. a3 catalogued where AI fails with real numbers attached. a4 drew the consent and confidentiality rails. a5 put instruments in their hands with the limits written down. Tonight is the part none of that covers: your team hands you nine claims they can defend and one they cannot, and somebody in the room needs a decision by Thursday.
The four-part confidence sentence 12 min live
Most confidence language does no work. It either overstates, because a flat assertion is what gets applause in a meeting, or it hedges so thoroughly that nobody can act on it, and both failures come from the same missing structure. A useful confidence statement has exactly four parts: the claim, the weight you are putting on it, the basis that weight rests on, and what would change your mind. Drop any one of the four and you have produced a mood rather than a finding. The fourth part is the one people leave out, and it is the one that makes the sentence honest - because it converts your position into something the next piece of evidence can actually move.
Here are the three versions of the same finding, in the order they usually appear in real meetings. The first wins the room and cannot survive a question. The second survives every question and helps nobody. The third is the one to train your team to write.
"Institutions hold the budget for lecture-notes tools, so we need to sell to procurement, not to students."
"There are indications that budget authority may in some cases sit at the institutional level, although student-led adoption is also a factor and the picture is complex and evolving."
"Institutions rather than individual students hold the budget for lecture-notes tools (claim). We hold this at moderate confidence in the GRADE sense: likely true, and further evidence could change it (weight). It rests on four tier-1 and tier-2 sources that agree, against one that contradicts it specifically for community colleges (basis). Three department-level purchases in our two target markets, or one contradicting procurement study, would move us (what would change our mind)."
LiveBorrowing GRADE so "moderate" is not a personality trait4 min▶
The word "moderate" means whatever the person saying it wanted it to mean, which is why weight language quietly fails. Evidence synthesis solved this a long time ago, and you can borrow the solution wholesale. GRADE grades certainty at four levels: high, where "We are very confident that the true effect lies close to that of the estimate"; moderate; low, where "Our confidence in the effect estimate is limited"; and very low.
Two more pieces of GRADE are worth stealing for commercial work:
- Where a body of evidence starts. Randomised trial evidence starts at high certainty; observational evidence starts at low. So a claim built on survey and interview data does not begin at high and get argued down - it begins at low and has to earn its way up.
- The five reasons to downgrade: risk of bias, inconsistency, indirectness, imprecision, publication bias. Those are five specific, nameable objections. Your team saying "moderate, downgraded for indirectness because the studies are on secondary schools and we sell to universities" is a completely different conversation from "we are fairly confident".
You do not need the full apparatus. You need the four labels used consistently and a habit of naming the downgrade, so that weight becomes an argument about evidence instead of an argument about who sounds more certain.
Self-studyThe p-value rule to enforce out loud3 min read▶
Fast evidence arrives with statistics attached, and one habit does more damage in commercial decisions than any other: treating a threshold as a verdict. The American Statistical Association's 2016 statement on p-values sets out six principles, and the one to enforce in your meetings is this: "Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold." Note the words "business or policy". This is not an academic nicety being borrowed; the statisticians wrote your use case into the sentence.
The second quotable line closes the loophole people reach for next: "P-values do not measure the probability that the studied hypothesis is true." So "p is 0.03, so there is a 97% chance this works" is not a cautious reading of the evidence, it is a different claim entirely.
The practical move for a leader is to ask two questions whenever a significance claim appears. How big is the effect, and how precise is it? A significant result on a large sample can be tiny, and a tiny effect can be commercially irrelevant even when it is real. That is why the weight in your confidence sentence should reference the size and the precision, never the threshold.
The audit trail, without the ceremony 11 min live
The question that ends careers is not "was this right". It is "how did you decide", asked long after everyone has forgotten. An audit trail is what makes that question answerable, and it is cheap if you build it while working rather than reconstructing it afterwards. Six things to keep, per decision: the question and the claim list; the search record - which databases, on what dates, with what strategy; the screening log, with exclusion reasons; the extraction table, one row per source with its numbers; the contradictions found and how they were handled; and the confidence statement per claim. Six artifacts, all of which your team produced anyway during a1 through a5. The discipline is keeping them, not making them.
This is PRISMA-style discipline, borrowed without the ceremony. PRISMA 2020 is a reporting guideline for systematic reviews, and its two visible pieces are worth knowing even if you never produce one: its flow diagram records three bands - identification, screening and included, with the 2009 "eligibility" band removed and eligibility now a box inside screening - and its checklist has seven sections and 27 items. You are not going to run 27 items on a two-week market question. You are going to steal the idea: a reader should be able to see what you looked at, what you excluded, and why.
LiveWhat the screening log buys you, specifically4 min▶
Of the six artifacts, the screening log is the one teams skip and the one that pays the most, because it is the only record of what you decided not to look at. Three concrete returns:
- It converts an accusation into a conversation. "You ignored the community-college evidence" is unanswerable without a log. With one, the answer is "we excluded it at screening because our two target markets do not have that segment, and here is the line where we wrote that down before searching".
- It makes the work repeatable when the decision comes back. Six months later the question returns with a wider scope. If the exclusion reasons are recorded, you re-run the search with one rule relaxed instead of starting over.
- It catches the reason that was actually a bias. Writing exclusion reasons down forces you to notice when the real reason was "it was paywalled" or "it disagreed with the pitch". Both are legitimate constraints and neither is a methodological reason, so both have to be visible.
Keep it boring. A spreadsheet with source, decision, and reason is a real screening log. The value is entirely in it existing before the argument starts.
Disclosure is not a check. Guo, Dong and Hao, Frontiers in Cell and Developmental Biology 11:1339390, was retracted on 2024-02-16 over AI-generated figures (retraction notice DOI 10.3389/fcell.2024.1386861). The detail that should stay with you as a leader is that the AI use was disclosed. The authors followed the rule. It still failed, because disclosure is a statement about process and nobody performed the one step that process was supposed to protect - somebody looking at the output and asking whether it was right. Every governance rail from a4 has this same shape. Writing down that a tool was used costs nothing and protects nothing on its own; the protection is the human check, and the audit trail is where you record that the check happened.
Self-studyRecording contradictions instead of resolving them3 min read▶
The contradiction record is the most counter-intuitive of the six artifacts, because the instinct is to resolve disagreement before writing anything down. Resist it. What you want on the page is the disagreement itself and the reasoning you applied to it, for a practical reason: when new evidence arrives, the only way to know whether it changes your position is to know why you weighted the old evidence the way you did.
A usable format is three columns. What the disagreement is, which side the stronger designs fall on, and how the claim was worded as a result. If a claim went out flat despite a contradiction, that is fine as long as the row explains why - a single cross-sectional study against three cohort studies is a defensible discount, and it stops being defensible only if you hid it.
The pay-off shows up when someone brings you the contradicting paper in a meeting. Either your row already accounted for it, and you can say so in one sentence, or it did not, and you have just learned something in the cheapest possible way.
Saying "we do not know yet" without losing the room 10 min live
"We do not know" sounds like a failure to deliver, so people bury it in a qualifier and the organisation loses the single most useful thing research produces - a map of where the risk actually sits. The fix is not courage, it is form. An unknown stated with a price, a test and a next step is a contribution. An unknown stated bare is a shrug. The lab's tenth claim is the worked example for all of this: "our meeting-notes engine transfers to lecture audio without retraining" rests on a single vendor-funded preprint with n=48 and nothing replicating it, and it never survives the audit at any rung - 0 to 1 to 2 to 4 to 7 to 9 of 10 claims, and the tenth stays red by design.
LivePrice the unknown, then name the test4 min▶
Two moves, always in this order. Price it: what does being wrong cost, and what does waiting cost? Both halves matter. A leader who only prices the error looks paralysed; a leader who only prices the delay looks reckless. Then name the test: what specific thing would settle this, and how many days is it.
On claim ten, that comes out as: "Untested. Our engine's transfer to lecture audio rests on one vendor-funded preprint with 48 participants and no replication, so we treat it as unknown rather than weak. Being wrong costs a rebuild of the audio pipeline after we have sold the product, which is the expensive direction. Waiting costs two weeks. A benchmark on 30 real lectures across our two target markets, scored against human transcripts, would settle it, and it is a two-week piece of work. Until it runs, we do not price on transfer and we do not put it in a pitch."
Notice what that paragraph does in a room. It does not ask for patience, it makes a specific offer with a date attached, and it moves the argument from "are you confident" to "do we spend two weeks". The second argument is one you can actually win.
LiveOffer a reversible first step3 min▶
Most decisions presented as one big yes or no are actually a sequence, and the first item in the sequence is usually cheap and reversible. Finding that item is the highest-leverage thing you can do with an unknown, because it lets the organisation move while the evidence is still being gathered.
For Cadence: committing to a higher-education product line is not reversible, but running the lecture benchmark, talking to six procurement offices, and shipping a limited pilot to one institution all are. Each of those also generates the evidence the audit is missing, which is the part people miss - a reversible first step is not a delay tactic, it is data collection with a deadline.
The sentence to practise: "I would not commit to the roadmap slot yet. I would spend two weeks and one engineer on the benchmark, and I would take the procurement conversations now because they are slow and free. If both come back positive we commit in March with the transfer question closed."
Self-studySeparate the decision from the evidence3 min read▶
This is the distinction that makes the other three techniques usable, and it is worth saying explicitly to your leadership team at least once: a leader is allowed to decide under uncertainty. What a leader is not allowed to do is pretend the evidence was stronger than it was.
Those are two separate acts, and conflating them causes most of the damage. Research teams learn to soften findings because they believe a weak finding blocks a decision, and executives learn to distrust research because it keeps arriving pre-softened. Pull them apart and both problems ease. The evidence says what it says, at the weight it can bear. The decision is a judgement about risk, timing and appetite, made by whoever owns it, and it can legitimately run ahead of the evidence.
Say the whole thing out loud when it happens: "the transfer claim is untested and we are proceeding anyway, because the market window is worth the risk and the rebuild is survivable". That sentence is defensible in a review nine months later. "We were confident it would transfer" is not, and it is the sentence that gets produced when the two acts were never separated.
Five things, and then this is just how you work 9 min live
A course is only worth the artifacts it leaves behind. Here is the whole leader track as an operating cadence: five standing commitments, each producing one artifact, each with a signal that tells you whether it is real or theatre. None of them needs a new headcount and none of them needs a tool purchase. What they need is for you to ask for the artifact often enough that producing it becomes cheaper than not producing it.
| Commitment | Artifact it produces | How you know it is working |
|---|---|---|
| A standing evidence standard (a2) | One page: the source tiers, the four certainty levels, and what a claim needs before it can carry a decision | Arguments in review meetings are about the weight of evidence rather than about who sounds more certain |
| A failure catalogue, reviewed quarterly (a3) | A living list of documented AI failure modes with their real numbers, and which of your workflows each one touches | Someone brings a new failure mode to the review unprompted, and a workflow changes because of it |
| The governance rails, written down (a4) | What may be uploaded where, what disclosure looks like per venue, and the participant-data rules | People ask before pasting, and the answer is findable in under a minute without asking you |
| A stack with stated limits (a5) | A tool list where every entry carries its limit next to its feature, plus the five procurement gates | A vendor demo gets a written question list back instead of a purchase order |
| A decision log (a6) | Per decision: the claims, their confidence statements, the audit trail, and the named revisit trigger | A revisit trigger actually fires and a decision gets re-opened without anyone having to defend themselves |
Try it yourself - this week ◐ 30-40 min total
- Take the last significant decision your team made on research input and write its four-part confidence sentence retrospectively, claim by claim. The parts you cannot fill in are the parts nobody had at the time.
- Open the most recent memo your team shipped and check it against the six audit-trail items. Whichever one is missing is the one to start requiring, and it is almost always the screening log.
- Find one claim in a current project that is genuinely unknown, and write the full unknown statement for it: the price of being wrong, the price of waiting, the test that would settle it, and the number of days. Then say it in a meeting exactly like that.
- Pick one of the five commitments in Part 4 and put a date on it. One is a real change; five announced at once is a memo nobody reads. Add a revisit trigger to whatever decision you make this month, and diarise it.
Official sources covered
This session takes the working core of the reporting and certainty standards and uses them as leadership language rather than as methodology. The normative documents stay with their bodies. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Which part of the four-part confidence sentence do people most often drop, and why does it matter?
Claim, weight and basis all tend to survive into a memo in some form. The fourth part is the one that gets cut, and it is the one that makes the sentence honest: naming the test or the observation that would change your position turns a stated view into something evidence can act on, and it gives your team a specific thing to go and find.
2 · The ASA's 2016 statement on p-values sets out six principles. Which one does this course make you score business claims against?
The statement names business and policy decisions explicitly, which is why it belongs in a leadership session rather than only in a statistics one. Option C is the misreading the statement rules out directly: "P-values do not measure the probability that the studied hypothesis is true." Ask for the size of the effect and its precision instead of the threshold.
3 · Nine months on, someone asks how a decision was made. What must the audit trail contain for that question to be answerable?
A reference list shows what you cited, not what you looked at and rejected, which is where the hard question always lands. And disclosure alone protects nothing - the retracted Frontiers paper disclosed its AI use and still failed, because nobody checked the output. The six artifacts are things your team already produces; the discipline is keeping them.