Where this sits in the track
Everything before this session produced inputs. This session produces a verdict, and it is the one you should run before writing the memo rather than after someone else has read it. The audit is deliberately unkind: it assumes a reader who is looking for the weakest claim, because that reader exists in every board pack, every viva, every client review and every peer review you will ever be part of.
None of this is invented 8 min live
The audit is a compression of practices that evidence-synthesis communities spent decades formalising. Knowing the lineage matters for two reasons: it tells you what to read when a rule bites you, and it lets you defend the rule to someone who thinks you are being fussy. Each rule below traces to a standard, and each standard exists because people got burned.
| Audit rule | Where it comes from | What it is protecting you from |
|---|---|---|
| Decision-scoped claim | Structured question formats used across evidence synthesis - population, comparison, outcome, context spelled out before searching | Research that cannot come back negative, and therefore cannot inform a decision |
| Support exists in your set | Reporting standards that require the search to be documented - databases, dates, strategy - so a reader can see what was and was not looked at | Assertion dressed as finding; a claim whose evidence you assumed rather than gathered |
| Support is good enough | Study-design hierarchies and risk-of-bias assessment: a randomised trial and a vendor blog are not interchangeable inputs | Promotional material carrying a decision, and single-case anecdote treated as pattern |
| Nothing retracted | Screening and deduplication stages in review protocols, plus retraction-watch practice | Citing work that the record has already withdrawn - a credibility loss you cannot argue your way out of |
| Numbers are recorded | Data-extraction forms: one row per study, with sample, measure, effect and location in the source | Figures that drift, lose their denominators, or turn out to be from a different population |
| Disagreement handled | Systematic-review practice of seeking all eligible evidence, not just confirming evidence, and reporting heterogeneity | A flat claim over a split literature - the single easiest thing for a reviewer to break |
| It holds up twice | Certainty grading, which downgrades confidence for imprecision and small single studies, and the replication literature that motivated it | Building a strategy on one small study that no one has been able to reproduce |
LiveThe triage habit: fatal, work, honesty4 min▶
A red row is not a verdict on you. It is a routing instruction. Sort every failure into one of three buckets and the whole audit takes minutes instead of an afternoon:
- Fatal (rewrite the claim or abandon it): not decision-scoped, no support inside your set, tier-4-only support. No amount of careful wording fixes these - the claim itself or the evidence base has to change.
- Work (go and do a specific task): unrecorded number - extract it properly; retracted source - screen and remove; missing population match - find a source in the right population or narrow the claim to the one you have.
- Honesty (write the sentence you were avoiding): unhandled contradiction becomes a qualified claim; thin evidence becomes a stated hypothesis. Both are more defensible than the flat version, and both usually make you more useful to the decision-maker, not less.
The order to work in: fatal first, because those may change what you need to gather; then work; then honesty last, when the evidence has stopped moving.
LiveThe confidence sentence4 min▶
Most researchers have two settings: flat assertion and mush. The useful middle is a sentence with four parts - the claim, the weight, the basis, and what would change it:
"Institutions hold the budget for note-taking tools, so we should sell top-down."
"There are indications that budget ownership may vary, and further research could be beneficial to clarify the picture."
"Institutions most likely hold the budget - moderate confidence, based on national IT-spend statistics and twelve buyer interviews with published methods, against one student survey pointing the other way. We would revise this if a per-seat student-paid competitor showed institutional-scale revenue."
The last clause is the professional tell. Naming what would change your mind proves you have a position rather than a preference, and it gives your reader something to watch for instead of something to argue about.
Self-studyWhich claims can never be rescued3 min read▶
One of tonight's ten claims - that the meeting-notes engine transfers to lecture audio without retraining - fails at rule 7 and stays failed even with the replication check on. That is deliberate, and it is the most realistic thing in the lab. The support is a single vendor-funded preprint with a sample of 48 and nothing independent behind it. Three tempting wrong moves:
- Soften the language and ship it anyway. "Early evidence suggests our engine transfers well" - which is how an untested assumption enters a roadmap wearing research clothes.
- Add supporting sources until it looks settled. Any pile of vendor material you assemble here supports the claim you already prefer. That is not evidence, it is decoration.
- Drop the claim silently. Also wrong: the decision genuinely depends on it. Removing it hides a risk rather than resolving it.
The correct output is a fourth option: mark it unknown, quantify what not knowing costs, and specify the test. "Untested - one vendor-funded preprint, n=48. A two-week internal benchmark on 20 real lecture recordings would settle it, and until it does we should not price on transfer working." That sentence is worth more to a board than any of the ten survivors.
Four tiers, no arguments 5 min live
Rule 3 needs a shared tier system or it becomes a taste fight. The version below is deliberately crude enough that two colleagues will classify a source the same way in under ten seconds, which matters far more than theoretical elegance. It is what the lab's corpus is tagged with.
Self-studyFunder, date, population - the three checks people skip3 min read▶
Tier alone is not enough. Three attributes decide whether a technically-good source is good for your claim:
- Funder and interest. A vendor-funded study is not worthless, but it belongs in a different mental column - and it needs a non-conflicted source alongside it before it carries anything.
- Date, against the thing that changes. A 2019 study of software procurement may still be fine; a 2019 study of model capability is archaeology. Ask what has changed in the mechanism, not just how many years passed.
- Population match. The most common silent failure in applied research: excellent evidence about a different group. Higher education is not K-12, and corporate L&D is neither. If the population differs, either narrow your claim or say you are extrapolating.
Add all three as columns in the b5 extraction table if they are not there yet. They cost seconds at capture and save the memo later.
Full audit, then triage every red row ★ 13 min · everyone runs it
Start with all five levers ON - the state your work should be in by now - and read the two failures first. Then switch levers OFF one at a time and watch which claims collapse. Working backwards like this is the fastest way to feel what each practice is actually buying you, and it is a much better teacher than climbing.
Five levers on: 9 of 10. Read the survivor rows too - each one names the tier and the source ids carrying it. That is what a defensible claim looks like.
Turn Contradiction check off. Three claims die at once. Note the wording of each failure: the disagreement existed the whole time.
Turn it back on, then turn Extraction table off. Different claims die - the ones carrying numbers. Say which bucket that failure is in (work, not honesty).
Turn Source tiering off. Watch the source count fall from 22 to 9 and most of the memo evaporate. This is the rung people skip.
Finally turn Sharp question off. Three claims become unfalsifiable slogans. Notice they read like the executive summary of a bad report.
Pricing "can we look at more sources" ★ 9 min · explore
The ladder computed live - 0 → 1 → 2 → 4 → 7 → 9 - and then the anti-lever. From the best state, adding 40 more hits takes your reading pile from 22 to 62 and your surviving claims from 9 to 8. The cause is specific and worth memorising: one of the 40 is retracted, it lands in the support for a claim that was clean, and the audit catches what unscreened breadth imported.
Look at where the ladder's big jumps are: contradiction (+3) and replication (+2) beat everything the gathering levers gave you.
In the lab above, turn Add 40 sources on from the 9-of-10 state. Read the new failure row and name the mechanism.
Now turn Source tiering off while the 40 are still in. Watch how much worse untiered breadth is than untiered narrowness.
Write the sentence you will use next time someone asks for a wider sweep. It should mention inclusion rules, not effort.
The reviewer who ends your credibility is rarely reading closely. They are scanning for one thing they recognise: a number without a denominator, a citation they know was withdrawn, a flat claim on a topic they know is contested. Every one of those is a screening or extraction failure, not an intelligence failure - which is the good news, because screening and extraction are exactly the parts of research you can now do at speed. Use the speed on the discipline, not instead of it.
Try it yourself - this week ◐ 35-45 min total
- Run the backwards drill on your own current question: assume you did everything right, then remove one practice at a time and write down which of your claims would collapse. That list is your risk register.
- Take your b5 extraction table and add the three skipped columns - funder, date-against-mechanism, population match. Re-check every claim that depends on a row where any of the three is uncomfortable.
- Write one calibrated confidence sentence, with all four parts, for the claim your decision leans on hardest. Include what would change your mind.
- Find one claim of yours that should be marked unknown. Write the unknown version, including the test that would settle it and what not knowing costs. Bring it to b8 - it belongs in the synthesis, not in a drawer.
Official sources covered
This session teaches the working core of evidence-synthesis practice as a working researcher applies it - reporting discipline, study-design hierarchy, risk of bias, certainty grading - compressed into an audit you can run in minutes. The normative documents stay with their bodies and are linked in the track's source map. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · A claim's only support is a vendor blog and a forum thread. Which triage bucket?
Tier-4-only support is one of the three fatal failures. Wording cannot rescue it: promotional material is evidence of what someone wants you to believe, so you either find independent evidence or you stop making the claim.
2 · Which part of the calibrated confidence sentence do people most often leave out?
Naming the disconfirming evidence you would accept is what separates a position from a preference. It also gives the decision-maker something concrete to monitor, which is often the most actionable line in the whole memo.
3 · One claim fails rule 7 and no replication exists. What is the correct output?
Softening smuggles an untested assumption into a roadmap; deleting hides a risk the decision actually depends on. The unknown, priced and paired with a test, is frequently the most valuable sentence you will write.