How this track works
Ten sessions, one running decision: should Cadence, an AI meeting-notes product, build a lecture-notes product for higher education next year? It is a decision with the shape of most real research questions - part market, part evidence, part regulation, part unknown - and it is small enough to actually finish. Tonight the loop and the audit. b2 turns the decision into falsifiable claims. b3 is search strategy and source tiering. b4 is screening at scale. b5 builds the extraction table. b6 is the evidence lab in full. b7 is contradictions and statistics. b8 is synthesis. b9 is interviews and qualitative coding. b10 assembles the memo with its audit trail. Every session leaves an artifact that goes into that memo.
Question, gather, screen, extract, challenge, synthesise 7 min live
AI compresses two of these six steps dramatically and leaves the other four exactly as hard as they were. Question: turn a decision into claims that could be false. Gather: find sources on purpose rather than accepting what ranked first. Screen: apply inclusion rules written before you looked. Extract: record what each source actually says, with its numbers. Challenge: go hunting for what disagrees, and for whether the finding replicates. Synthesise: write it with confidence language the evidence can bear. The compression is in gather and extract. The judgement is everywhere else, and the audit only tests the judgement.
LiveThe five levers of this track4 min▶
Each lever is a session, and each one unlocks a specific set of claims in tonight's audit. They are also, in order, the five reasons research gets dismantled in a review:
- Sharp question - a claim nothing could falsify cannot be supported, however many sources you pile behind it. "AI is transforming education" is not a finding. (b2)
- Source tiering - if you accept whatever ranked first, your evidence base is a popularity sample. The peer-reviewed study and the regulator's own guidance rarely rank first. (b3)
- Extraction table - a claim that carries a number needs a row recording the source, sample and measure behind it. Unrecorded numbers drift with every retelling. (b5)
- Contradiction check - the source that disagrees exists whether or not you looked. If you did not, your claim is stated flat where it should be qualified. (b7)
- Replication check - one small study is a hypothesis. Building a decision on it is a choice you should make consciously, and say out loud. (b7)
There is a sixth toggle: Add 40 sources. It is not a lever. Turn it on when the audit is at its best and watch what happens - we spend all of b4 on why.
Self-studyWhy the question is the whole ballgame3 min read▶
Most weak research is not badly sourced. It is badly asked - and then sourced beautifully. Compare:
"Explore whether the education market is an opportunity for AI note-taking, and summarise the key trends and drivers."
"We will decide in two weeks whether to build a lecture-notes product for higher education. Test these claims, each of which could be false: (1) institutions, not students, hold the budget; (2) recording lectures needs documented consent in both target markets; (3) procurement takes 6 to 12 months; (4) our meeting-notes engine transfers to lecture audio without retraining. For each, tell me the strongest evidence for, the strongest evidence against, and what you could not find."
The second version can come back "claim 4 is unsupported" - which is the most valuable sentence in the whole project, and one the first version can never produce. Notice the last clause too: asking explicitly what could not be found is how you convert a model's silence into information rather than into false comfort.
How a hostile reviewer reads your memo 6 min live
The audit is not a grade. It is a simulation of the person who will read your work looking for the one soft claim - a board member, a supervisor, a client's data team, a peer reviewer, a regulator. That person applies roughly seven rules, in a fixed order, and they are the rules tonight's lab implements. Learn them and you can pre-empt every one of them before you send anything.
LiveThree failures that are fatal, four that are a qualifier away4 min▶
Not every red row means delete the claim. Knowing which is which is what makes you fast:
- Fatal - rewrite or drop: not decision-scoped (rule 1), no support in your set (rule 2), tier-4 support only (rule 3). These are not fixable by wording. You need a different claim or different evidence.
- Fixable by work: unrecorded number (rule 5) - go and record it. Retracted source (rule 4) - screen properly and remove it.
- Fixable by honesty: unhandled contradiction (rule 6) becomes "the literature is split; the stronger studies point this way." Thin evidence (rule 7) becomes "one study of 48 suggests X; we treat this as untested."
- The honest unknown: occasionally a claim fails rule 7 and no replication exists at all. The correct output is not a softer sentence - it is "we do not know, and here is the test that would tell us." One of tonight's ten claims is exactly this, and it stays red all the way to b10 on purpose.
Self-studyAsking a model to attack its own answer3 min read▶
Models tend to agree with the framing they are handed, which is exactly wrong for research. Two prompt habits push against it:
"Here is my claim and the sources I am relying on. Argue the strongest case that this claim is wrong. Name the specific sources or study designs that would undermine it, tell me what evidence would be missing if it were false, and rank my sources by how much weight they can actually bear. Do not soften the critique."
"What did you NOT find? List, separately: (a) what you searched and found nothing on, (b) claims you could only support with vendor or promotional material, (c) where your corpus likely has a coverage gap for this question."
The second one is the underrated habit. A summary that reads as complete is not evidence of completeness - and asking for absence explicitly is the cheapest way to find out where the floor is thin. Verify what comes back, though: an absence claim is still a claim.
Audit ten claims, live ★ 12 min · everyone runs it
Below is the Cadence education question as most people would first answer it: a model asked a broad question, nine sources it happened to surface, and ten claims. Zero of the ten survive the audit. That is not a trick - read the failure rule on each row. Three are unfalsifiable, six have no support inside the source set you actually gathered, one rests on nothing but a vendor blog. Then start turning levers on and watch specific claims come back to life.
All levers off: 0 of 10, 9 sources in play. Read every failure rule before you touch anything.
Turn on Sharp question. The three unfalsifiable claims are replaced by claims that can be tested - one immediately survives. 1 of 10.
Add Source tiering: 9 sources becomes 22, and the peer-reviewed and regulator sources arrive. 2 of 10.
Add Extraction table (4 of 10), then Contradiction check (7 of 10). Note how many claims were only ever missing a qualifier.
Add Replication check: 9 of 10. Now read the one that still fails - and notice the audit is right about it.
The climb, and the 40-source trap ★ 10 min · explore
The whole ladder, computed in front of you: 0 → 1 → 2 → 4 → 7 → 9 surviving claims. Then the row that will change how you work. With the audit at its best, turn on Add 40 sources: your reading pile goes from 22 to 62, and surviving claims drop from 9 to 8 - because one of those 40 hits is retracted, and it lands in the support for a claim that was previously clean. Nearly three times the reading, one claim worse off.
Read the rungs as sessions: question (b2), tiering (b3), extraction (b5), contradiction and replication (b7).
Note the two biggest single jumps: contradiction (+3) and replication (+2). Challenge work pays more than gathering work.
Now the anti-lever. Predict what it will do before you click. Most people expect a small gain.
Read why the claim died. Then say the rule out loud: breadth without screening is not more evidence, it is more risk.
"Let's just look at more sources" is the most reflexive move in research and one of the least effective. Volume helps only after tiering and screening are in place - otherwise every extra hit is another chance to import something retracted, duplicated, promotional or off-population, and every one of those is a hole a reviewer can put a finger through. The professional version of thoroughness is not a bigger pile. It is a defined search, written inclusion rules, and a record of what you excluded and why - which is exactly what b3 and b4 build.
Try it yourself - this week ◐ 30-40 min total
- Run the ladder until you can predict which claim revives at each rung. Then turn the anti-lever on from a clean audit and write one sentence explaining the loss to a manager who thinks more sources is always better.
- Take a memo, deck or paper you wrote recently and list its load-bearing claims. Run the seven rules by hand on each. Mark every failure as fatal, fixable by work, or fixable by honesty.
- Rewrite one unfalsifiable claim you have actually shipped into a claim that could be false, then say what evidence would settle it.
- On any current question, run the absence pass from Part 2 and keep the answer. It is the start of the "what we could not find" section your b10 memo will need.
Official sources covered
This track teaches the working core of the established evidence-synthesis standards and the current guidance on AI in research, with a live audit on the key pages. The normative documents stay with their bodies and are linked each session. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Which two steps of the loop does AI genuinely compress?
Gathering and extraction go from hours to minutes, and that is a real gain. Framing the question, writing inclusion rules, hunting for disagreement and calibrating confidence language are unchanged - and the audit only tests those.
2 · A claim fails the audit with "unhandled contradiction". What is the usual correct fix?
Contradiction is fixable by honesty, not by volume. Piling on agreeable sources is how you get an unfalsifiable memo. Naming the split and weighting it by study quality is both more defensible and more useful to whoever decides.
3 · From a clean 9 of 10, adding 40 more sources drops the audit to 8. Why?
Unscreened breadth is how retracted, duplicated or off-population material enters a table. The audit does not care how much you read; it cares what your claims rest on. Volume becomes useful only after tiering and screening are in place.