How the leader track works
Six sessions, one running decision: should Cadence, an AI meeting-notes product, build a lecture-notes product for higher education next year? It has the shape of most real questions that land on a research leader's desk - part market, part evidence, part regulation, part genuinely unknown - and it is small enough to actually finish. Tonight is the map: which parts of the pipeline AI compressed, which it did not touch, and the three decisions you have to make about your own team this quarter. a2 is the evidence standard you publish so everyone means the same thing by "strong evidence". a3 is the failure catalogue, with the documented numbers behind each failure. a4 is consent and governance for participant material and for disclosure. a5 is the stack - what each tool can and cannot do, and where your data goes. a6 is deciding on fast evidence: confidence language, the audit trail, and saying "we do not know yet" without losing the room.
Which stages compress, and which do not 7 min live
Research runs in six stages: frame the question, gather sources, screen them against rules, extract what each one says, challenge the result, then synthesise it with confidence language the evidence can bear. AI compressed two of those six dramatically. Gather and extract went from days to minutes, and that gain is real - do not let anyone talk you out of it. The other four moved not at all. Framing a question so it could be wrong, writing screening rules before you look, going hunting for what disagrees, and calibrating how confident the write-up is allowed to sound: all still human, all still slow, and all still the places a review comes apart. The trap is that the two fast stages are the visible ones, so a compressed pipeline looks like a finished one.
LiveThe recall problem, in two sets of numbers4 min▶
Set fabrication aside completely for a moment - assume every reference your team brings back is a real paper. There is still a coverage problem, and it is measured:
- Chelli et al., JMIR 2024;26:e53164 tested how much of a real systematic review's reference list the tools could find. Recall was 11.9% for GPT-3.5, 13.7% for GPT-4, and 0% for Bard. Read that as a leader: even when the tool is not making things up, it is surfacing a small fraction of the literature that a proper search would have found - and it will present that fraction in complete, confident prose.
- Bramer et al., Systematic Reviews 2017;6:245 looked at 58 systematic reviews and measured single-database recall: Embase 85.9%, MEDLINE 78.8%, Web of Science 68.1%, Google Scholar 34.4%. All four combined reached 98.3%. Their conclusion was that a proper search should use at least those four as a minimum.
Put the two together and you have the leader's version of the point: coverage is a design decision, not a property of the tool you happened to open. If nobody on your team can tell you which sources were searched and which were not, the recall number for that project is unknown - and unknown recall is not the same as good recall.
LiveWhat the compression is genuinely worth3 min▶
This is not a session about distrusting the tools. Gather and extract were the two most tedious stages in the pipeline and they are now cheap. That matters in three specific ways:
- More questions get asked. Work that was never worth a two-week review now gets one. That is a real expansion of what your function can cover.
- Extraction becomes repeatable. Pulling sample, measure and number into a table used to be the job nobody wanted. Cheap extraction means you can insist on the table, every time, which is the single biggest defensibility upgrade available to most teams.
- Time moves to challenge. The hours you no longer spend gathering can be spent hunting for disagreement - the stage that pays most and gets cut first.
The failure mode is not using the tools. It is banking the saving as speed instead of spending it on the four stages that did not compress. That choice is a management choice, not a researcher choice, which is why it lands on this page.
Self-studyWhy the four hard stages stayed hard3 min read▶
Each of the four resists compression for a different structural reason, and knowing which is which stops you from waiting for a tool that is not coming:
- Question. Only you know what decision is being made, when, and what would change it. A model handed a vague brief will answer the vague brief beautifully. "Is education an opportunity?" cannot fail; "institutions, not students, hold the budget" can.
- Screen. Inclusion rules are a commitment made before you see the results. Anything written afterwards is a rationalisation of what you already found, no matter how principled it sounds.
- Challenge. Finding the source that disagrees requires wanting to find it. A system optimised to be helpful is not the right instrument for that, and asking it to attack its own answer helps only if someone checks the attack.
- Synthesise. Confidence language is an accountability statement. "The evidence suggests" versus "one study of 48 suggests" is the difference between a defensible memo and a retraction, and no tool can pick that for you because no tool carries the consequence.
Why "we finished the review in a day" is not by itself good news 6 min live
A day is a fact about the process, not about the evidence. It tells you the two compressible stages compressed, which you already knew. It tells you nothing about whether the question could have come back negative, which sources were never searched, whether a number was checked against its page, or whether anyone went looking for the study that disagrees. So replace the question. Ask: what would a hostile reader break first? Not a rude reader - a competent one who wants to find the single softest claim, because that is what a board member, a client's data team, a supervisor, a peer reviewer or a regulator actually does. That reader applies roughly seven rules, in a fixed order, and every one of them is answerable in advance.
LiveThe seven rules, at a leader's altitude4 min▶
Your researchers learn these as a working audit in the b-track. You need them as a review checklist you can run out loud in a meeting, in this order:
- Is it decision-scoped? If nothing could falsify the claim, no amount of evidence supports it either.
- Is there any support in our own source set? A claim whose sources are not in the set is an assertion with a citation-shaped gap behind it.
- Is the support good enough? Vendor blogs and forum posts, on their own, cannot carry a decision - however many of them agree.
- Is anything retracted? Breadth without screening is exactly how withdrawn work enters a table.
- Is the number recorded? A figure with no source, sample or page will drift with every retelling until it is simply wrong.
- Was disagreement handled? A flat claim laid over a split literature is the easiest thing in the world to break.
- Does it hold up twice? One small study is a hypothesis. Building on it is allowed; doing so silently is not.
Three of those are fatal and cannot be fixed by rewording - not decision-scoped, no support in the set, weakest-tier support only. Two are fixable by work: record the number, screen out the retraction. And two are fixable by honesty: qualify the contradiction, name the thin evidence as thin. Knowing which bucket a failure lands in is what makes you fast in a review rather than slow and anxious.
LiveThe ladder - the shape of the problem, in one row of numbers3 min▶
The b-track runs a live audit on the Cadence decision: ten claims, a teaching corpus of sources, and a rule engine that applies the seven rules. Your researchers turn on one discipline at a time and watch the number of surviving claims climb:
| Discipline turned on | Claims surviving | Sources read |
|---|---|---|
| Nothing - a broad question, whatever surfaced | 0 of 10 | 9 |
| + sharp, falsifiable question | 1 of 10 | 9 |
| + source tiering | 2 of 10 | 22 |
| + extraction table | 4 of 10 | 22 |
| + contradiction check | 7 of 10 | 22 |
| + replication check | 9 of 10 | 22 |
| then + 40 more unscreened sources | 8 of 10 | 62 |
Three leader-grade readings of that table. First, zero of ten survive at the start - and that starting point is what a competent, well-intentioned, fast piece of work looks like before any discipline is applied. Second, the two biggest jumps are contradiction (+3) and replication (+2): challenge work pays more than gathering work, and challenge work is the first thing a deadline removes. Third, and the one to quote in your next planning meeting: adding 40 more sources to an already-good review made it worse - 62 sources read, 8 claims surviving instead of 9, because one of the bulk hits was retracted and it landed in the support for a claim that had been clean. The tenth claim never survives at all, by design, and the correct output for it is "we do not know, here is what it would cost to find out".
The Tow Center for Digital Journalism, reported in Columbia Journalism Review on 2025-03-06, ran 1,600 queries across eight AI search tools. Collectively the tools were wrong on more than 60% of queries. The spread matters more than the average: Perplexity was 37% incorrect, ChatGPT Search 67%, Grok-3 94%. And the finding a research leader should carry into every procurement conversation: the paid tiers performed worse, because they were more willing to produce a confident answer than to decline. That is not a bug someone forgot to fix - it is what happens when a product is tuned to be useful. Your team is not being lied to; they are being served fluency where they needed abstention. Which is exactly why "it came back with an answer" is never evidence that an answer existed.
Self-studyThree questions to ask in every research review2 min read▶
You do not need to audit the work yourself. You need three questions that make the audit visible, and they take about ninety seconds:
- "Which claim here is the weakest, and why?" A team that cannot answer this has not read its own work adversarially. A team that answers instantly is doing the job.
- "What did we search, and what did we not search?" This converts unknown recall into known recall. It is also the question that surfaces "we used one tool" before a client does.
- "What could we not find?" Absence is information, and it never appears in a summary unless someone asks for it explicitly. Treat a confident memo with no "what we could not find" section as incomplete rather than clean.
Ask these three consistently for a month and the behaviour changes upstream - people start preparing the answers before the meeting, which is the whole point.
Three decisions a research leader has to make now 8 min live
None of these three can be delegated, because all three are about what your function is willing to put its name on. They are also all writable in a single page each, this week, which is the standard to hold yourself to. Waiting for the tooling to settle is not a plan - the tools change monthly and the rules underneath them do not.
LiveDecision 1 - what may be delegated to AI, and what never may3 min▶
Write the line explicitly, so nobody has to guess in a rush. The defensible split follows the compression map from Part 1 almost exactly:
| Delegate freely | Delegate with verification | Never delegate |
|---|---|---|
| Finding candidate sources; drafting search strings; deduplicating; translating; reformatting | Extracting numbers into a table; summarising a paper you will read; proposing counter-arguments | Deciding what the question is; writing inclusion rules; deciding what counts as strong evidence; choosing the confidence wording |
| Nothing load-bearing rests on the output | Output enters the record only after a human checks it against the page | These carry the accountability, so they stay with a named person |
Two rails worth adding on the same page. Confidential participant or client material does not go into a general-purpose tool - a4 covers why stripping names is not sufficient. And every number that reaches a slide has a human who can point to the page it came from. If you only write one sentence this quarter, write that one.
LiveDecision 2 - what evidence standard the team shares3 min▶
Right now, "strong evidence" almost certainly means something different to each of your researchers, and the disagreement only surfaces when a client pushes back. A shared standard is a one-page document with four parts, and a2 builds all four in detail:
- Source tiers. Four tiers, from peer-reviewed synthesis and regulator text down to vendor blogs and forum posts, with one hard rule: a claim resting only on the bottom tier fails. Tag the tier at the moment of capture, never afterwards.
- A certainty word list. Agree what "the evidence indicates", "one study suggests" and "we do not know" mean, and never let them float. The formal version of this is GRADE, and it exists precisely because clinical teams had the same argument first.
- Extraction columns. Every claim carrying a number needs a row: source, year, sample, measure, number, page, funder, population. No row, no number in the deck.
- Design labels are not appraisals. "It is a systematic review" is not a quality claim - a2 covers the published critique of the evidence hierarchy that makes this precise.
The point of publishing it is not bureaucracy. It is that a team with a shared standard argues about evidence, and a team without one argues about taste - and the second argument has no ending.
Self-studyDecision 3 - what gets written down as the audit trail3 min read▶
The test is simple and unforgiving: six months from now, someone challenges this decision and the people who made it have left. What is on file? A defensible trail is short. Aim for one page per project, captured as the work happens rather than reconstructed afterwards:
- The claims, written so each one could have come back false.
- What was searched - which databases and tools, which queries, and what was deliberately not searched.
- The inclusion rules, with their date, so it is visible they preceded the results.
- The extraction table, one row per source, with the page reference for every number.
- Disagreement found, and how each claim was qualified in response.
- Where AI was used, on which stage, and who verified the output. Disclosure norms differ by venue and a4 covers that divergence.
- What we could not find, and the one thing we would test next.
Notice what this list does for you beyond compliance: it is the same seven items the hostile reader asks about, so keeping it is not overhead on top of the work - it is the work, recorded once instead of defended repeatedly.
Try it yourself - this week ◐ 30-40 min total
- Take the most recent piece of research your team shipped. List its load-bearing claims, then run the seven rules from Part 2 on each one by hand. Mark every failure as fatal, fixable by work, or fixable by honesty - and count how many are fatal.
- For that same piece, answer "what did we search and what did we not search?" in writing. If nobody can answer it, you have found your first process gap and it is a cheap one to close.
- Draft Decision 1 as a single page: the three columns of delegate freely, delegate with verification, and never delegate. Circulate it as a draft and see which row people argue about - that row is where your real risk is.
- Run the three review questions from Part 2 in your next research review, exactly as written, and note how long the answers take. That timing is your baseline for a2.
Official sources covered
This track teaches the working core of the established evidence-synthesis standards and the current published evidence on AI in research. Normative documents stay with their bodies and are linked each session. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Which two stages of the research pipeline did AI genuinely compress?
Gathering and extraction went from days to minutes, and that gain is real. Framing the question, writing inclusion rules before you look, hunting for disagreement and calibrating confidence language are unchanged - and those four are where a review actually comes apart.
2 · Chelli et al. found recall of a real systematic review's references at 11.9% for GPT-3.5 and 13.7% for GPT-4. What does that tell a research leader?
Recall and fabrication are different failures. Recall is about coverage: what the tool never saw. Bramer's single-database figures make the same point from the other direction - coverage is a design choice, and unknown recall is not the same as good recall.
3 · In the Tow Center / CJR study of 1,600 queries across eight AI search tools, what happened with the paid tiers?
The tools were collectively wrong on more than 60% of queries, ranging from Perplexity at 37% incorrect to Grok-3 at 94%, and the paid tiers were worse rather than better. Fluency is the product; abstention is what research needs. "It came back with an answer" is never evidence that an answer existed.