Going looking for the source that disagrees 8 min live
Your extraction table from b5 is a record of what you found. It is not yet a record of what is out there, because you searched for your claim and the search obliged. The contradiction check is the deliberate opposite move: spend ten minutes trying to break each load-bearing claim, and write down what happened. It is the single biggest jump in the whole ladder, worth three claims on its own, and the reason is unglamorous. Most claims that fail this rule are not wrong. They are stated flat where the evidence only supports a qualified sentence.
LiveFour ways to hunt for disagreement5 min▶
Do all four on any claim the decision actually rests on. They fail differently, which is the point.
- Search for the opposite claim explicitly. Write the negation of your sentence and search for that. "Students, not institutions, buy lecture-notes tools" is a different query with a different result set, and it is the query a sceptical reader will run.
- Check whether the finding has ever been contradicted. Scite classifies citations as supporting, contrasting or mentioning, so you can see how a paper has been cited rather than only how often. Read the distribution before you read the result: across its corpus roughly 92.6% of citations are mentioning, 6.5% supporting and 0.8% contrasting. So a paper with no contrasting citations tells you almost nothing - that is the base rate, not a clean bill of health. A paper that does carry contrasting citations is the informative case, and you go read them.
- Ask the model for the strongest case against. Not "are there counterarguments" but "argue that this claim is wrong, name the study designs that would undermine it, and rank my sources by how much weight they can bear."
- Record how you handled each contradiction. One line per claim: what disagreed, how strong it was, and what you did about it. This line is what turns "I considered the counter-evidence" from an assertion into an audit trail.
Self-studyThe qualifying move, written out3 min read▶
The move everyone resists because it feels like hedging. It is the opposite: it is the version that carries information about the shape of the evidence.
"Institutions, not students, hold the budget for lecture-notes tools."
"The literature is split. Two student-survey sources report student-paid adoption; three procurement studies and one sector body report institutional budget holding. The institutional sources are larger and independently funded, so we plan for institutional procurement and treat student-paid as a secondary channel."
Notice what the second version gives a decision-maker that the first cannot: the size of the disagreement, which side the stronger studies fall on, and an explicit consequence. A hedge removes information. A qualifier adds it. If your qualified sentence feels vaguer than your flat one, you have hedged rather than qualified - go back and put the weighting in.
Replication, with the real numbers 7 min live
Rule 7 of the audit asks whether a claim holds up twice. People treat that as pedantry until they see the numbers from the large replication projects, which are not marginal. Across five well-run efforts the pattern repeats: roughly half to two thirds of findings replicate in the same direction, and the effects that do replicate come back materially smaller than the originals. Two operating consequences follow, and they are the whole of this session's second habit.
| Project | Scope | Replicated | Effect size on replication |
|---|---|---|---|
| Open Science Collaboration 2015 Science 349(6251):aac4716 | 100 studies | 97% of originals statistically significant vs 36% of replications; 68% retained significance when meta-analytically combined | About half the original magnitude; 47% of original effect sizes fell inside the replication's 95% CI |
| Camerer et al. 2016 Science 351(6280):1433-6 | 18 economics experiments | 11 (61%) in the same direction | About 66% of original |
| Camerer et al. 2018 Nature Human Behaviour 2:637-644 | 21 studies from Nature and Science | 13 (62%) | About 50% of original |
| Many Labs 1 Klein et al. 2014 | 13 effects across 36 samples, N=6,344 | 10 of 13 | Almost no variation attributable to sample or lab setting |
| Many Labs 2 Klein et al. 2018, AMPPS 1(4):443-490 | 28 findings, 125 samples, 15,305 participants, 36 countries | 15 (54%) significant in the original direction | Median comparable Cohen's d 0.60 originals vs 0.15 replications |
LiveThe two operating rules that fall out of this table4 min▶
- A single study is a hypothesis. Not a fact with a citation attached. If a claim your decision depends on rests on one study, that is a position you can take, but you take it consciously and you write it down as such. The Many Labs 1 finding sharpens this: the variation was not coming from which lab ran it or which sample they used, so "it was a different context" is a weaker excuse than it sounds.
- Effect sizes shrink on replication, so plan for the smaller number. Half the original magnitude in the Open Science Collaboration, about two thirds in the 2016 economics set, about half again in the 2018 set, and a median comparable d falling from 0.60 to 0.15 in Many Labs 2. If you build a business case on the published effect, you have built it on the optimistic end of the range. Model the replication-sized effect and see whether the decision still holds.
Neither rule requires you to dismiss single studies. It requires you to price them. "One study of 48 suggests X; we treat this as untested" is a legitimate sentence in a memo. "X is true (Smith 2024)" is not, if Smith 2024 is alone.
Self-studyHow to check replication in ten minutes3 min read▶
You are not running a replication. You are answering three questions about the evidence you already have.
- Is this one study or a body? Look for a systematic review, a meta-analysis, or a second independent team reporting the same direction. One team reporting it three times is still one team.
- Who paid, and who ran it? A vendor-funded preprint is not disqualified, but it needs an independent counterpart before it can carry a decision. This is exactly the failure mode of the tenth claim in the lab.
- How big was it? Record the sample size in the extraction row. A small sample plus a large published effect is precisely the combination the replication projects found shrinking.
The sentence that saves you later is the one you write now. When a claim rests on a single small study, the correct output is not a softer adjective. It is a marked unknown with a price and a test: what it would cost us to be wrong, and what evidence would settle it. That is the pattern b8 builds into prose and b10 ships in the memo.
Reading numbers properly 9 min live
You will be handed p-values, intervals and forest plots by models, vendors and colleagues, and asked to decide. The skill is not computation. It is knowing which question each number answers, and which question it cannot. Start with the distinction that fixes the most memos: statistical significance tells you about the presence of an effect, and effect size tells you whether the effect matters.
LiveEffect size versus significance, and confidence intervals5 min▶
Sullivan and Feinn (J Grad Med Educ 2012;4(3):279-282) make the point with a worked example worth memorising. The Physicians' Health Study ran on more than 22,000 subjects. Its result was significant at P < .00001. Its r² was .001 - the intervention accounted for about a tenth of one percent of the variation in outcome. Statistically overwhelming, substantively tiny. Nothing was wrong with the study; the error is only in the reader who stops at the p-value.
For effect size, Cohen's conventions are the common vocabulary: d = 0.2 small, 0.5 medium, 0.8 large. Treat them as conventions rather than laws, and always report the size alongside the verdict.
Confidence intervals are the better default, and the Cochrane Handbook is explicit about how to use them: report the interval together with the exact P value, and authors should not describe results as "statistically significant" or unduly rely on thresholds. In practice: read the whole interval, and ask whether the decision would be different at each end of it. If your decision flips somewhere inside the interval, the study has not settled your question no matter what its p-value says.
LivePublication bias, p-hacking and HARKing4 min▶
Publication bias. The Cochrane Handbook defines a funnel plot as "a simple scatter plot of intervention effect estimates from individual studies against a measure of each study's size or precision". The temptation is to read asymmetry as proof that small negative studies went missing. Do not. Asymmetry is not diagnostic - it has several possible causes, Egger's test has low power, and the rule of thumb is to use asymmetry tests only when there are at least 10 studies. So when a synthesis with six studies tells you the funnel looks fine, the honest reading is that nobody could tell.
P-hacking (Head et al., PLOS Biology 2015;13(3):e1002106) is the practice of trying analyses until one crosses the threshold. The behaviour list is what you look for in a methods section: interim peeking at results and continuing to collect, selective outcome reporting, post-hoc outlier rules, and adding or dropping covariates until the result appears. The detection method is elegant and worth knowing: compare how many reported p-values fall in the 0.045 to 0.05 bin against the 0.04 to 0.045 bin, because a pile-up just under the threshold is hard to explain innocently.
HARKing (Kerr, PSPR 1998;2(3):196-217) is presenting a post-hoc hypothesis as if it had been specified in advance. It is the reason pre-registration exists, and the reason "we found that..." and "we predicted and found that..." are very different sentences. When a paper's hypothesis fits its data suspiciously well and there is no pre-registration, note it in your extraction row rather than in your head.
The ASA's 2016 statement gives you the sentence to score every threshold claim against: "Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold." And the one that clears up the most confusion: "P-values do not measure the probability that the studied hypothesis is true."
| Check | What it asks | What it catches |
|---|---|---|
| Effect size | How big is it, in Cohen's terms or in the study's own units? | The significant-but-trivial result: P < .00001 with r² = .001 |
| Confidence interval | Would my decision change at either end of the interval? | Precision dressed up as certainty; reliance on thresholds |
| Sample and population | How many, and were they the people I am deciding about? | Small-n findings, and results imported from the wrong population |
| Replication | Has an independent team found this in the same direction? | The single study treated as settled; the shrinking effect |
| Publication bias | Are there at least 10 studies before anyone reads asymmetry? | Funnel plots over-interpreted; missing small negative studies |
| P-hacking signals | Peeking, selective outcomes, post-hoc outliers, covariate churn? | Results that exist because the analysis was run until they did |
| HARKing | Was the hypothesis pre-registered, or fitted afterwards? | Exploratory findings presented as confirmed predictions |
Self-studyAsking a model to run the checks on a paper3 min read▶
These prompts work because they ask for structure and for absence, not for a verdict. Verify what comes back against the paper itself - an extracted number is a claim like any other.
"For this paper, give me a table with: reported effect size and its units, the confidence interval, the exact p-value, the sample size, the population studied, the funder, and whether the study was pre-registered. Put 'not reported' where it is not reported, and quote the page or section for each cell."
"Read the methods section only. List any of the following you can see evidence of: interim analyses with continued collection, outcomes reported that were not pre-specified, outlier rules introduced after the data, covariates added or dropped between models, or a hypothesis that appears fitted to the result. Quote the sentence for each. If you see none, say so explicitly."
The "not reported" cells are the useful part. A missing interval or a missing funder is information about how much weight the source can bear, and it belongs in your extraction table rather than in a footnote you never write.
Rungs 4 and 5, live ★ 16 min · everyone runs it
This is the session that owns the two biggest jumps in the ladder. The lab below starts where b5 left you: sharp question, source tiering and extraction table all on, 4 of 10 claims surviving, 22 sources in play. Turn on the contradiction check and you go to 7. Add the replication check and you go to 9. Then read the one that still fails, because the audit is right about it and will stay right about it all the way to b10.
Start at 4 of 10. Read the failure rules on the six failing rows and sort them: which are waiting on a qualifier, and which are waiting on evidence that may not exist?
Turn on Contradiction check: 7 of 10. Three claims come back at once, the biggest single jump in the ladder, and not one of them needed a new source.
Read those three survivors as sentences. Each one moved from flat to qualified. That is the whole mechanism of rung 4.
Add Replication check: 9 of 10. Two more claims survive because a second independent source pointed the same way, and the sources stay at 22 - this rung costs judgement, not reading.
Read the claim that still fails: checked, and it does not replicate. Our meeting-notes engine transferring to lecture audio rests on a single vendor-funded preprint with n=48 and nothing replicating it. Do not fix it. b8 teaches you to write it.
Try it yourself - this week ◐ 35-45 min total
- Take the three load-bearing claims from your own current question and run the contradiction hunt on each: search the negation, check how the key paper has been cited, ask a model for the strongest case against, and write your one-line handling record. Keep the four lines.
- Rewrite one flat claim you have already shipped as a qualified claim with the weighting visible. Then read both aloud and note which one a sceptic could flip in a sentence.
- Find a study your work depends on and answer three questions in writing: one study or a body, who funded and ran it, and how big was the sample. If it is one small study, draft the "we treat this as untested" sentence now.
- Run the checks table over one paper end to end. Record every "not reported" cell, and use the 10-study rule on any publication-bias claim the paper makes.
Official sources covered
This session works from the published replication record and the standing statistical guidance. Every figure on this page comes from the sources below and is cited where it appears. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · In the Open Science Collaboration's 2015 replication of 100 studies, 97% of the original findings were statistically significant. What share of the replications were?
36% of replications reached statistical significance against 97% of originals, and the replication effects came back at roughly half the original magnitude. 68% is a different figure from the same paper: the share retaining significance when the original and replication were combined meta-analytically. The operating rule is to plan for the smaller number.
2 · A synthesis of six studies reports a symmetrical funnel plot and concludes there is no publication bias. What is the correct response?
The Cochrane rule of thumb is to use funnel-plot asymmetry tests only with at least 10 studies. Below that, a symmetrical plot is uninformative rather than reassuring. Asymmetry is also not diagnostic in the first place, and Egger's test has low power, so the finding is "unknown", not "clean" - and not grounds to throw out the whole synthesis.
3 · A vendor deck says "the result was significant at p < 0.05, so the product works". Which ASA principle answers it most directly?
That is the ASA principle to score threshold claims against. Option A is the misreading the ASA explicitly corrects: p-values do not measure the probability that the studied hypothesis is true. Ask instead for the effect size, the confidence interval, the sample and whether anyone has found it twice.