Screening at scale

Six lines, written before the first search result loads. For the Cadence question they read something like this:

• Population - higher education students and institutions. Not K-12, not corporate learning, not general consumers. Name what is out as well as what is in.
• Date window, with a stated reason - 2021 onward, because the procurement and consent landscape we care about post-dates remote-teaching normalisation. The reason matters more than the year: a window with no rationale is a preference.
• Study or source type - peer-reviewed studies, regulator or ministry guidance, and institutional procurement documents. Vendor material may be read for context and marked as such, never as support.
• Language - English plus our two target markets. Write it down even when it is obvious, because it is a real limitation of your evidence base and you will need it in the write-up.
• Outcome or measure required - the source has to report something about adoption, budget authority, consent practice or transcription performance. A source with an opinion and no measure does not clear the bar.
• Exclusions - the mirror of the above, stated positively so a second person could apply them: no K-12-only samples, nothing before 2021, no vendor material as support, no source without a measure.

Now the test. Once written, the rule set is allowed to change - but only in one direction: you may make it stricter, and you record the change and the date. Loosening a rule after you have seen the pile is how an excluded source becomes an included one because you liked its conclusion.

PRISMA 2020 (Page MJ et al., BMJ 2021;372:n71) is a reporting guideline, not a quality-appraisal tool - it tells you what to disclose, not whether a study is any good. Its checklist has seven sections with 27 items, some with sub-items: Title (1), Abstract (2), Introduction (3-4), Methods (5-15), Results (16-22), Discussion (23), Other information (24-27). New in the 2020 revision: certainty assessment, data and code availability, and a separate abstract checklist.

The part worth stealing for commercial work is the flow diagram, and the detail people get wrong is its shape. It has three bands, not four:

• Identification - what you found, and what you removed as duplicates before screening anything.
• Screening - records screened on title and abstract, records excluded, reports sought for retrieval, reports not retrieved, and reports assessed for eligibility with the exclusions and their reasons. The 2009 diagram had a separate "Eligibility" band; in 2020 it was removed and the eligibility assessment became a box inside Screening.
• Included - the studies that made it, which is the set your claims are allowed to rest on.

There are four template variants (new versus updated review, databases-only versus with other sources). You do not need the templates, the checklist or the word "PRISMA" anywhere near a client deck. You need the log: a sheet with found, deduped, screened, excluded-with-reason, retrieved, assessed, included, and a total that adds up. Ten minutes of bookkeeping that turns "we looked at a lot of sources" into a number somebody else could reproduce.

Screening is genuinely a good use of a model - it is repetitive, rule-shaped and enormous. It stops being a good use the moment the tool is deciding what the rules are, or dropping things silently. Four conditions keep it defensible:

• The tool applies your rules, not its judgement. Paste the rule set in. "Is this relevant?" is not a rule.
• A reason per exclusion, in the tool's output. If it cannot say which rule a record failed, you cannot log it and you cannot defend it.
• Sample-check its decisions against your own. Screen 40 records yourself, blind, and compare. Disagreements are usually a rule you wrote ambiguously, which is useful either way.
• Nothing gets dropped silently. Excluded is a status with a reason attached, never an absence from a list.

The UNCLEAR bucket is the part that earns its keep. It is where the abstract is silent about population or measure - and those are exactly the records a confident filter would have quietly binned. You read those yourself.

Elicit aggregates Semantic Scholar, OpenAlex and PubMed, deduped to roughly 138M papers, and has a workflow built for this job (Systematic Literature Review, plus Extract Data for what we do in b5). The Pro tier screens up to 5,000 papers. The limitation to know before you trust a screening decision: full-text analysis only happens for open access material (or where you supply your own subscription plus their browser extension). Otherwise it works from title and abstract only.

Notice where that lands. Title-and-abstract screening is exactly the pass where "we could not tell from the abstract" is the correct answer a third of the time - and a tool without the full text cannot ever resolve it, so it has to guess or defer. Deferring is fine. Guessing looks identical in the output.

Second thing worth knowing, and it is a general lesson rather than a complaint about one vendor: Elicit's own published accuracy figures are inconsistent across its own pages - 96% and 99.4% for extraction, 99.5% and 97/99% for screening. Any of those numbers might be right. None of them arrives with a methodology, a corpus or a definition of what counted as correct, and they do not agree with each other. That is not a reason to avoid the tool. It is a reason your confidence in it should come from your own sample-check on your own pile, which takes twenty minutes and produces a number you can actually cite.

Run it twice: once when a source clears full-text screening, and once again the week you ship. Retractions land months and years after publication, so a library that was clean in March is not automatically clean in August, and the memo carries the date you checked.

• Collect the DOIs for everything in your included set. A source with no DOI is not exempt, it is harder - check the journal's own page for a notice.
• Query Crossref per DOI and read the update information: retractions, corrections, expressions of concern. All three change how much weight a source can carry, not just retraction.
• Record the check in your log with the date. "Retraction-checked 2026-08-08 via Crossref" is one line and it answers a question a reviewer would otherwise have to ask.
• Re-check before you ship, and re-check anything a claim depends on heavily. The load-bearing sources deserve the extra minute.

Then say the quiet part out loud: because this check costs minutes and needs no account, not having done it is a choice. It is the single easiest thing in this entire track to be caught not doing.