Why we buy the bottom of the stack first
The running decision is still the same one: should Cadence, an AI meeting-notes product, build a lecture-notes product for higher education next year? By now your team has an evidence standard (a2), a catalogue of where AI fails (a3) and governance rails (a4). Tonight they need instruments. A research stack has four layers, and they are not equally interesting to buy: the top two have the nicest demos and the bottom one decides whether any of it is defensible. So we start at the bottom, then work up, and at every stop we ask the only two questions that matter in procurement - what does it actually cover, and what will it not tell me that I would need to know.
Ground truth, and why it has to come first 10 min live
Every clever thing higher in the stack is reading from a corpus, and it can only be as complete, as current and as honest as that corpus is. Three sources make up the layer almost everyone actually uses, all three are free to start, and all three have a documented limit your team should be able to recite. OpenAlex is the widest open index: 323,171,764 works and 123,255,918 authors, licensed CC0, which is why so many tools sit on top of it. It also documents a metered credit model - $1 a day free with an API key, $0.01 a day without one, search at $0.001 per query, PDF download at $0.01 per item - and an unauthenticated search still returned successfully when we tested it, so the metering is not hard-enforced today. Treat it as the forward contract anyway: it tells you where the cost of scale is going to land.
Crossref holds 185,277,841 records, needs no sign-up, and has the one feature on this page with a direct integrity consequence: it carries retraction and post-publication update information. That makes Crossref the place your team checks whether a citation has been withdrawn, which is not an optional nicety when a partner is about to quote your memo. It also has a courtesy rule worth knowing before someone builds a script: adding a mailto to your requests moves you into the polite pool and triples your rate limit, 3 per second instead of 1.
Semantic Scholar covers 214M papers, 2.49B citations and 79M authors, and it is where the counter-intuitive rate limit lives. Unauthenticated access is documented as "rate-limited to 1000 requests per second shared among all unauthenticated users", while an API key's introductory limit is 1 request per second. Read that twice: the key buys you stability, not throughput. And one field needs a rule wrapped around it before your team touches it - tldr is machine-generated. It is useful for triage and it must never be quoted as the authors' words.
LiveThe three sentences your team should be able to say from memory4 min▶
Not the totals. The consequences:
- OpenAlex is the widest and the most reusable - CC0 licensing is why it sits under so many products. Its metered credit model is documented, so plan for scale to cost something even though an unauthenticated search was not blocked at test time.
- Crossref is where you check for withdrawal - it carries retraction and post-publication updates. If a claim in a client memo is load-bearing, its source gets a Crossref check before the memo ships. Add a
mailtoand you are in the polite pool at 3 requests per second rather than 1. - Semantic Scholar's key buys stability, not speed - unauthenticated is documented as 1000 requests per second shared across everyone unauthenticated, an introductory key is 1 per second. And
tldris machine-generated: fine for triage, never a quote.
The reason to hold these in your head as a leader is procurement. When a vendor tells you their corpus is enormous, the follow-up is not "how enormous" - it is "which of these three is under it, when was it last refreshed, and do you pass through retraction status".
Self-studyMachine-generated summaries and the quoting rule3 min read▶
Semantic Scholar's tldr is the cleanest example of a field that is genuinely useful and genuinely dangerous in the same breath. It is a one-line machine-generated summary of a paper, and it is very good at helping you decide whether to open the PDF. What it is not is a sentence the authors wrote, or a sentence any human checked against the paper's actual limits.
The rule to write into your team's standard is short: generated text can inform a screening decision and can never appear inside quotation marks attributed to a source. If a sentence is going to carry weight in a memo, somebody opened the source and read the passage it came from. Semantic Scholar's openAccessPdf field is the practical bridge for that - it is how you get from triage to actually reading.
Search and screening, limits stated as loudly as features 12 min live
This is the layer your team will ask for by name, and it is the layer where a good demo hides the thing you needed to know. Elicit is the most complete of them: five workflows (Research Agent, Research Reports, Systematic Literature Review, Find Papers and Chat with Papers, Extract Data), a deduped corpus of roughly 138M papers built from Semantic Scholar, OpenAlex and PubMed, Pro at $49 a month screening up to 5,000 papers, and an API at 100 requests per minute per IP. Now the limit that decides defensibility, and it belongs in the same breath as the corpus number: full-text analysis happens only for open-access material, or for material you hold a subscription to plus their browser extension. Otherwise it is working from the title and abstract. Everything downstream of that is a claim about an abstract wearing the clothes of a claim about a study.
Elicit is also, usefully, the live example of the accuracy-number problem. Across its own pages it states extraction accuracy as both 96% and 99.4%, and screening accuracy as 99.5% and also 97/99%. None of those is obviously wrong. The point is that a number without a methodology attached is not citable, and a vendor that publishes two different numbers for the same capability has told you that neither one came with one.
| Tool | What it does | Corpus | The limit that matters |
|---|---|---|---|
| Elicit | Five workflows: research agent, reports, systematic literature review, find and chat with papers, extract data. Pro $49/mo screens up to 5,000 papers; API 100 requests per minute per IP | ~138M papers, deduped from Semantic Scholar, OpenAlex and PubMed | Full text only for open access, or your own subscription plus their browser extension. Otherwise title and abstract only. Its own accuracy claims are inconsistent across its own pages |
| Consensus | The Consensus Meter buckets conclusions into agree, disagree and inconclusive | Not verified here; the site blocked a direct fetch, so treat details as search-index sourced | The Meter analyses the top 20 papers for a query. That is a sample of the top of one ranking, not a synthesis of a literature |
| Scite | Classifies citations as supporting, contrasting or mentioning. Method is peer-reviewed: Nicholson et al., Quantitative Science Studies 2(3):882, 2021 | Citation statements across the published record | Reported distribution is about 92.6% mentioning, 6.5% supporting, 0.8% contrasting. So the absence of contrasting citations is nearly uninformative. Classifier precision and recall are unverified |
| Undermind | A convergence estimator: it fits the decay of the discovery curve to judge whether a search has become comprehensive. Genuinely distinctive | arXiv only, 2.3M papers, GPT-4 era whitepaper (Jan 2024) | Single-repository coverage, and a circular evaluation - it grades Google Scholar results "using the same relevance classification subroutine that Undermind employs" |
| SciSpace | Not assessed | Not verified | The site blocked verification. This page says nothing further about it |
LiveReading Undermind's whitepaper as a leadership exercise4 min▶
Undermind is worth ten minutes of your attention even if you never buy it, because it is the rare vendor doing something methodologically interesting and the rare vendor whose benchmark you can dismantle without any specialist knowledge.
The interesting part: the convergence estimator. Rather than telling you it found good papers, it fits the decay of its own discovery curve - new relevant results per unit of search effort - and uses that decay to estimate how comprehensive the search has become. That is a real answer to the question every literature search ends on, which is "have I looked enough". Nobody else on this page attempts it.
The part to hand to a sceptical colleague: the comparison to Google Scholar is graded by Undermind's own relevance classifier, on both sides. A benchmark where the product under test also acts as the judge cannot separate "we find more relevant results" from "we find more results that we consider relevant". And the coverage is arXiv only, 2.3M papers - so for a higher-education market question with regulatory and pedagogical literature in it, the corpus is the wrong shape regardless of how good the estimator is.
Self-studyWhat the top-20 window and the contrasting-citation rate do to your memos3 min read▶
Two limits on this layer produce the same failure in practice, which is a claim written flat when it should have been qualified.
Consensus's Meter reads the top 20 papers for your query. If your query happens to sit on top of a settled ranking, that reads like consensus. If the disagreeing literature is at rank 25 because it is older, smaller or from a different field's vocabulary, the Meter cannot see it and will not tell you it cannot. The Meter is a fast orientation instrument, and it is not the contradiction check from a2 and b7.
Scite's distribution is the sharper trap. About 92.6% of classified citation statements are merely mentioning, 6.5% supporting, 0.8% contrasting. So a paper with zero contrasting citations is the overwhelmingly normal case, and reading that zero as "uncontested" is reading noise as signal. Where Scite genuinely helps is the other direction: if contrasting citations do show up, that is a strong and specific pointer to the argument you need to go read.
The abstract-only limit is how a defensible-looking memo goes soft. Picture the Cadence question: someone screens 900 papers on lecture capture and note-taking outcomes in Elicit, and the extraction table fills in beautifully. Most of that literature is paywalled, so most of those rows were built from titles and abstracts. Abstracts state findings; they very rarely state the sample's country, the comparison condition, the funder, or the caveat the authors put in their limitations section. Every one of those omissions is a hole a reviewer can put a finger through - and none of them show up as an error in the tool. This is why b5 insists that any number carrying weight gets confirmed against the full text, and why "we screened 900 papers" is a statement about effort rather than about evidence.
Reference managers, qualitative tools, notebooks 10 min live
The remaining layers rarely get their own line in a budget, which is precisely why their risks go unexamined. Reference managers are where your PDFs live, so they are a data-egress question, not an accuracy question. Qualitative tools apply codes to what humans said, and not one of them publishes a coding accuracy figure. Notebooks feel airtight because they are pointed at your own sources, and the guarantee people believe in is stronger than the one the provider actually gives.
LiveReference managers: the risk is egress, not accuracy4 min▶
All three of the common managers have an AI story, and all three of them move the question somewhere other than accuracy:
- Zotero has no first-party AI feature at all. Every AI capability is a community plugin, and Zotero's own warning is the sentence to read aloud in a governance meeting: "plugins have full access to your Zotero and your computer." So the decision is not about a model's quality, it is about who you are letting inside.
- Paperpile's official "Ask AI" is a router. It sends your full-text PDF out to ChatGPT, Claude, Gemini, Copilot or NotebookLM. Two consequences: accuracy is now a property of whichever model you picked, not of Paperpile, and your PDFs leave the tool by design rather than by accident.
- Mendeley references an "Ask AI" capability with no documented detail we could verify. That is a gap, and for a procurement decision an undocumented feature is a feature you cannot sign off.
None of the three publishes any accuracy claim. Which means the honest framing for your team is: this layer is a confidentiality and plugin-trust decision. Route it through the a4 governance rails - where does the document go, who is a sub-processor, and does any of it touch participant material.
LiveQualitative tools: everyone auto-codes, nobody publishes coding accuracy4 min▶
Three tools your insights team will name, with what each one actually documents:
- Dovetail - Magic Highlight auto-applies your existing tag structure, plus Magic Cluster, Magic Summary and Ask Dovetail. The chat is "Powered by the latest release of Claude". Transcription runs through Amazon Transcribe and AssemblyAI. It states customer data is not used for training. Its AI documentation carries no accuracy figure and no instruction to verify output.
- Looppanel - claims "95%+ accurate transcripts", scoped to English. Other pages on the same site say 90% and 93%+, so the claim is inconsistent with itself. 96 further languages are supported, but translation and non-English AI notes are Enterprise-only, which is a budget question disguised as a feature list.
- Marvin - cited answers and PII blur, which are genuinely good governance features. No accuracy claim, no language list.
The cross-cutting finding is the one to take to your team, and it is stronger than any individual gap: all three auto-apply codes to human speech, and not one of them publishes a coding or theme-assignment accuracy figure. Transcription accuracy is not coding accuracy. A 95% accurate transcript tells you the words are right; it tells you nothing about whether the theme the tool attached to those words is the theme a trained researcher would have attached. b9 builds the check for this with inter-rater agreement, which is the only honest answer available.
Self-studyNotebookLM: real limits, and a guarantee that is not there3 min read▶
NotebookLM, now being renamed Gemini Notebook, is the most trusted tool in this whole session, largely on a belief that is not in the documentation.
The verified limits are 500,000 words per source or 200MB, and on the base tier 100 notebooks holding 50 sources each (Plus 100, Pro 300, Ultra 500 to 600 sources). Google's own wording is the weaker "answers questions based on the information provided in your uploaded sources", and on very short sources it "references the entire document without a cited individual text".
The gap worth teaching is what is absent: Google's FAQ contains no guarantee that the model never uses knowledge beyond your sources, and no statement about hallucination risk. The popular framing - it cannot go beyond what you gave it - is stronger than anything Google actually says. That does not make the tool bad. It makes "grounded in my sources" a design intent rather than a contractual property, so a sentence pulled out of a notebook still needs the citation opened before it carries weight in a memo.
Five things to insist on in writing 7 min live
You do not need to become an expert on any of these products. You need five answers, in writing, from anyone selling you one. A documented corpus with an update cadence - which databases, and when was it last refreshed. An exportable audit trail - can I get the search record, the screening decisions and the extraction rows out, per claim, in a form a client's data team can read. A stated position on training and data residency - do my uploads train your models, where do they sit, which sub-processors see them. Full-text versus abstract-only clarity - for material we do not hold a subscription to, what did you actually read. An accuracy number with a methodology - the sample, the task, and who labelled the ground truth. If a vendor can answer four of the five, that is often a reasonable buy with a documented limit. If they cannot answer the fifth one, you have not been given a number; you have been given a decoration.
The difference between getting marketing copy back and getting a testable answer is almost entirely in how you ask. Send the first version of this and you will receive a case study. Send the second and you will receive either a document you can hold someone to, or a silence that is itself the answer.
"Which AI literature review tool is the most accurate? We want the best one for our research team."
"We screen roughly 800 papers a quarter, mostly paywalled education and clinical literature, and our memos have to survive a client's data team. Please answer in writing: (1) which databases your corpus is built from, and when it was last refreshed; (2) when we do not hold a subscription, does extraction read the full text or the title and abstract, and how is that flagged per source; (3) for every accuracy figure on your site, the methodology behind it - the task, the sample size, and who labelled the ground truth; (4) exactly what we can export as an audit trail, per claim, and in what format; (5) whether our uploads are used to train your models, where they are stored, and which sub-processors can access them. If any answer is 'not documented', say so."
Self-studyWhere build actually makes sense3 min read▶
Build is usually the wrong instinct in this space, with one exception that is genuinely worth a week of an engineer's time: the ground-truth layer. OpenAlex is CC0, Crossref needs no sign-up, and Semantic Scholar's key is free. A small internal utility that takes a list of DOIs and returns coverage, publication year and retraction status is a few hundred lines, and it does the one job no vendor on this page will do for you as a first-class feature - it tells you whether the sources under a live memo have been withdrawn since you cited them.
Everything above layer 1 is a build you will regret. Screening interfaces, extraction UIs and qualitative coding workspaces are large products with real design cost, and they change monthly. Buy those, with the limit written down, and keep your engineering effort on the check that protects your credibility.
Try it yourself - this week ◐ 30-40 min total
- Take whatever research tool your team already pays for and answer the five gates from Part 4 using only the vendor's public documentation. Note which gate you cannot answer. That gap is your next procurement conversation.
- Pick three load-bearing citations from a memo your team shipped in the last quarter and check each one in Crossref for retraction or post-publication updates. This takes about ten minutes and it is the cheapest credibility insurance on this page.
- Find one accuracy percentage on any vendor's site and try to locate its methodology - the task, the sample, and who labelled the ground truth. Write down what you found. Most of the time the answer is nothing, and that is the lesson.
- Ask whoever runs your qualitative work how the auto-applied codes get checked today. If the answer is that they get reviewed informally, you have found the b9 gap, and it is a bigger risk than any transcript error rate.
Official sources covered
Every figure on this page was taken from the provider's own documentation on 2026-08-08, and where a provider blocked verification the gap is reproduced as a gap rather than filled. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Your analyst screened 900 paywalled papers in Elicit and the extraction table looks complete. What is the first thing to check?
Elicit reads full text only for open-access material, or for material you hold a subscription to plus their browser extension. Otherwise it works from title and abstract. Abstracts state findings but rarely the population, comparison, funder or limitations, so every one of those omissions travels silently into your table.
2 · Where do you check whether a citation has been withdrawn, and what does adding a mailto to your requests do?
Crossref holds 185,277,841 records, needs no sign-up, and carries retraction and post-publication updates - which is what makes it the withdrawal check before a memo ships. The polite pool is a courtesy convention with a real payoff: 3 requests per second rather than 1.
3 · Scite shows zero contrasting citations for a study you want to build a recommendation on. What does that tell you?
The reported corpus distribution is roughly 92.6% mentioning, 6.5% supporting and 0.8% contrasting. Zero contrasting citations is the overwhelmingly normal case, so reading it as agreement is reading noise as signal. Scite earns its keep in the other direction: when contrasting citations do appear, they point you straight at the argument to go read.
Leader session 5 cheat sheet · pin this
mailto for the polite pool: 3 requests per second instead of 1.tldr is machine-generated - never quote it as the authors.