learn-ai-consulting-with-phoebe / Consultant session 3 of 10
Learn AI Consulting with Phoebe · Consultant track · Session 3 of 10

Research that holds up

Week 2 at Northbridge. The consulting cousin of the fake legal case is the invented statistic: plausible, well-formatted, and nonexistent. It arrives in perfect prose, with a decimal point for credibility, and week 2 - the research week - is when it tries to enter your engagement. This session is how it fails: a lead-not-fact rule, a source log with an owner column, and the confidentiality discipline that keeps client data out of your research prompts.

🟡 Consultant track Week 2 of the engagement The source-discipline session 45 min + self-study depth
0-3 · Welcome 3-20 · Concepts 20-40 · The source log 40-45 · Q&A
Part 0

Week 2: where the invented statistic hunts

Your hypothesis tree from b2 just became three workstreams, and every one of them is hungry for facts: market size, rival pricing, procurement norms, retention benchmarks. AI can feed that hunger faster than any analyst in history - and a fraction of what it serves will be invented. Not maliciously, and not obviously: models produce what a market-size stat looks like, and what it looks like is exactly what a real one looks like. The defense is not sharper eyes. It is a process that assumes every AI-supplied fact is unverified until proven otherwise, and a working artifact - the source log - that makes the proof visible, dated, and owned.

The plan: the lead-not-fact rule and the anatomy of the incident that made it industry law (Part 1), the source log itself with worked rows (Part 2), and the confidentiality half of research discipline (Part 3). Then twenty minutes building the Cadence source log - including one deliberate encounter with an invented statistic, so you know the feeling before it costs you.

★ What you walk out with today A source log for the Cadence engagement with the 8 claims it needs, two claims sourced and logged for real, one firsthand experience of chasing an AI-invented number to nowhere, and the one-line label you will put under vendor-claimed figures forever after.
Part 1 · covers the rule

The lead-not-fact rule 7 min live

One sentence, applied without exception: an AI-supplied fact is a LEAD until a human has seen its source. Leads are valuable - a research tool that surfaces ten leads an hour is worth having. But a lead travels a fixed route before it can touch a deliverable, and there is exactly one shortcut, and the shortcut is the one that makes the news.

AI output a plausible number LEAD not yet a fact Source-check gate a human sees the source itself source found no source FACT logged, with an owner BIN discarded, no regrets Client deliverable facts with owners only the shortcut: an unverified number straight into the deliverable - this is the Deloitte path Every lead takes the gate or the bin. The red route is faster, invisible in the draft, and priced in headlines.
🔍 Click to zoom - the fact's journey: gate to fact or bin, and the red shortcut that makes the news
LiveWhy models invent numbers, and what one invoice looks like4 min▶

No model looks up a market size the way you do. It generates the shape of a market-size statistic - a currency, a magnitude that fits the industry, a growth rate that rhymes with everything else it has read - and fills the shape plausibly. That is why invented numbers are not detectable by inspection: they are optimized, in a literal sense, to look right. The tell is never in the sentence. It is in the absence behind the sentence, which only a source-check reveals.

The invoice for skipping that check already exists. In October 2025, Deloitte Australia partially refunded the Australian government on a A$439,000 report after an academic found fabricated references and a misattributed court quote - errors Deloitte attributed to generative AI use it later disclosed. The refund was the small cost; the headline was the real price, because a consulting firm's product is the client's ability to trust what it hands over. Note what the incident did not require: bad strategy, wrong conclusions, or a hostile client. Unverified production details were enough.

Real worldRead the Deloitte anatomy as a process diagram and it is exactly the red arrow above: AI-assisted material entered a deliverable without every reference passing a human source-check. The fix now standard across the industry is not "no AI" - it is the gate: a human sees the source of every citation, quote, and number before it ships, and their initials say so. That initials column is Part 2's whole reason to exist.
Self-studyLeads are still worth collecting2 min▶

Do not let the rule curdle into refusing AI research help - that forfeits real gains for imaginary safety. Ten leads an hour, three of which survive the gate, still beats two hand-found facts in the same hour. The rule changes the accounting, not the tool: leads are free, facts are expensive, and only facts enter deliverables. Even retrieval-based tools that return citations get the same treatment - a citation is a lead to a source, not proof you have seen it. Click through. Every time.

Part 2 · covers the artifact

The source log 8 min live

The rule needs a place to live, and it is deliberately boring: a table. Every claim in the research pack gets a row - claim, source (link, title, page), date accessed, owner initials, confidence. If a number is in the deck but not in the log, that is a finding. If it is in the log without initials, that is a finding too.

the two columns that keep you out of the news Claim Source Date Owner Confidence Consultants with GPT-4 were 25.1% faster HBS working paper (SSRN 4573321) 2026-08-11 PF Verified ~30% time saved on info gathering (McKinsey, on Lilli) mckinsey.com - the vendor's own claim 2026-08-11 PF Vendor-claimed label it in the deck Every claim in the research pack gets a row. No row, no fact - and the owner initials are the Deloitte lesson. Confidence takes three values: verified · vendor-claimed · estimate. Two of the three need a label in the deliverable.
🔍 Click to zoom - the source log: one verified row, one vendor-claim row, five unforgiving columns
LiveThree source classes, and AI's proper research jobs4 min▶

Not all sources earn the same confidence. Primary - filings, transcripts, papers, the client's own data room - is what "verified" means; the log's first worked row ("consultants with GPT-4 were 25.1% faster - HBS working paper, verified") points at the study itself, not at coverage of it. Reputable secondary - major press - is acceptable with the outlet named. Vendor and self-reported numbers get logged, used, and labeled in the deliverable: "McKinsey claims ~30% time saved - vendor claim, label it" is the second worked row. The claim may be true; the point is the client can see whose claim it is. Estimates - yours or AI's - carry the same obligation.

Inside that discipline, AI has real research jobs. Coverage sweeps: "here are my 8 claims for the Cadence engagement - what angles am I missing?" catches the branch nobody staffed. Summarizing documents you supply: feeding it the actual filing and asking for the pricing section is inside the frontier, because the source is in your hand. Drafting search strategies: where to look, what terms, which source class each claim probably needs. And retrieval tools that return citations still get the log treatment - the citation goes in the source column only after a human has clicked it and seen the page.

Self-studyWhere deep-research technique lives1 min▶

This session is the consulting discipline: what counts as a fact, who owns it, and how it is labeled when it reaches a client. The machinery of deep research - grounding techniques, retrieval pipelines, multi-source synthesis, research-agent workflows - lives in learn-ai-research-with-phoebe. Take that course to get faster at producing leads; keep this session's log to decide what survives the gate. The two compose: better machinery fills the log faster, and the log keeps the machinery honest.

Part 3 · covers the other rail

Confidentiality in research 5 min live

Week 2 has its own signature temptation: pasting client data into a research tool to "analyze the market like Cadence." The research question is legitimate; the paste is the incident. Session 1's rail, restated operationally for research week:

The research-week version of Rail 1 Enterprise-tier tools only - the ones with contractual no-training terms. And anonymized inputs even there: "a B2B SaaS note-taker at ~$20 ARPU considering an enterprise tier" carries the full analytical weight of the question without the client's name attached. The model does not need to know it is Cadence to reason about seat caps and procurement cycles - and if the prompt works only with the name in it, the prompt was not about the market.
Real worldSamsung learned the paste lesson in about twenty days in 2023 - employees put confidential source code and a meeting transcript into ChatGPT, and by May generative AI was banned on company devices. The consulting version is worse because the pasted material belongs to your client. Session b8 makes this muscle memory with a drill: the lab's confidentiality gate reads your draft, and a leaky one - client names, contract figures, board material - is capped at a score of 15 no matter how brilliant the prose. The gate is fictional; the clause it simulates is in your firm's policy today.
Everyone logs

Build the Cadence source log 🗂 ★ 20 min · everyone works

No lab widget yet - the scorer debuts in session 5. Today's artifact is the log itself, plus one deliberate encounter with an invented statistic under controlled conditions.

List the 8 claims the engagement needs. Start from the tree: market size for enterprise note-taking, rival pricing, procurement cycle norms, retention benchmarks, seat-expansion patterns, security-review requirements, pricing-page positioning, willingness-to-pay signals. Adjust to your own tree's branches.

Classify each claim BEFORE searching. Which source class should each come from - primary, reputable secondary, or vendor? Deciding first stops you from accepting whatever class shows up first.

Run one AI coverage sweep. "Here are my 8 claims - what angles am I missing for an enterprise-tier go-to-market question?" Add the one suggestion that genuinely surprised you as claim 9.

Source two claims for real. Any public source. Fill complete rows: claim, source with title and link, date accessed, your initials, confidence. Notice how long a real row takes - that is the honest price of a fact.

Meet the invented statistic on purpose. Ask AI for a market-size figure WITHOUT any grounding or documents, then try to find its source. Follow the trail until it goes cold. That dead end, experienced once deliberately, is the session's most durable lesson.

Write the vendor-claim label. One line you would put under a vendor-claimed number in a client deck - short enough for a footnote, clear enough that the client knows whose claim it is. Keep it; you will reuse it for years.

✦ If you finish early Take your b2 proposal draft and give every number in it a log row. Any number that cannot fill its row gets flagged or cut - you just ran your first citation audit, three sessions before b9 makes it a standing QA gate.
Before next session

Homework 📝 ◐ 20 minutes

Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · The lead-not-fact rule says an AI-supplied fact becomes usable in a deliverable when...

Consistency across chats proves nothing - models converge on the same plausible shapes. And a citation is a lead to a source, not proof anyone opened it. The gate is a human seeing the source itself; only then does the claim get a log row and an owner.

2 · The source log's owner-initials column exists because...

Deloitte Australia refunded and made headlines over fabricated references it attributed to generative AI use - production details nobody had verified end to end. Initials in the row make Rail 2 concrete: a specific human vouches for each fact, and the log shows exactly which ones nobody has vouched for yet.

3 · The three source classes are primary, reputable secondary, and vendor/self-reported. The one that must be labeled in the deliverable itself is...

"McKinsey claims ~30% time saved" is usable - labeled. Vendor numbers describe the vendor's best story about itself; the label moves the trust decision to the client, where it belongs. Estimates carry the same obligation.

Coverage, honestly

What this session covered from the official universe

✓Deloitte Australia incident (Oct 2025) - the liability anchor, retold in fullfabricated references + a misattributed court quote, attributed to generative AI use later disclosed; partial refund; the headline as the real price
✓Source-log discipline: claim, source, date, owner, confidence + three source classesfield practice, labeled honestly - the working artifact this course adds on top of the incident record
◐Deep-research technique (learn-ai-research-with-phoebe)pointer only: this session is the consulting discipline, that course is the machinery
◐Vendor data-handling terms (OpenAI enterprise privacy; Anthropic commercial terms)restated operationally for research week; b8 owns the full drill with the lab's confidentiality gate

Certificates, videos, and graded assessments stay with the official providers - this course teaches the working core.

Consultant session 3 cheat sheet

The ruleAn AI-supplied fact is a LEAD until a human has seen its source. Gate or bin - never the red shortcut.
Why models inventThey generate what a stat looks like, not what it is. Invented numbers are undetectable by inspection - only the source-check reveals the absence.
The invoiceDeloitte Australia, Oct 2025: fabricated references + a misattributed quote, attributed to genAI use later disclosed. Partial refund; headline was the price.
The log's columnsClaim · source (link/title/page) · date accessed · owner initials · confidence. No row, no fact.
Confidence valuesVerified · vendor-claimed · estimate. The last two get a label in the deliverable itself.
Three source classesPrimary (filings, transcripts, papers) · reputable secondary (major press, outlet named) · vendor/self-reported (labeled).
AI's research jobsCoverage sweeps ("what am I missing?"), summarizing documents YOU supply, drafting search strategies. Citations still get the log treatment.
Research-week Rail 1Enterprise-tier tools only, anonymized even there: "a B2B SaaS note-taker at ~$20 ARPU" - the name adds nothing to the analysis.