learn-ai-consulting-with-phoebe / Consultant session 9 of 10
Learn AI Consulting with Phoebe · Consultant track · Session 9 of 10

QA and the readout: the audit Deloitte wishes it ran

Week 6. The deck is beautiful, the storyline scores 100, and none of that survives one fabricated citation in front of a client. Deloitte Australia's analysis may well have been fine - the firm still refunded and made the news over unverified production details. This session is the gate between "done" and "shipped": the citation audit that traces every fact to its source, the hostile-client rehearsal that finds your weak answers before the CFO does, and the readout where the room belongs to humans.

🟠 Consultant track Week 6 of the engagement The last gate 45 min + self-study depth
0-3 · Welcome 3-20 · Concepts 20-40 · Run the audit 40-45 · Q&A
Part 0

The gate between done and shipped

Everything the engagement produced so far - b3's source log, b5's storyline, b6's deck, b7's pre-read - converges on one meeting in week 6. Two things stand between the work and that room. First, a standing QA pass that would have saved Deloitte Australia a refund and a headline. Second, a rehearsal in which AI plays the most skeptical person your client employs. Neither takes long. Both are the difference between a deliverable and a liability.

Live - presented in session Self-study - read after class Sources covered
★ What you walk out with today A three-outcome citation audit you can run on any deliverable, the hostile-client rehearsal prompt and its discipline (every question gets an answer with a source row), and the readout architecture: answer in the first two minutes, exhibits on demand, Q&A as the main event you rehearsed for.
Why this session exists Rail 2 from session 1 - every fact reaching a client has a human owner - was a rule. Today it becomes a procedure. Rules get forgotten under deadline; procedures with checklists and fresh eyes do not.
Part 1 · covers the QA gate

The citation audit 8 min live

The standing QA pass: before anything ships, every number, quote, and reference in the final deliverable gets traced back to its row in the source log you started in b3. Week 2's bookkeeping cashes out in week 6 - a fact with a log row takes seconds to verify; a fact without one is a red flag by definition.

Fact inventory 62% hit the seat cap 210 in procurement 94% vs 71% retention $38-45 rivals · $30 entry ARPU 2.1x every quote + reference every chart's source line The audit gate fresh eyes - NOT the author. Checklist-driven, timeboxed, every fact. VERIFIED - a human saw the source stays in the deliverable as written; the log row records who checked it VENDOR-CLAIMED - kept, but labeled the deliverable says who claims it, in the text - not in a hidden footnote CUT - no source found no source, no sentence. The claim leaves before the client arrives. Tally strip: every fact lands in exactly one lane. A fact with no lane is a fact with no owner - and that is Rail 2 broken.
🔍 Click to zoom - the citation audit: three outcomes, no fourth option
LiveThree outcomes, and who runs the pass4 min▶

Every fact in the final deliverable gets exactly one of three stamps. VERIFIED: a human saw the source with their own eyes - the fact stays, and the log row records who checked it. VENDOR-CLAIMED: the fact comes from a party with an interest in it (a vendor's own benchmark, a press-reported company statement) - it stays, but the deliverable labels it as a claim in the text itself. CUT: no source found. No source, no sentence - however plausible, however well it fits the story. The invented statistic from b3 is plausible and well-formatted by construction; that is exactly why "it sounds right" is not an outcome.

And who runs it: not the author. Fresh eyes catch what familiarity forgives - the author reads what they meant, the auditor reads what is on the page. Timebox it, run it from a checklist, and run it every time, not just when the deliverable feels risky. The Deloitte lesson, retold in two sentences: an academic found fabricated references and a misattributed court quote in a government report, errors Deloitte attributed to generative AI use it later disclosed. The partial refund was modest; the headline was the real cost.

Real worldNote what the Deloitte Australia audit would have needed to catch the problem: not better strategy review, not a smarter partner read - just one person tracing each reference to a real document before shipping. The errors were production details, the kind of work most teams now delegate to AI most casually. That is why the audit is a standing gate and not a judgment call: the cheap-looking work carries the expensive risk, and it looks exactly as polished as the verified kind.
Self-studyThe audit checklist, written out2 min▶

Steal this and adapt it: (1) list every number, named quote, and external reference in the final document - this is the fact inventory; (2) for each, find its source-log row from b3; (3) open the source and confirm the fact says what the deliverable says it says; (4) stamp VERIFIED, VENDOR-CLAIMED, or CUT; (5) for every VENDOR-CLAIMED, confirm the label appears in the deliverable text; (6) for every CUT, either find a real source or delete the sentence; (7) sign the audit sheet with your name and the date. Step 7 is the point - Rail 2 says every fact has a human owner, and a signature makes ownership visible.

Part 2 · covers the rehearsal

AI as the hostile client 7 min live

The audit protects the facts. The rehearsal protects you. Feed the anonymized executive summary to AI and ask for the ten hardest questions a skeptical CFO would ask - then answer them before the room does.

LiveTen hostile questions, and the discipline that makes them useful4 min▶

Why this works: the model is excellent at generating challenge patterns - it has read every objection ever published. Ask for "the ten hardest questions a skeptical CFO would ask about this recommendation" and it will reliably surface the cannibalization worry, the pricing-anchor challenge, the capacity question, the why-now push. This is squarely inside the frontier from session 1: pattern generation, not fact retrieval.

The discipline is what turns the parlor trick into QA: every question gets a one-line answer WITH its source row. "Enterprise-style seats retain at 94% vs 71% for individuals - source log row 12" is an answer. "Retention is strong" is not; the scorer from b5 already taught you why. And the two questions you cannot answer this way are the most valuable output of the exercise: each becomes either week-6 work (go find the answer) or an honest caveat in the appendix (say what you do not know). Both beat improvising in the room.

One more rehearsal, from b7's difficult news: role-play the $15-tier partisan - the stakeholder who wanted the cheaper tier and did not get it. Have AI play them, argue their case at full strength, and rehearse the disagreement before the room does it live. You know this person will be in the readout. Meet them in private first.

Real worldRemember the jagged-frontier study's warning: outside its capability, AI-assisted consultants were 19 percentage points less likely to be right, and the wrong answers read just as confident. The hostile-client rehearsal inverts that risk into protection - you are not asking AI for facts, you are asking it for attacks, and a fluent, confident attack is exactly what you want to practice against. The frontier map tells you this is one of the safest delegations in the whole engagement.
Self-studyThe pre-mortem variant2 min▶

A second rehearsal prompt, for the night before: "Assume this recommendation failed in 12 months - write the post-mortem." The model will produce a plausible failure story: enterprise sales cycle longer than modeled, the 210-account pipeline converting worse than hoped, a rival dropping price below $30. Then check the deck: does it answer those failure modes, with a caveat, a mitigation, or a monitoring plan? Where it does not, you have found week-6 work while it still costs nothing. The pre-mortem finds structural risks the question list misses, because it forces a narrative of failure instead of a list of doubts.

Part 3 · covers the meeting

The readout itself 5 min live

The meeting has an architecture, and it is not "walk the deck". The pre-read from b7 already delivered the answer; the meeting exists for the argument, and the Q&A is the main event - the one you spent Part 2 rehearsing for.

The meeting, minute by minute Min 0-2 the answer The argument exhibits on demand Q&A - the main event the ten questions, already rehearsed The ask pilot · owner · date Before the room: AI did everything rehearsable. hostile questions, the pre-mortem, the pre-read draft, timing the argument During the room: AI does nothing. the room belongs to humans - judgment, eye contact, the signature The pre-read delivered the answer already - the meeting is for the argument.
🔍 Click to zoom - the readout: what AI did before, and why it does nothing during
LiveAnswer in two minutes, close with the ask3 min▶

Open with the answer - in the first two minutes, restated plainly, because the pre-read already delivered it and pretending otherwise wastes the room's patience. Exhibits on demand, not on parade: the appendix holds every chart you might need; the meeting shows only the ones a question summons. The Q&A is the main event - not the risk at the end of the presentation but the reason the meeting exists, and the part you rehearsed ten questions deep.

What AI does during the readout: nothing. What it did before: everything rehearsable - the question list, the pre-mortem, the pre-read draft, the timing run. That division is session 1's frontier map applied to a meeting: preparation is inside the frontier, the room is not.

And close with the ask, stated as a decision: approve the $30 per seat tier pilot for the 210 accounts already in procurement or security review, name the owner, set the review date. A readout that ends without an ask was a seminar. Northbridge does not bill for seminars.

Everyone audits

Run the audit on the capstone summary 🔍 ★ 20 min · everyone works

No lab widget today - the audit IS the exercise, run by hand the way you will run it on real work. Your subject: the b5 final summary, the one that scored 100. A perfect scorer result and a perfect audit are different tests, and it needs to pass both.

Take the b5 final summary. The 100-scoring draft from the storyline lab - its facts are the Cadence case facts you have typed all course. If you rewrote your own summary for b5's homework, keep it next to you; it goes through the same audit in step 6's spirit next week.

Build the fact inventory. List every number the summary asserts: 62% hitting the 5-seat cap in 3 weeks, 210 accounts in procurement or security review, 94% vs 71% retention, rivals at $38-45 per seat, the $30 recommendation, the 2.1x ARPU lift. Every one goes on the audit sheet - no fact is too obvious to list.

Assign each fact an outcome. All six case facts trace to the Cadence data room, so all six stamp VERIFIED - write the log row next to each. Then practice the middle lane: write one VENDOR-CLAIMED label sentence, for example "Competitor pricing of $38-45 per seat is as published on rival rate cards, not independently negotiated figures." Feel how the label lives in the text, not a footnote.

Generate ten hostile questions. With AI if you have an approved tool at hand - feed it the summary and ask for the ten hardest questions a skeptical CFO would ask. By hand otherwise: cannibalization, pricing anchors, capacity, why-now, what-if-a-rival-cuts-price will get you to ten.

Answer five with source rows. One line each, number plus row - the "retention is 94% vs 71%, row 12" pattern. Then mark the two hardest remaining questions as either week-6 work or an honest appendix caveat. Write the caveat sentence; do not just gesture at it.

Write the readout's first two sentences. Sentence one: the answer - launch the enterprise tier at $30 per seat in Q1. Sentence two: the ask - approve the pilot for the 210 accounts, name the owner, set the review date. If those two sentences work, the rest of the meeting is defense in depth.

✦ If you finish early Run the pre-mortem on the recommendation: "assume the $30 tier failed in 12 months - write the post-mortem." Then check which failure modes the summary already answers and which would need a caveat. That gap list is exactly what a partner asks for in the week-6 review.
Before next session

Homework 📝 ◐ 25 minutes

Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · The citation audit gives every fact one of three outcomes, and the pass is run by...

A human saw the source, or the claim is labeled as someone's claim, or the sentence leaves - there is no fourth lane. And the author reads what they meant; fresh eyes read what is on the page. Familiarity forgives exactly the errors that end up in headlines.

2 · AI makes a good hostile client because...

Pattern generation is deep inside the frontier; that is the safe delegation. The discipline is the other half: an answer without a source row is just more prose, and the two questions you cannot answer become week-6 work or honest caveats - never improvisation.

3 · The readout architecture says...

The pre-read already delivered the answer - the meeting is for the argument. Exhibits parade only when summoned, the Q&A is the event you spent Part 2 rehearsing for, and the close is an ask: approve the pilot, name the owner, set the date.

Coverage, honestly

What this session covered from the official universe

✓Deloitte Australia incident (Oct 2025)the audit's reason: fabricated references + a misattributed court quote, attributed to generative AI use later disclosed; partial refund, and the headline as the real cost
✓Citation audit + hostile-client rehearsalfield practice, labeled honestly - assembled from Rail 2 and the frontier evidence, not lifted from a published curriculum
✓Source-log discipline (continuity from b3)week 2's log rows are what make week 6's audit take minutes instead of days - the discipline cashes out here

Certificates, videos, and graded assessments stay with the official providers - this course teaches the working core with a live scorer.

Consultant session 9 cheat sheet

Three outcomesVERIFIED (human saw the source) · VENDOR-CLAIMED (kept, labeled in the text) · CUT (no source, no sentence).
Not the authorFresh eyes catch what familiarity forgives. Timeboxed, checklist-driven, run every time - a procedure, not a vibe.
The Deloitte lessonFabricated references + a misattributed quote, attributed to AI use later disclosed. Refund modest; headline expensive.
Hostile client promptFeed the anonymized summary to AI: "the ten hardest questions a skeptical CFO would ask." It has read every objection ever published.
The disciplineEvery question gets a one-line answer WITH its source row. The two you cannot answer become week-6 work or honest caveats.
Pre-mortem variant"Assume this failed in 12 months - write the post-mortem." Then check the deck answers those failure modes.
Readout architectureAnswer in the first two minutes · exhibits on demand, not on parade · Q&A as the rehearsed main event · AI does nothing in the room.
The askApprove the $30 tier pilot for the 210 accounts, name the owner, set the review date. No ask, no readout - just a seminar.