learn-rag-with-phoebe / Leader session 1 of 6
Learn RAG with Phoebe · Leader track · Session 1 of 6

What RAG is, and why it beats a bigger prompt

Every vendor now says their product "uses your data with AI". This session gives you the one mental model that lets you tell the real thing from the demo: retrieval-augmented generation - fetch the few relevant passages, hand them to the model, and make it answer from them with a citation. By the end you can say what RAG is in one breath, tell when it beats fine-tuning or a giant prompt, and spot the moment a team is reaching for the wrong tool.

🟢 Leader track Leaders: C-level · managers · curious minds No code, ever Start here
0-3 · Welcome 3-20 · Concepts 20-40 · Exercises 40-45 · Q&A
Part 0

Why this hour is worth your calendar

An LLM on its own is a brilliant colleague with no access to your files and a habit of answering confidently even when it is guessing. RAG is the fix the whole industry converged on: instead of retraining the model or pasting your entire drive into the prompt, you retrieve the handful of passages that actually bear on the question and let the model answer from those. It is cheaper than fine-tuning, more current than the model's training data, and - done right - it can cite its sources and admit when it does not know. That last part is why leaders care: RAG is the difference between an assistant that sounds right and one you can audit.

Live - presented in session Self-study - read after class ★ Try it now prompt Sources covered at the end
★ What you walk out with today A one-breath definition of RAG you can use in a board meeting, a three-doors decision (RAG vs fine-tuning vs a bigger prompt) that sorts any "use our data with AI" idea, the two-question screen for whether RAG actually pays off, and five questions to put to your data team this week.
Part 1 · covers the retrieve-augment-generate idea

Retrieve, augment, generate 9 min live

RAG is three moves in a row, and the names say exactly what happens. The model never sees your whole knowledge base - it sees the few passages a search step pulled for this one question. The picture below is the entire concept.

Question from a user 1 · Retrieve search your docs Your documents the knowledge base 2 · Augment passages + question 3 · Generate the model answers Cited answer The model only ever sees the retrieved passages - not the whole knowledge base.
🔍 Click to zoom - retrieve, augment, generate: the whole idea on one line
LiveThe one-breath definition3 min

Here is RAG in a sentence you can say out loud in a meeting: "When someone asks a question, we search our own documents for the few passages that matter, paste just those into the prompt, and have the model answer from them - with a citation."

  • Retrieve: a search step finds the most relevant passages. Not keyword search - meaning search, which is why "how do I get my money back" can find a paragraph titled "Refunds" that shares no words with the question.
  • Augment: those passages are added to the prompt alongside the question. The model's context becomes "here is what our documents say; now answer."
  • Generate: the model writes an answer grounded in that supplied text, ideally quoting or citing the source it used.

The OpenAI cookbook frames it well: this is giving the model short-term memory through the prompt, which is more reliable for factual recall than baking facts into the model itself.

★ Try it now (any chat AI)Explain retrieval-augmented generation to a board member in four sentences, using the analogy of a brilliant new hire who is not allowed to memorize the company handbook but can look things up in it before answering. Then name one risk this design removes and one it does not.
LiveWhy not just let the model memorize everything?3 min

Two reasons, and both matter to a budget holder.

  • Your knowledge changes; the model's training does not. A price list, a policy, last week's incident - none of that is in the model. Retrieval reads whatever is in your documents right now, so the answer is as current as your files.
  • Memorized facts cannot be cited or removed. When knowledge lives in the model's weights you cannot point to where an answer came from, and you cannot delete one document's influence. When it lives in retrievable documents, every answer can carry a source, and removing a document removes its answers. That is the auditability leaders keep asking for.
Real world

The policy that changed on a Friday. A support team (anonymized) updated its refund window from 14 to 30 days. With RAG, the change was one edited document and the bot answered "30 days" that afternoon. The team that had fine-tuned a model on the old policy needed a retraining cycle - and in the meantime the model kept confidently saying "14".

Self-studyMeaning search, in one paragraph2 min read

The "retrieve" step does not match words - it matches meaning. Every passage is turned into a list of numbers (an embedding) that captures what it is about, and the question is turned into the same kind of list. The search finds the passages whose numbers point in the same direction as the question's. You do not need the math today - the builder track's first session lets you feel it in a live playground - but the leadership takeaway is simple: RAG can find the right paragraph even when it shares no keywords with the question. That is what makes it useful on real, messy documents, and it is also why it sometimes retrieves something plausible but wrong - which is exactly what session a4 teaches you to measure.

Part 2 · covers RAG vs fine-tuning vs long-context

Three doors for "use our data with AI" 8 min live

When a team wants an AI to use company knowledge, there are three doors - and they are constantly confused for each other. RAG adds knowledge. Fine-tuning changes behavior. A bigger prompt just pastes everything in. Most "we need to fine-tune on our data" requests are really RAG requests.

Door 1 · RAG Adds fresh, citable knowledge from your docs ✓ current · ✓ cites sources ✓ cheapest to change Use for: facts, policies, docs Door 2 · Fine-tuning Changes style, tone, format, behavior ✓ consistent voice + format ✗ not for facts · ✗ no citations Use for: "always answer like this" Door 3 · Bigger prompt Paste everything into a long context window ✓ no setup · simple ✗ costly per call · ✗ no citations Use for: one small, whole doc Rule of thumb: new KNOWLEDGE -> RAG. New BEHAVIOR -> fine-tune. They combine; they are not rivals.
🔍 Click to zoom - three doors, and the rule that sorts them
LiveKnowledge vs behavior - the line that ends the argument3 min

The cleanest test: are you trying to teach the model new FACTS, or a new WAY of answering?

  • New facts - your products, policies, tickets, contracts - belong in RAG. Facts change, need citing, and should be deletable. Baking them into the model is expensive and un-auditable.
  • New behavior - "always answer in our house style", "always return JSON in this shape", "adopt this clinical tone" - is what fine-tuning is for. It shapes how the model responds, not what it knows.
  • They stack. A mature system often fine-tunes for voice AND uses RAG for facts. The question is never "which one" as a rivalry - it is "which one solves the problem in front of us".
The phrase to listen for When someone says "let's fine-tune the model on our documentation", nine times out of ten they want the model to answer from the documentation - that is RAG. Fine-tuning on documents teaches style and vocabulary, not reliable recall of what those documents say.
Live"Just paste it all in" - when the big prompt is and isn't the answer3 min

Modern models accept enormous prompts, so a fair question is: why retrieve at all - why not paste the whole knowledge base every time? For one small document, that is genuinely the right, simple move. At any real scale it breaks down:

  • Cost. You pay for every token in, on every single call. Pasting a 300-page manual to answer one question, thousands of times a day, is a bill nobody approves twice.
  • Focus. Bury the one relevant paragraph in 300 pages and the model's accuracy drops - the well-documented "lost in the middle" effect. Retrieval hands it a clean, short context instead.
  • Citations. When everything is in the prompt, "where did this come from" has no good answer. When a retriever picked five passages, the answer points right at them.
Real world

The proof-of-concept that shocked finance. A team (anonymized) shipped a "paste the whole handbook" assistant that worked beautifully in the demo. The first month's API bill, at production volume, was larger than a year of the vector-database plan RAG would have needed. The fix was not a better model - it was retrieving five passages instead of five hundred pages.

Self-studyDoes RAG even pay off here? The two-question screen2 min read

RAG is not free - it adds a search system, a place to store embeddings, and a pipeline to keep them fresh. Before funding one, two questions sort most cases:

QuestionIf yesIf no
Does the answer live in documents the model was not trained on (and that change)?RAG is a strong fitA plain prompt may be enough
Do you need to trust, cite, or audit the answer?RAG earns its keep - it grounds and citesLong-context might do for a one-off

Two yeses is the classic RAG use case: a support bot, an internal policy assistant, a research helper over your own corpus. Two nos and you are over-engineering - the discipline the whole course keeps returning to.

Exercise 1 of 2

Sort your own "AI on our data" ideas ★ 12 min · pen and paper

The three doors become yours the moment you sort real proposals through them. No code - a pen, the diagram above, and eight honest minutes.

List 6 real ideas from your organization that involve "using our data with AI" - a support bot, a contract assistant, a branded-tone writer, a policy Q&A, anything proposed or rumored.

For each, ask the line-ending question: is this new knowledge (facts from documents) or new behavior (a way of answering)? Write RAG, fine-tune, or both next to it.

For the RAG ones, run the two-question screen: does the knowledge live in changing documents, and do you need to cite or audit the answer? Mark the ones with two yeses - those are the real RAG projects.

Circle the one idea most likely to be mislabeled "fine-tuning" when it is really RAG. That circle is today's return on 45 minutes.

★ Your sorting partner (paste into any chat AI)You are my sorting partner for AI-on-our-data ideas. I will list 6 ideas. For each, ask me: (1) is the goal new knowledge (facts from documents) or new behavior (a way of answering)? (2) if knowledge, does it live in documents that change, and do we need to cite or audit the answer? Then label each idea RAG, fine-tuning, both, or "just a prompt", and flag any idea I seem to be mislabeling. Be stingy about complexity. Here are my ideas: [list your 6]
Exercise 2 of 2

The "is it really RAG" vendor screen ★ 8 min · your next vendor meeting

Four questions that tell you whether a vendor's "AI that uses your data" is real retrieval or a demo trick - politely, in one meeting.

"When our documents change, how fast does the assistant's answers change?" Real RAG updates as fast as you edit a file. "We retrain quarterly" means it is not retrieval.

"Can it show me which source passage each answer came from?" Grounded retrieval cites. If it cannot point at a source, you cannot audit it.

"What does it do when the answer isn't in our documents?" A good system refuses or says "I don't know". One that always answers is one that will confidently make things up.

"Where do our documents and their embeddings live, and who can retrieve them?" This is the data-governance question - and the one session a5 turns into a checklist.

Keep it curious, not prosecutorial Vendors with real retrieval enjoy these questions - the answers are their selling points. Only the ones running a paste-it-all-in demo find them uncomfortable.
Homework

Before session a2 ◐ 30 min total

★ Questions to ask your data team this week
  1. Of everything we call "AI on our data", which are really RAG (new knowledge) and which are fine-tuning (new behavior)?
  2. For our top RAG idea, where does the knowledge live, how often does it change, and do we need answers to be cited?
  3. Are we anywhere pasting whole documents into prompts at volume - and what is that costing per month?
  4. When our assistant does not know something, does it refuse, or does it answer anyway?
  5. What is the simplest version of our most ambitious idea - and have we tried it before building a pipeline?
Source material

Official sources covered

The leader track teaches from published engineering guidance and course curricula - no vendor marketing. This page covers:

OpenAI cookbook · Question Answering Using EmbeddingsPart 1 · knowledge as prompt "memory" · the retrieve-then-answer pattern
Anthropic · Contextual Retrieval (intro framing)Part 1 · why retrieval beats memorizing · deep numbers land in a3/a4
RAG vs fine-tuning vs long-context (industry consensus)Part 2 · the three doors and the knowledge-vs-behavior rule
Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · The three moves of RAG, in order, are...

Retrieve, augment, generate. The model only ever sees the passages the search step pulled - never the whole knowledge base.

2 · A team says "let's fine-tune the model on our support docs so it can answer questions about them." The best reframe is...

New knowledge (facts from documents) is RAG's job; fine-tuning changes style and behavior. Fine-tuning on docs teaches vocabulary, not reliable recall.

3 · The single biggest reason NOT to just paste your whole knowledge base into every prompt is...

Cost scales with every token in on every call, and the "lost in the middle" effect hurts accuracy. Retrieval hands the model a short, focused, citable context instead.

Leader session 1 cheat sheet · pin this

RAG in one breathSearch our documents for the passages that matter, paste just those into the prompt, answer from them - with a citation.
The three movesRetrieve → augment → generate. The model sees the retrieved passages, never the whole base.
Three doorsRAG adds knowledge · fine-tuning changes behavior · a bigger prompt just pastes it all in.
The line that ends the argumentNew FACTS → RAG. New WAY of answering → fine-tune. They stack; they are not rivals.
Why not memorizeYour knowledge changes and the model's training does not; memorized facts cannot be cited or deleted.
Why not paste it allYou pay per token every call, accuracy drops in huge contexts, and there is nothing to cite.
Does it pay off?Knowledge in changing docs + a need to cite or audit = the classic RAG case. Neither = over-engineering.
Vendor screenHow fast do answers change when docs change? Can it cite a source? What does it do when it doesn't know?