Why this hour is worth your calendar
An LLM on its own is a brilliant colleague with no access to your files and a habit of answering confidently even when it is guessing. RAG is the fix the whole industry converged on: instead of retraining the model or pasting your entire drive into the prompt, you retrieve the handful of passages that actually bear on the question and let the model answer from those. It is cheaper than fine-tuning, more current than the model's training data, and - done right - it can cite its sources and admit when it does not know. That last part is why leaders care: RAG is the difference between an assistant that sounds right and one you can audit.
Retrieve, augment, generate 9 min live
RAG is three moves in a row, and the names say exactly what happens. The model never sees your whole knowledge base - it sees the few passages a search step pulled for this one question. The picture below is the entire concept.
LiveThe one-breath definition3 min▶
Here is RAG in a sentence you can say out loud in a meeting: "When someone asks a question, we search our own documents for the few passages that matter, paste just those into the prompt, and have the model answer from them - with a citation."
- Retrieve: a search step finds the most relevant passages. Not keyword search - meaning search, which is why "how do I get my money back" can find a paragraph titled "Refunds" that shares no words with the question.
- Augment: those passages are added to the prompt alongside the question. The model's context becomes "here is what our documents say; now answer."
- Generate: the model writes an answer grounded in that supplied text, ideally quoting or citing the source it used.
The OpenAI cookbook frames it well: this is giving the model short-term memory through the prompt, which is more reliable for factual recall than baking facts into the model itself.
LiveWhy not just let the model memorize everything?3 min▶
Two reasons, and both matter to a budget holder.
- Your knowledge changes; the model's training does not. A price list, a policy, last week's incident - none of that is in the model. Retrieval reads whatever is in your documents right now, so the answer is as current as your files.
- Memorized facts cannot be cited or removed. When knowledge lives in the model's weights you cannot point to where an answer came from, and you cannot delete one document's influence. When it lives in retrievable documents, every answer can carry a source, and removing a document removes its answers. That is the auditability leaders keep asking for.
The policy that changed on a Friday. A support team (anonymized) updated its refund window from 14 to 30 days. With RAG, the change was one edited document and the bot answered "30 days" that afternoon. The team that had fine-tuned a model on the old policy needed a retraining cycle - and in the meantime the model kept confidently saying "14".
Self-studyMeaning search, in one paragraph2 min read▶
The "retrieve" step does not match words - it matches meaning. Every passage is turned into a list of numbers (an embedding) that captures what it is about, and the question is turned into the same kind of list. The search finds the passages whose numbers point in the same direction as the question's. You do not need the math today - the builder track's first session lets you feel it in a live playground - but the leadership takeaway is simple: RAG can find the right paragraph even when it shares no keywords with the question. That is what makes it useful on real, messy documents, and it is also why it sometimes retrieves something plausible but wrong - which is exactly what session a4 teaches you to measure.
Three doors for "use our data with AI" 8 min live
When a team wants an AI to use company knowledge, there are three doors - and they are constantly confused for each other. RAG adds knowledge. Fine-tuning changes behavior. A bigger prompt just pastes everything in. Most "we need to fine-tune on our data" requests are really RAG requests.
LiveKnowledge vs behavior - the line that ends the argument3 min▶
The cleanest test: are you trying to teach the model new FACTS, or a new WAY of answering?
- New facts - your products, policies, tickets, contracts - belong in RAG. Facts change, need citing, and should be deletable. Baking them into the model is expensive and un-auditable.
- New behavior - "always answer in our house style", "always return JSON in this shape", "adopt this clinical tone" - is what fine-tuning is for. It shapes how the model responds, not what it knows.
- They stack. A mature system often fine-tunes for voice AND uses RAG for facts. The question is never "which one" as a rivalry - it is "which one solves the problem in front of us".
Live"Just paste it all in" - when the big prompt is and isn't the answer3 min▶
Modern models accept enormous prompts, so a fair question is: why retrieve at all - why not paste the whole knowledge base every time? For one small document, that is genuinely the right, simple move. At any real scale it breaks down:
- Cost. You pay for every token in, on every single call. Pasting a 300-page manual to answer one question, thousands of times a day, is a bill nobody approves twice.
- Focus. Bury the one relevant paragraph in 300 pages and the model's accuracy drops - the well-documented "lost in the middle" effect. Retrieval hands it a clean, short context instead.
- Citations. When everything is in the prompt, "where did this come from" has no good answer. When a retriever picked five passages, the answer points right at them.
The proof-of-concept that shocked finance. A team (anonymized) shipped a "paste the whole handbook" assistant that worked beautifully in the demo. The first month's API bill, at production volume, was larger than a year of the vector-database plan RAG would have needed. The fix was not a better model - it was retrieving five passages instead of five hundred pages.
Self-studyDoes RAG even pay off here? The two-question screen2 min read▶
RAG is not free - it adds a search system, a place to store embeddings, and a pipeline to keep them fresh. Before funding one, two questions sort most cases:
| Question | If yes | If no |
|---|---|---|
| Does the answer live in documents the model was not trained on (and that change)? | RAG is a strong fit | A plain prompt may be enough |
| Do you need to trust, cite, or audit the answer? | RAG earns its keep - it grounds and cites | Long-context might do for a one-off |
Two yeses is the classic RAG use case: a support bot, an internal policy assistant, a research helper over your own corpus. Two nos and you are over-engineering - the discipline the whole course keeps returning to.
Sort your own "AI on our data" ideas ★ 12 min · pen and paper
The three doors become yours the moment you sort real proposals through them. No code - a pen, the diagram above, and eight honest minutes.
List 6 real ideas from your organization that involve "using our data with AI" - a support bot, a contract assistant, a branded-tone writer, a policy Q&A, anything proposed or rumored.
For each, ask the line-ending question: is this new knowledge (facts from documents) or new behavior (a way of answering)? Write RAG, fine-tune, or both next to it.
For the RAG ones, run the two-question screen: does the knowledge live in changing documents, and do you need to cite or audit the answer? Mark the ones with two yeses - those are the real RAG projects.
Circle the one idea most likely to be mislabeled "fine-tuning" when it is really RAG. That circle is today's return on 45 minutes.
The "is it really RAG" vendor screen ★ 8 min · your next vendor meeting
Four questions that tell you whether a vendor's "AI that uses your data" is real retrieval or a demo trick - politely, in one meeting.
"When our documents change, how fast does the assistant's answers change?" Real RAG updates as fast as you edit a file. "We retrain quarterly" means it is not retrieval.
"Can it show me which source passage each answer came from?" Grounded retrieval cites. If it cannot point at a source, you cannot audit it.
"What does it do when the answer isn't in our documents?" A good system refuses or says "I don't know". One that always answers is one that will confidently make things up.
"Where do our documents and their embeddings live, and who can retrieve them?" This is the data-governance question - and the one session a5 turns into a checklist.
Before session a2 ◐ 30 min total
- Finish sorting all 6 ideas if you ran out of time, and keep the sheet - session a3 reuses it to talk cost, and a4 to talk trust.
- Run the sorting-partner prompt on the two ideas you were least sure about, and notice where the AI pushes back on your label.
- Use at least two of the four vendor-screen questions in a real conversation this week.
- Optional: skim the builder track's session b1 intro - you do not need the code, but seeing the live retrieval playground makes "meaning search" click.
- Of everything we call "AI on our data", which are really RAG (new knowledge) and which are fine-tuning (new behavior)?
- For our top RAG idea, where does the knowledge live, how often does it change, and do we need answers to be cited?
- Are we anywhere pasting whole documents into prompts at volume - and what is that costing per month?
- When our assistant does not know something, does it refuse, or does it answer anyway?
- What is the simplest version of our most ambitious idea - and have we tried it before building a pipeline?
Official sources covered
The leader track teaches from published engineering guidance and course curricula - no vendor marketing. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · The three moves of RAG, in order, are...
Retrieve, augment, generate. The model only ever sees the passages the search step pulled - never the whole knowledge base.
2 · A team says "let's fine-tune the model on our support docs so it can answer questions about them." The best reframe is...
New knowledge (facts from documents) is RAG's job; fine-tuning changes style and behavior. Fine-tuning on docs teaches vocabulary, not reliable recall.
3 · The single biggest reason NOT to just paste your whole knowledge base into every prompt is...
Cost scales with every token in on every call, and the "lost in the middle" effect hurts accuracy. Retrieval hands the model a short, focused, citable context instead.