learn-rag-with-phoebe / Leader session 2 of 6
Learn RAG with Phoebe · Leader track · Session 2 of 6

Anatomy of a RAG system

A RAG system is not one thing you buy - it is nine moving parts, and each one is a decision your team either made on purpose or fell into by accident. This session hands you the whole machine on a single page, then teaches you to read it as a set of questions: what happens once when we build the index, what happens every time someone asks, and which of the nine parts is quietly the weakest. By the end you can walk into a design review, point at any part, and ask the one question that tells you whether it was thought through.

🟢 Leader track Leaders: C-level · managers No code, ever 45 min
0-3 · Welcome 3-20 · Concepts 20-40 · Exercises 40-45 · Q&A
Part 0

Why knowing the parts changes the conversation

In session a1 you learned the one-breath definition: retrieve the passages that matter, augment the prompt, generate a grounded answer. That is the idea. This session is the machine that makes the idea real. When a vendor or a team says "we built a RAG system", they made nine separate choices - how documents get in, how they were cut up, how they are searched, whether results are re-scored, whether answers are cited, and how anyone knows it works. Most of those choices are invisible until something goes wrong. Learn the nine parts and each one becomes a question you can ask before it goes wrong - which is the whole point of being the person funding the build.

Live - presented in session Self-study - read after class ★ Try it now prompt Sources covered at the end
★ What you walk out with today The nine parts of a RAG system in plain English, the split between index-time work (done once) and query-time work (paid per question), a leader question for every part, and a red-yellow-green way to spot the weakest part of any system you already have.
Part 1 · covers the whole pipeline and index-time vs query-time

The pipeline on one page 9 min live

Here is the entire machine. Read it left to right in two halves. The first half - documents in, chunked, embedded, stored - is index time: you do it once (and repeat only when documents change). The second half - a question comes in, gets retrieved, reranked, augmented, answered, cited - is query time: it runs on every single question, so it is where you pay per use. Underneath runs an evaluate loop that watches the whole thing.

INDEX TIME · done once (repeat only when docs change) QUERY TIME · runs on every question 1 · Documents the raw sources 2 · Ingest load + clean 3 · Chunk cut into passages 4 · Embed meaning as numbers 5 · Store vector database 6 · Query the question 7 · Retrieve fetch top passages 8 · Rerank re-score the best 9 · Augment passages + query 10 · Generate the model answers 11 · Cite point at the source Evaluate loop is retrieval finding it? are answers grounded? Index once and store it; then pay per question at query time. The two halves have very different cost shapes.
🔍 Click to zoom - the whole RAG machine, split into index-time and query-time
LiveIndex time vs query time - the split that governs your bill3 min

The single most useful line to draw through a RAG system is the line between index time and query time, because they cost money in completely different ways.

  • Index time is a one-off (per batch of documents). Ingesting, chunking, embedding, and storing your documents happens once. You pay it again only when documents change or you re-index. It is a project cost, not a running cost.
  • Query time is a per-question cost. Every question runs retrieve, rerank, augment, generate, cite. This is the meter that spins all day, forever. A design that is a little wasteful here is a bill that grows with adoption.

Leaders who miss this line fund the wrong thing - they worry about the one-time indexing cost and ignore the per-query cost that actually scales. Session a3 puts real numbers on both. For now, just hold the shape: build once, pay per use.

★ Try it now (any chat AI)Explain to me, using a library as the analogy, the difference between the one-time work of cataloguing every book (index time) and the per-visit work of answering a reader's question (query time) in a retrieval system. Then tell me which of the two costs grows as more people use the system.
LiveThe nine parts in plain English4 min

Strip away the jargon and every RAG system is these nine jobs. Six are the moving parts leaders fund; the diagram numbers a few sub-steps, but this is the list to memorize.

PartWhat it does, in one lineWhen
IngestPull documents in and clean them upIndex
ChunkCut each document into passage-sized piecesIndex
EmbedTurn each passage into a list of numbers that captures its meaningIndex
StoreKeep those numbers in a vector database you can search fastIndex
RetrieveFor a question, fetch the closest-in-meaning passagesQuery
RerankRe-score the fetched candidates and keep the truly best fewQuery
AugmentAssemble the passages plus the question into one promptQuery
GenerateHave the model write an answer from the supplied passagesQuery
CiteAttach the source each part of the answer came fromQuery

Notice how ordinary each job is. There is no magic - just a pipeline. The skill is knowing which job is weak, which is why Part 2 turns every one of these into a question you can ask.

Self-studyA document becomes "nodes" - the vocabulary you will hear2 min read

Your team will not always say "chunk". The most common framework, LlamaIndex, calls the raw input a Document and each chunk a Node - so "we generated 40,000 nodes from 1,200 documents" just means "we cut 1,200 files into 40,000 passages". Do not let the vocabulary hide the plainness of the idea: a node is a chunk, a chunk is a passage, and the size of those passages is a real design choice you will interrogate in Part 2. Knowing the word lets you follow the conversation; knowing it is just a chunk keeps you from being impressed by it.

Part 2 · covers the LlamaIndex five stages and the question each part earns you

What each part is for, and the question it earns you 8 min live

The industry has a tidy way to group the nine jobs into five stages, and it is worth borrowing because it is how your team already thinks. LlamaIndex names them Loading, Indexing, Storing, Querying, and Evaluation. The first three are index-time; querying is per-question; evaluation runs across everything. Here they are as a ladder - and then each rung becomes a question you can ask out loud.

1 · Loading ingest documents into the system index time 2 · Indexing chunk + embed into searchable form index time 3 · Storing keep the index so you do not rebuild it index time 4 · Querying retrieve + rerank + augment + generate + cite per question 5 · Evaluation measure retrieval quality + answer groundedness across all Five stages, one owner each. If nobody owns Evaluation, nobody knows whether the other four work.
🔍 Click to zoom - the five stages as a ladder, from loading to evaluation
LiveEach part, and the question that tells you it was thought through4 min

This is the heart of the session. You do not need to build any of these - you need to ask the question that separates a deliberate choice from an accident.

PartThe question it earns you
Ingest"Which documents are in, which are out, and how do updates get in?"
Chunking"How did we cut our documents, and did we test that choice?"
Embedding"Which embedding model turns our text into meaning, and what did it cost to run over everything?"
Store"Where do the embeddings live, and how fast can we search them?"
Retrieval"How many passages do we fetch, and how do we know they're the right ones?"
Reranking"Do we re-score results before showing them, or do we trust the first search?"
Augment"What exactly ends up in the prompt, and could sensitive text leak in?"
Generate"Does the model answer only from the passages, or is it allowed to improvise?"
Citations"Can every answer point to its source?"
Evaluation"What's our groundedness score, and who owns it?"

The querying stage, in LlamaIndex terms, is a small chain you will hear named: a retriever fetches candidates, a node postprocessor reranks or filters them, and a response synthesizer writes the answer. Three names, three of your questions - retrieval, reranking, generate.

LiveWhy chunking and citations are the two parts leaders should watch3 min

Of the nine parts, two repay a leader's attention far more than the rest.

  • Chunking decides what "a passage" even is. Cut too big and the model drowns in irrelevant text; cut too small and a passage loses the context that made it meaningful. This one choice, made early and rarely revisited, quietly caps the quality of everything downstream. Ask whether it was tested, not just picked.
  • Citations are your audit trail. A system that cannot point at the source of an answer cannot be trusted in any regulated or high-stakes setting. Citations are not a nice-to-have feature bolted on at the end - they are a design decision made back at chunking, because you can only cite what you tracked from the start.
Real world

The assistant that could not cite. A team (anonymized) shipped a helpful internal Q&A bot, then legal asked "show me where each answer came from." The system had chunked documents without keeping track of which file each passage came from - so it could answer but never cite. Retrofitting citations meant re-doing the ingest and chunk steps. A question at design review - "can every answer point to its source?" - would have caught it in an hour instead of a quarter.

The tell of a mature system Ask "who owns evaluation?" If the answer is a name and a number - "Priya owns it, groundedness is at 0.9" - the system is engineered. If the answer is a shrug or "the whole team", nobody is measuring whether retrieval actually finds the right passages, and the other eight parts are running blind.
Self-studyHow LangChain names the same parts2 min read

You will hear the same nine jobs under different names depending on which toolkit your team uses. LangChain, the other common framework, describes its pipeline as: load documents, split them with a text splitter, create embeddings, put them in a vector store, then a retriever runs a similarity search that fetches the top k passages, which are handed to the model to generate. It is the exact same machine - loading, chunking, embedding, storing, retrieving, generating - just with the plumbing named differently. When your team says "we set k to 5", they mean "we fetch the five closest passages", which is your retrieval question in disguise. Do not let two vocabularies convince you there are two different systems; there is one anatomy, and you now know all nine parts of it.

Exercise 1 of 2

Map your own system's nine parts ★ 12 min · pen and paper

The anatomy becomes yours the moment you draw it for a real system. Take the RAG idea you sorted as a "real RAG project" in session a1 - or one your team already runs - and lay its nine parts on paper.

Write the nine parts down the left of a page: ingest, chunk, embed, store, retrieve, rerank, augment, generate, cite. Draw the index-time / query-time line after "store".

For each part, mark one of three things: exists (we do this on purpose), missing (we do not do this at all), or accidental (it happens but nobody decided how).

Now write the owner's name next to each part. Any part with no name is a risk - it means a real decision is being made by default, not by a person.

Circle the one part that is both important and un-owned. That circle is the single most valuable output of this exercise - it is the conversation to have this week.

★ Your mapping partner (paste into any chat AI)You are helping me map the anatomy of a RAG system I am responsible for. The nine parts are ingest, chunk, embed, store, retrieve, rerank, augment, generate, cite. I will describe my system in a few sentences. For each of the nine parts, ask me one sharp question to find out whether it exists on purpose, is missing, or happens by accident, and who owns it. At the end, tell me which part looks weakest and why. Here is my system: [describe it]
Exercise 2 of 2

The "which part is weakest" walkthrough ★ 8 min · a system you have

A traffic-light pass over the nine parts turns a vague worry - "I'm not sure our bot is any good" - into a specific, fundable fix. Rate each part red, yellow, or green.

Go part by part and assign a colour. Green: we chose this deliberately and tested it. Yellow: we chose it but never checked it works. Red: it is missing, or nobody knows.

Pay special attention to the two you learned to watch: chunking (was the cut tested?) and citations (can every answer point to a source?). A red on either is a headline finding.

Look at evaluation last. If evaluation is red, quietly downgrade your confidence in every green - because a green you never measured is really a yellow.

Pick the single reddest part that is also cheap to fix. That is your first move - and session a3 will help you see why the cheap fixes are usually chunking, reranking, and a refusal threshold, not a bigger model.

Weakest-link thinking beats average thinking A RAG system is only as good as its worst part. Nine greens and one red does not average out to "pretty good" - the one red part (say, no citations, or untested chunking) can sink the whole thing. Fund the reddest part first, not the flashiest.
Homework

Before session a3 ◐ 30 min total

★ Questions to ask your data team this week
  1. Can you walk me through our nine parts - ingest to cite - and tell me which are deliberate and which just happen?
  2. How did we chunk our documents, and did we test that choice against any alternative?
  3. At query time, how many passages do we retrieve, and do we rerank them before the model sees them?
  4. Can every answer point to the exact source passage it came from - and if not, why not?
  5. What is our groundedness score, who owns evaluation, and how often do they look at it?
Source material

Official sources covered

The leader track teaches from published engineering guidance and course curricula - no vendor marketing. This page covers:

LlamaIndex · Understanding RAG (the five stages)Parts 1 & 2 · Loading, Indexing, Storing, Querying, Evaluation · Documents to Nodes · the query chain
LangChain · RAG pipeline (load, split, embed, store, retrieve, generate)Part 2 · the same anatomy under different names · top-k retrieval
DeepLearning.AI · Building Applications with Vector DatabasesPart 1 · what the store step is and why fast search matters
Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · The difference between index-time and query-time work is best described as...

Index once, then pay per question. Index-time work is a project cost; query-time work is the running cost that grows with adoption.

2 · "Chunking" in a RAG system means...

Chunking cuts documents into passages - LlamaIndex calls each one a Node. The size of the cut is a real, testable design choice that caps downstream quality.

3 · Which part of the anatomy most directly enables an answer to point at its source?

Citations are only possible if the system tracked each passage's source from the start. That is why "can every answer cite its source?" is a design-review question, not a final-feature request.

Leader session 2 cheat sheet · pin this

The nine partsIngest, chunk, embed, store, retrieve, rerank, augment, generate, cite. One pipeline, no magic.
Index time vs query timeIndex once (ingest → store); pay per question at query time (retrieve → cite). The meter spins at query time.
The five stagesLlamaIndex: Loading, Indexing, Storing, Querying, Evaluation. First three are index-time.
Documents to nodesA Document is your file; a Node is a chunk. "40k nodes" just means "40k passages".
The query chainRetriever fetches → node postprocessor reranks → response synthesizer answers.
Two parts to watchChunking (was the cut tested?) and citations (can every answer point to a source?).
Weakest linkA system is only as good as its worst part. Fund the reddest part, not the flashiest.
Who owns evaluation?A name and a number = engineered. A shrug = the other eight parts run blind.