Why cost and quality are two different maps
In session a2 you learned the nine parts and the line between index-time work (done once) and query-time work (paid per question). This session puts money and quality on that same anatomy - and the lesson is that the two do not line up. The biggest cost in a RAG system is the model writing the answer, on every call, forever. But the biggest quality gains come from cheap places: how you cut the documents, whether you re-score results, and whether the system knows when to refuse. A leader who understands this stops approving expensive upgrades that barely help and starts funding cheap changes that quietly double the accuracy.
Where the money actually goes 9 min live
RAG cost falls into three buckets, and only one of them grows with use. Index-time embedding is a one-off. Storage is a steady monthly hosting fee. Per-query cost - embedding the question, optionally reranking, and the model generating the answer - is paid on every single question, and inside it the generation step dominates. Picture the three buckets, sized the way they actually behave at scale.
LiveThe three cost buckets, and which one keeps you up at night3 min▶
Every RAG bill sorts into three buckets. Knowing which is which stops you from optimizing the one that does not matter.
- One-time indexing. Embedding all your documents is paid once (again only when you re-index). It can feel large the day you run it, but it is a project cost, not a running one. Do not let its size scare you off a good design.
- Storage. Hosting the vector database is a steady monthly fee that grows slowly with the size of your corpus. Predictable, and usually the smallest of the three at real scale.
- Per-query cost. This is the one that keeps you up at night, because it grows with adoption. It is the query embedding (tiny), optional reranking (small), and the model generating the answer (large). Success - more people using the system - makes this bucket bigger.
The uncomfortable truth: the more successful your assistant, the more the per-query bucket dominates. So the per-query design - especially how much text you feed the model and how often - is the number a leader should watch.
LiveWhy generation dominates - and why "paste everything" is the expensive trap3 min▶
Inside the per-query bucket, the model writing the answer costs far more than the search that found the passages. Search over numbers is cheap and fast; a language model reading a prompt and composing a reply is where the tokens - and the money - are spent. Two consequences follow directly:
- Feed the model less, pay less. The cost of generation scales with how many tokens go in and come out. Handing it five tight passages instead of fifty loose ones is a direct saving on every call.
- The long-context anti-pattern from a1 is a cost story. "Just paste the whole document into every prompt" means you pay for every token in, on every single call. It looks free in a demo of ten questions; at production volume it is the bill that shocks finance. Retrieval exists partly to keep the generation prompt short.
The month the meter moved. A team (anonymized) launched an internal assistant and watched usage climb from a few dozen questions a day to a few thousand. The one-time indexing cost never changed and storage barely moved - but the per-query generation cost tracked usage almost exactly, because each answer called the model. The lever that brought the bill back under control was not a cheaper model; it was retrieving fewer, better passages so each call fed the model less text.
Self-studyA note on why no pricing tables appear here2 min read▶
You will notice this session names shapes and ratios but not dollar-per-token vendor prices. That is deliberate and honest: model and vector-database pricing changes often enough that any table printed here would be wrong within a quarter, and quoting a specific vendor's rate would date the whole course. What does not change is the shape - one-time indexing, steady storage, per-query generation that dominates and scales with use. Learn the shape and you can drop today's live prices into it in five minutes. The one durable cost figure worth carrying is Anthropic's published contextual-retrieval number in the next section, because it is a ratio (cost per million tokens with prompt caching) rather than a list price, and it illustrates how cheap a strong quality lever can be.
Where quality is won - and the cheap levers 8 min live
Now the good news. The levers that most improve answer quality are mostly cheap, and the expensive ones give less than you would hope. Below is the lever board: on the left, high-impact low-cost moves; on the right, high-cost moves that barely move the needle. A leader's job is to keep the team reaching left before it reaches right.
LiveThe cheap wins: chunking, reranking, and knowing when to refuse3 min▶
Three levers give the most quality per dollar, and none of them means buying a bigger model.
- Better chunking. Cutting documents into the right-sized passages helps retrieval find the truly relevant text. It is a one-time index-time change with almost no extra running cost, and it caps the quality of everything downstream - so it pays back on every query forever.
- Adding reranking. Instead of trusting the first search, you fetch a larger candidate set and re-score it, keeping only the best few. Because you rerank a small set, the cost is small - but the accuracy lift is large, since the model now sees the genuinely best passages, not just the first ones the search returned.
- A refusal threshold. Teaching the system to say "I don't know" when retrieval comes back weak costs essentially nothing and buys the thing leaders value most: trust. An assistant that refuses when it should is worth more than one that always answers.
LiveContextual retrieval - real numbers for a real lever3 min▶
Here is a concrete, published example of a cheap lever with big returns. Anthropic's contextual retrieval technique adds a short piece of context (roughly 50 to 100 tokens) to each chunk before embedding it, so a passage carries a hint of where it came from. Measured as the reduction in retrieval failures (the share of times the right passage is missing from the top 20 results, baseline 5.7%):
| Lever added | Reduction in retrieval failures |
|---|---|
| Contextual embeddings | -35% |
| + contextual keyword search (BM25) | -49% |
| + reranking | -67% |
Two-thirds fewer retrieval failures by stacking cheap levers - not by buying a bigger model. And the cost of adding that context is a one-time roughly $1.02 per million document tokens, thanks to prompt caching and a small model (Claude 3 Haiku) doing the contextualizing. That is the shape of a good RAG investment: a small, one-time index-time spend that lifts quality on every query forever.
The upgrade nobody needed. A team (anonymized) was ready to sign off on a far pricier flagship model to "fix accuracy". A quick test first added reranking to their existing retrieval and set a refusal threshold. Accuracy rose enough that the model upgrade was shelved - the fix was in retrieval, at a fraction of the cost, exactly as the lever board predicts.
Self-studyBuild vs buy - the honest frame2 min read▶
Most teams should buy the hard, undifferentiated parts and build only what is theirs. A managed vector database (hosted storage and fast search) and an embeddings API (someone else runs the model that turns text into numbers) let a small team stand up real retrieval in days, with predictable per-query costs and no infrastructure to babysit. Self-hosting the vector store and running your own embedding models can be cheaper at very large scale, or necessary for strict data-residency rules - but it trades a running fee for a team's time and on-call burden, which is rarely the right trade early. The leader's frame: buy the plumbing, build the parts that make your assistant yours - your documents, your chunking choices, your evaluation, your refusal policy. Session a5 turns the data-residency side of this into a governance checklist. Note that fast vector search typically returns in well under 100 milliseconds, so "buy" rarely costs you speed - it is a cost and control decision, not a performance one.
Sketch the cost model for your RAG idea ★ 12 min · pen and paper
Take the RAG idea you mapped in session a2 and put money on it - not exact dollars, but the shape. The goal is to find your dominant cost before your team does.
Draw three boxes: one-time indexing (embed all our docs once), storage (monthly vector DB hosting), and per-query (query embed + optional rerank + generation).
Estimate rough size for each. You will not have exact numbers - use small / medium / large. Ask: how big is our corpus (indexing), and how many questions a day do we expect (per-query)?
Now project forward: at 10x today's usage, which box grows? Circle it. For almost every real assistant it is the per-query box, driven by generation.
Write one sentence: "Our dominant cost is ____, and the lever that controls it is ____." For most, the honest answer is "generation, controlled by feeding the model fewer, better passages".
The "cheapest quality win" audit ★ 8 min · a system you have
For a RAG system you run or are considering, rank the six levers by return on spend. The exercise trains the instinct to reach left on the lever board before reaching right.
List the six levers: better chunking, add reranking, a refusal threshold (the cheap three) and bigger embedding model, retrieve more passages, bigger LLM (the expensive three).
For your system, mark which of the cheap three you have not yet done. Each un-done cheap lever is quality you are leaving on the table for almost no money.
Check whether anyone has proposed an expensive lever - a bigger model, more passages - to "fix accuracy". If so, ask whether the cheap three were tried first. Usually they were not.
Write your first move: the single cheapest lever you have not pulled. For most systems it is reranking or a refusal threshold - large accuracy or trust gain, small cost.
Before session a4 ◐ 30 min total
- Finish your three-bucket cost sketch and write the one-sentence "dominant cost / controlling lever" line. Keep it - a4 connects cost discipline to how you measure quality.
- Run the cheapest-quality-win audit and name the one cheap lever you have not yet pulled.
- Find out whether anyone in your org has proposed a model upgrade to fix accuracy, and whether reranking and a refusal threshold were tried first.
- Optional: read Anthropic's contextual-retrieval write-up for the full numbers - you do not need the code, just the ratios and the one-time cost figure.
- Which of our three cost buckets - one-time indexing, storage, per-query - is biggest today, and which grows fastest as we scale?
- Are we anywhere feeding the model more text than it needs, and what would retrieving fewer, better passages save per month?
- Have we added reranking and a refusal threshold - and if not, why are we considering a bigger model first?
- Could contextual retrieval reduce our retrieval failures, and what would the one-time contextualizing cost be for our corpus?
- For storage and embeddings, are we buying managed services or self-hosting - and is that a cost, control, or data-residency decision?
Official sources covered
The leader track teaches from published engineering guidance and course curricula - no vendor marketing, and no pricing tables reproduced (list prices date too fast; only durable ratios are cited). This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Inside the per-query cost of a RAG system, the dominant expense is usually...
Generation dominates per-query cost and grows with adoption. That is why feeding the model fewer, better passages saves more than almost anything else.
2 · A team says accuracy is too low and wants to buy a bigger language model. The disciplined first move is...
The cheap levers live in retrieval. Anthropic's contextual retrieval stacked cheap levers to cut retrieval failures by two-thirds before any model upgrade was needed.
3 · Why do you rerank only a small candidate set (say ~150 fetched, ~20 kept) rather than reranking the whole corpus?
Fast vector search narrows millions of passages to a small candidate set in under 100ms; reranking that small set is cheap, while reranking everything would be prohibitively slow.