The gap between "it works" and "it keeps working"
Everything so far assumed a fixed pile of documents and one query at a time. Production breaks both assumptions. Documents change daily, so your index goes stale. Thousands of queries arrive, so cost and latency stop being free. And no one is watching the demo anymore, so quality drifts unnoticed. Production RAG is the discipline of keeping a retrieval system fresh, fast, cheap, and honest while it runs unattended. None of it is glamorous. All of it is why some RAG systems last years and others die in a month.
Keeping a RAG system alive 11 min live
Four forces act on a live RAG system: documents drift (freshness), the same work repeats (caching), users wait (latency), and quality slips silently (monitoring) - all under a cost ceiling. Here is the production loop and where the milliseconds and dollars go.
LiveFreshness: re-index without re-doing everything3 min▶
Your documents will change - new pages, edits, deletions. The naive move is to re-embed the entire corpus on a schedule, which is slow and expensive. The production move is to re-embed only the chunks that changed and re-index those, leaving the rest untouched.
- Track what changed. Hash each chunk (or watch source timestamps). If the hash is unchanged, skip it - its vector is still valid.
- Re-embed the delta, re-index it. Only new and edited chunks hit the embedding model; deleted ones are removed from the index.
- Cadence is a design decision, not a fixed number. There is no authoritative "re-index every N hours". A legal knowledge base that changes quarterly and a news feed that changes hourly need different cadences. Pick yours from how fast your documents actually move and how stale an answer you can tolerate.
LiveCaching: stop paying for the same work twice3 min▶
Embedding is deterministic: the same text always produces the same vector. So embedding a document you have already embedded is pure waste. Two caches pay off:
- Cache document embeddings. Compute each chunk's vector once and store it. This is really what your vector store is - the point is to never re-embed unchanged chunks (ties straight back to freshness).
- Optionally cache frequent answers. If the same popular question arrives constantly, cache its final answer (with a sensible expiry) and skip retrieval and generation entirely. Watch staleness - a cached answer over changed docs is a subtle bug.
Prompt caching for the RAG prompt itself. Anthropic's contextual retrieval technique contextualizes every chunk once at index time - and keeps that one-time cost low (roughly $1.02 per million document tokens) precisely by using prompt caching. The same idea helps at query time: if a large, stable block of context or instructions repeats across requests, prompt caching means you are not re-paying to process it every call.
LiveLatency budget: know where the time goes3 min▶
A user-facing RAG query has a time budget, and you should be able to name where every millisecond goes. Roughly, in order:
- Embed the query. One short embedding call. Small.
- Vector search. A good ANN index returns in well under 100ms even over millions of vectors. Cheap.
- Rerank (optional). The cross-encoder or rerank API you added in b5 costs real time but sharply improves precision - a deliberate trade.
- LLM generation. Almost always the largest slice, and it grows with output length. This is where you spend your latency budget, so it is where optimization (streaming, smaller models, shorter answers) pays back most.
LiveMonitoring: catch drift before users do2 min▶
In the demo you watched every answer. In production nobody does - so you instrument instead. Log the signals that tell you Recall's quality is slipping:
- Retrieval quality signals. Log top scores, whether the top-k cleared your b7 refusal threshold, and periodically run the b9 eval on a sample of real traffic.
- Refusals. Track how often Recall says "I don't know". A sudden spike means retrieval broke (or docs went stale); a drop to zero can mean the refusal guard stopped working and hallucinations are leaking through.
- Latency and cost per query. Watch the budget and the meter over time - both creep quietly.
The graduation, and the handoff to LangChain 10 min live
Ten sessions ago Recall was an idea. Here is everything you built, in one picture - split into what happens once at index time and what happens on every query - and where it goes next.
LiveThe 2026 frontier: contextual and agentic RAG3 min▶
RAG did not stop moving. Two directions are where the field is heading, and both build directly on what you now understand:
- Contextual retrieval. Anthropic's technique prepends a short, chunk-specific context blurb to each chunk before embedding, so an isolated chunk carries the meaning of its whole document. It measurably cuts failed retrievals, and prompt caching keeps the one-time cost low.
- Agentic RAG. Instead of always retrieving, the model decides when to call a retrieve tool - and can retrieve multiple times, refine its query, or combine sources - as steps in a reasoning loop. Retrieval becomes one tool an agent chooses to use, not a fixed first step.
Self-studyRecall's graduation checklist2 min read▶
Recall is production-shaped when it can answer yes to all of these. This is your handoff document - the difference between a demo and a system.
- Freshness plan. Only changed chunks are re-embedded and re-indexed, on a cadence tied to how fast your docs actually change.
- Caching. Document embeddings computed once; frequent answers cached with expiry; prompt caching for stable repeated context.
- Latency budget. A named target with time allocated across embed, search, rerank, generate - and generation is the slice you optimize.
- Monitoring. Retrieval scores, refusal rate, latency, and cost logged; a sample of real traffic run through the b9 eval.
- Cost cap. A ceiling per month, with alerts, so nothing runs away silently.
- Eval gate. No change ships unless the b9 scorecard holds or improves.
Write Recall's production readiness checklist ★ 12 min
Your final build-along is a document, not code - and it is the most valuable thing you will write in this track. Turn the graduation checklist into a concrete, filled-in plan for Recall, with real numbers you would actually defend.
Freshness plan. How do your documents change, how fast, and what re-index cadence follows? State the staleness promise ("answers reflect changes within X").
Latency budget. Write your target (e.g. under 2s to first token) and allocate ms across embed / search / rerank / generate. Name the slice you would cut first if over budget.
Monitoring plan. List the exact signals you will log - top retrieval scores, refusal rate, latency, cost per query - and what threshold triggers an alert.
Cost cap + eval gate. Set a monthly ceiling and the rule that no change ships unless the b9 scorecard holds. Now Recall has a constitution, not just code.
After the track ◐ light · your choice
- Ship Recall somewhere real, even small - your own notes, a team wiki, a docs site. A RAG system you actually use teaches more in a week than any exercise.
- Try contextual retrieval on one corpus: prepend a chunk-specific context blurb before embedding and re-run your b9 eval. Watch context recall move.
- Sketch what Recall would look like as an agent tool - when should the model retrieve, when should it answer directly, when should it retrieve twice? That question is the doorway to the next course.
- Then start learn-langchain-with-phoebe and wire Recall's retriever into an agent. This is where it gets fun.
That is the whole track. You started not knowing what an embedding was, and you finish with a production-shaped, evaluated retrieval assistant you built yourself. Be proud of that - and keep building. See you in LangChain. - Phoebe
Official sources covered
Taught from Anthropic's production RAG writing plus Pinecone and LlamaIndex operational guidance. This page covers the working ideas; hosted-infra specifics stay with the source.
Three questions before you go 🎯 ◐ 90 seconds
1 · Your knowledge base updates daily. What is the production-right way to keep the index fresh?
Embeddings are deterministic, so unchanged chunks keep valid vectors. Re-embed only the delta (new/edited chunks) and re-index them - re-embedding everything is the classic cost sink.
2 · In a typical query-time latency budget, which stage is usually the largest?
Vector search is well under 100ms and query embedding is tiny. Generation is almost always the largest slice and grows with output length - so it is where latency optimization pays off most.
3 · What best describes agentic RAG, the direction the field is heading?
In agentic RAG retrieval becomes a tool the model chooses to use - it can retrieve zero, one, or several times and refine its query - orchestrated in a reasoning loop (the LangChain/LangGraph handoff).