RAG & agentic search

Three problems, one mechanism:

• Knowledge cutoff: the model was trained up to a date; your world moved on.
• Private data: your runbooks, tickets, and contracts were never in training data.
• Context limits: even a huge context window cannot hold your whole wiki, and stuffing it costs money and attention - relevant needles do better in a small, clean haystack.

The mechanism: at question time, retrieve the top-k most relevant chunks from your corpus and paste them into the prompt with the question. Claude answers from what it was shown, grounded and citable.

A chunk is the unit of retrieval: too small and it lacks the context to be understood, too big and one relevant sentence drags in a page of noise. Two families:

• By size: cut every N characters or tokens (typically 500-1,000 tokens) with 10-20% overlap so a sentence straddling a boundary survives in at least one chunk. Dumb, robust, works on anything.
• By structure: split on headings, paragraphs, or sections, so each chunk is a semantically complete unit. Better retrieval when documents HAVE structure - and yours mostly do.

Store metadata with every chunk - source document, section title, URL - as a dict like {"text": ..., "source": ..., "section": ...}. You will need it for citations in Part 2 and for contextual retrieval in Part 3.

An embedding maps a text to a vector such that similar meanings land close together. Search = embed the query, embed the chunks (once, at index time), rank by cosine similarity. Anthropic does not ship an embeddings endpoint - it recommends external providers, and Voyage AI is the usual pick (any embedding provider slots in the same way).

• This IS a vector store for corpora up to tens of thousands of chunks - a numpy matrix and a dot product. Reach for a real vector database (pgvector, Qdrant, Chroma) when you need persistence, filters, or millions of vectors, not before.
• input_type matters: Voyage embeds queries and documents slightly differently for better matching - pass it.
• Embed once, cache forever: chunks only need re-embedding when they change. Store vectors next to the chunks.

Ask "what does error E-4102 mean?" and semantic search shrugs - E-4102 has no meaning to embed. BM25, the classic lexical algorithm behind decades of search engines, nails it: it rewards exact token matches, weighted by rarity. Product codes, error IDs, people's names, SKUs, legal clause numbers - lexical territory, all of it.

Hybrid search runs both indexes and merges: normalize each score list, take a weighted sum (start 50/50), rank by the combined score. That is the "multi-index pipeline" from the official module - two indexes over the same chunks, one merged ranking. Semantic catches paraphrases ("staff offboarding" finds "employee exit process"), lexical catches identifiers, and each covers the other's blind spot.

Question in, cited answer out. Everything above, assembled - this is the skeleton to steal:

• XML-style tags around chunks (6.2's lesson applied): Claude reliably distinguishes retrieved material from the question, and the source attribute enables citations.
• The "say you could not find it" line is load-bearing. Without it, Claude answers from training data when retrieval misses - the classic RAG hallucination, and it looks exactly like a correct answer.
• Agentic search is this flow with the loop from 6.3: expose retrieve as a tool and Claude decides when to search, reads the results, and searches again with a reformulated query if they are weak. Ten extra lines, noticeably better on vague questions.

Fast retrieval is approximate - the right chunk is usually in the top 50 but not always in the top 5. So retrieve wide and cheap, then let a slower, smarter model re-score the candidates against the question and keep the best few. A dedicated reranker (Voyage rerank-2, Cohere Rerank) is fastest; Claude itself works fine as the judge:

Run the candidate scoring concurrently (or as a 6.3-style batch for offline jobs) - it is 50 independent calls. When latency matters, the dedicated reranker does the same job in one call: vo.rerank(question, documents, model="rerank-2", top_k=5).

The quiet failure of chunking: a chunk saying "the fee increases to 3.5% after the first year" embeds fine - but for WHICH product, WHICH contract? The surrounding document knew; the chunk forgot. Anthropic's published fix, contextual retrieval, asks Claude to write a 1-2 sentence situating context for each chunk and prepends it BEFORE embedding and BM25 indexing. Their measurements: substantially fewer retrieval failures (roughly a one-third to one-half reduction, more when combined with reranking).

It costs one small-model call per chunk, once, at index time - and prompt caching (6.5) makes it cheap, since every chunk of the same document reuses the cached doc_text. Index-time spend for query-time quality is almost always a good trade.

"The answers are bad" has two different diagnoses, and 6.2's eval discipline applies to each separately:

• Retrieval metrics: for each test question, did the chunk containing the answer surface in the top k? Build 20-30 question-to-gold-chunk pairs and compute recall@k. No API calls needed, runs in seconds - this is where most failures live, and where chunking, hybrid weights, reranking, and contextual retrieval all show up as measurable deltas.
• Answer metrics: GIVEN the right chunks in context, is the answer correct, cited, and faithful? Grade with the model-based grader from 6.2. Failures here mean prompt work, not retrieval work.