Building a golden set: the data your score depends on

The instinct is to hand-craft a handful of perfect questions. The better instinct, at least once you are automating the grade, is more questions at slightly lower per-item polish.

• Anthropic's guidance is blunt about this. Prioritize volume over quality: more questions with slightly lower-signal automated grading beats fewer, hand-graded ones. Coverage catches more regressions than polish does.
• Statistics needs a denominator. A Hit Rate over twelve questions swings wildly when one flips. Over a hundred, one flip barely moves the number - and you can actually tell whether a change helped.
• Automated grading is what makes volume affordable. If every item needs a human to grade, you cap out fast. Structure for machine grading (Part 2, quality 4) and volume stops being expensive.

A set that over-represents easy questions gives a high score that means nothing in production. The mix of your golden set should look like the mix of questions your system actually gets.

• Mirror the real-world task distribution. If a third of Recall's real traffic is Jira incident lookups (corpus B), roughly a third of the golden set should be too. Match topics, phrasings, and difficulty to reality.
• Sample from real logs when you have them. The truest distribution is the one your users already produced. Pull real queries, strip anything sensitive, and let them seed the set.
• Cover all three corpora. Recall spans personal notes (A), Jira incidents (B), and the company KB (C). A golden set that ignores one corpus cannot catch a regression that only hits that corpus.

The average question is not where systems break. Dedicate part of the set to the hard tail on purpose.

• Reserve cases for edge behavior. Empty results, questions spanning two corpora, near-duplicate chunks, out-of-scope questions the system should decline - these are where scores should drop, and where you want to know.
• Include ambiguous cases where even humans would struggle to reach consensus. Anthropic recommends this deliberately: a question with no single clean answer stress-tests how your system and your grader handle genuine uncertainty.
• Label edge cases so you can slice. Tag them, so you can report the score on the hard tail separately from the easy body. A flat average hides a system that aces the easy and fails the hard.

The final quality is mechanical but decisive: the set has to be gradeable by a machine, or volume and CI are both off the table.

• Pick a grading mode per item. Multiple-choice (exact option match), string match (expected substring or id), code-graded (a function checks the output), or LLM-graded (a model judges against a rubric). Retrieval golden sets are usually string/id match - cheap and deterministic.
• Make the expected field unambiguous. For Recall's retriever the expected field is a chunk id (like C-refund), not free text - so grading is a set membership check, not a fuzzy comparison.
• Save the hardest grading for last. LLM-graded items are powerful but pricey and need their own validation (that is b5 and b6). Start with the cheap deterministic modes wherever the task allows.

The core idea is small: hand the model a chunk, ask it for questions a real user would ask that this chunk answers, and get back the expected answer and the source id along with each question. Here is a minimal generator over Recall's corpora.

When you want more than a per-chunk prompt, RAGAS automates the whole thing. It builds a knowledge graph from your documents, then generates a mix of question types so the set is not all one shape.

• Knowledge graph first. RAGAS builds a graph from your docs and uses it to write questions that need one chunk (single-hop) or several (multi-hop).
• A deliberate mix. The default query distribution is roughly half single-hop and half multi-hop, split between abstract and specific multi-hop questions - so your set exercises both simple lookups and reasoning across chunks.
• testset_size is yours to set. Ask for 60 or 600. Then - same rule - curate before you trust the numbers it produces.

OpenAI's guidance on evals is consistent on one point: an LLM can generate synthetic eval data and even grade it, but you validate that generated set (or model-graded eval) against human judgment before you trust it at scale.

• Sample and check. Have a human review a random slice of the generated set. If the answers or expected ids are wrong too often, the whole set is suspect - fix the prompt or the source, do not paper over it with volume.
• Agreement is the gate. Only once a human agrees with the generator on a good sample should you scale generation up and lean on the numbers. Trust is earned per set, not assumed.
• This applies double to LLM graders. The same validate-against-humans rule governs the model-graded metrics in b5 and b6, where the model is not just writing questions but scoring answers.

Change k and watch the score, but keep your eye on the questions this time, not the metric. A trustworthy set would spread evenly across corpora A, B, and C, include a couple of genuinely hard cases, and have at least one ambiguous entry. Judge this seed set against that bar.

Notice how few questions there are: with twelve, a single flip moves Hit Rate by more than eight points. That volatility is the b1 set telling you it is a seed, not a finished golden set. Growing and diversifying it is your homework.