The leader track 🤝 · for C-level, managers & curious minds · 6 x 45 min · no code, ever
Thinking mode: why evaluation is the thing that lets you ship AI with confidence, what the metrics actually mean, and the scorecard to insist on - each session ends with questions to ask your data team.
Why evaluate at all?
Vibes do not scale. The cost of a confident wrong answer, and why eval is what turns a demo into something you can ship.
🎯Leader 2 · easyWhat "good" actually means
The RAG Triad, retrieval quality vs answer quality, and the handful of metrics a leader must be able to recognize.
🔀Leader 3 · mediumOffline vs online evaluation
Test against a fixed set before you ship; monitor live traffic after. When each applies and what each catches.
📋Leader 4 · mediumThe scorecard to demand
The five numbers to require before trusting or shipping an AI system - and how to read them without a data-science degree.
⚠️Leader 5 · decidingRisk, drift & governance
Regressions, model updates, drift, gaming the metric, and when you can (and cannot) trust an LLM to grade an LLM.
🏛️Leader 6 · decidingBuilding an eval culture
Who owns eval, eval-driven development, the roadmap from vibes to a gate, and where this is heading for agents.
The builder track 🛠️ · for practitioners · 10 x 45 min · Python + a browser scorecard
You build the whole eval discipline and point it at Recall, the RAG assistant from the RAG course. Every metric is felt in the live scorecard before you write the Python.
Your first metric
Hit Rate and MRR over a golden set, live in the browser. Watch the score move with k, and learn rank over threshold.
🗃️Builder 2 · easyBuilding a golden set
Size, diversity, synthetic vs curated, the edge cases to include - and structuring questions for automated grading.
🎣Builder 3 · mediumRetrieval metrics
Context precision and recall, hit rate, MRR, and NDCG - the numbers that tell you the retriever is doing its job.
✍️Builder 4 · mediumGeneration metrics
Faithfulness, response relevancy, noise sensitivity - measuring whether the answer is grounded, on-topic, and robust.
⚖️Builder 5 · hands-onLLM-as-judge
Pointwise, pairwise, reference-guided. Position, verbosity, and self-preference bias - and how to calibrate around them.
🧬Builder 6 · hands-onRAGAS in practice
Run the metric suite over Recall, generate a test set, and read the results - the framework the whole field reaches for.
✅Builder 7 · hands-onAnswer correctness
Factual correctness (claim F1) plus semantic similarity against a reference - grading against ground truth.
🚦Builder 8 · hardestRegression eval suites
The eval suite as a test suite. Threshold gates in CI so a prompt or retrieval change cannot quietly break Recall.
🔬Builder 9 · hardestTracing & observability
Trace, span, run. LangSmith, Langfuse, and Phoenix - turning a black-box bad answer into an inspectable timeline.
📡Builder 10 · hardestOnline eval & drift
Sampling live traffic, user feedback as scores, catching data and concept drift - and keeping Recall honest in production.
The running project: grade Recall 📊
Evaluate the RAG assistant you already met
The builder track points every technique at Recall, the retrieval assistant from learn-rag-with-phoebe, over the same three corpora. The RAG course ended by asking "is it good?" - this course answers it with numbers. The live scorecard runs a real 12-question golden-set evaluation in your browser (Hit Rate, MRR, Precision@1) - change k and watch the misses appear.
Choose your path 🗺️
Two doors, one course. Leaders never see code; builders never trust a metric they have not first watched move.
The knowledge map 🧠
Both tracks at a glance - hover a session to spotlight its concepts, click any node to jump in.