The leader track 🎯 · for the people who sign off · 6 x 45 min · no tools required
Judgement mode: why bots fail after a good demo, what "good" honestly looks like per archetype, how to read an eval report without being fooled by a definition, what 90% confident actually means, the risk tiers and the gate - ending with the sign-off you can defend and the rollback triggers you set before you need them.
Why bots fail after launch
The gap between a demo and production, what a wrong launch actually costs, and why the most confident launch stories in the industry were quietly revised a year later.
🧭 Leader 2 · easyThree bot archetypes
Chatbot, text-to-query, analytics - one slide that shows why they need different criteria, and what an honestly good number looks like for each. Hint: it is not 99%.
🔎 Leader 3 · mediumHow to read an eval report
The same models score 0.7% or 12% hallucination depending on who is measuring. The questions to ask before you believe a number, and how to spot a cherry-picked set.
📐 Leader 4 · mediumWhat "90% confident" really means
Three different claims hide behind that phrase. The interval table, live: what 20, 200 and 1,000 examples each buy you, and why a demo set can never be evidence.
🚦 Leader 5 · decidingRisk tiers and the launch gate
Not every bot needs the same bar. Blast-radius tiering, the six conjunctive gates, the staged rollout ladder, who signs what - and the EU rule that lands on every chatbot in Aug 2026.
🔁 Leader 6 · decidingAfter GA
The launch number is a snapshot. Drift, online evals, the rollback triggers you set in advance, and the loop that turns every incident into a test that can never fail twice.
The builder track 🛠️ · for AI engineers, ML engineers & technical PMs · 10 x 45 min · hands-on
Craft mode: the anatomy of a defensible eval, quality and operational metrics with their real formulas, LLM judges and their documented biases, then three full gating exercises - one per archetype - converging on the launch scorecard and the CI pipeline that keeps it honest after you ship.
Eval anatomy
The four parts every defensible number has, the grading ladder that says when a rule beats a judge, and why capability and regression suites must never be the same file.
🎯 Builder 2 · easyQuality metrics
Groundedness, faithfulness, answer relevancy, hallucination rate - the actual formulas, and the uncomfortable fact that changing the definition moves the number tenfold.
⏱️ Builder 3 · mediumOperational metrics
Time-to-first-token, p95 versus the average that hides your unhappiest users, cost per task, and conciseness as the rare metric that is a UX win and a cost win at once.
⚖️ Builder 4 · mediumLLM-as-judge
The four documented biases - position, verbosity, self-enhancement, weak reasoning - and the calibration procedure that turns a judge from an opinion into an instrument.
💬 Builder 5 · mediumGate the chatbot
No ground truth, so outcome metrics carry the weight. Containment versus resolution, the multi-turn collapse single-turn evals never see, and escalation as a gate.
🗄️ Builder 6 · hands-onGate the text-to-query bot
Executable ground truth changes everything. Execution accuracy, why schema linking causes most failures, abstention as a metric, and read-only rails that beat any score.
📊 Builder 7 · hands-onGate the analytics bot
A pipeline of gates: does the chart render, does it encode the right thing, does the takeaway follow from the numbers - and does the same question give the same answer twice.
🧩 Builder 8 · hands-onAgent-specific evals
When the bot takes steps, the answer is not the only thing to grade. Trajectory matching, tool-call accuracy, and pass^k - the metric that says capability is not reliability.
🚀 Builder 9 · hardestThe launch gate
Everything converges. Run all three bots through one live scorecard, watch confidence climb 76.4% to 92.5%, and find out which levers fixed the bot and which only fixed the measurement.
🔧 Builder 10 · hardestEval as CI/CD and in production
Blocking regression gates, a safety canary that must pass at 100%, online evals on sampled traffic, drift detection, and the refresh cadence that keeps the gate meaningful.
Choose your path 🗺️
Four doors into the same conviction: the number you may promise is the lower bound, not the one you observed.
The knowledge map 🧠
Both tracks at a glance - hover a session to spotlight its concepts, click any node to jump in.