Learn Agent Launch with Phoebe

Nineteen out of twenty is not ninety-five percent

Every AI bot demos well. That is what a demo is for. The hard question comes after: given all these eval numbers, do you actually ship it - and what exactly are you promising when you say you are confident? This course is the launch gate. It assumes you can already compute the metrics (two sibling courses go deep on those) and teaches the decision instead: how to pick criteria per bot archetype, how big a golden set has to be before it proves anything, how to build a six-gate scorecard where quality cannot buy off cost, and how to write a sign-off sentence that survives a hard question. Three real archetypes run through it end to end - a support chatbot with no ground truth, a text-to-query bot whose answers are executable, and an analytics bot where errors compound across a pipeline. At its centre, a live launch gate: toggle levers on a real Wilson confidence interval computed in your browser and watch the number climb from 76.4% - the honest reading of a nineteen-out-of-twenty demo - to 92.5%, where it can be signed. Every metric, threshold and case number on these pages traces to a primary source.

2tracks
16sessions
3bot archetypes gated
6launch gates
Where this sits among its siblings Learn Evals owns retrieval and generation metrics. Model Evaluation owns the classic and LLM-era metric canon. AI Agents owns what an agent is and how to build one. This course owns the one thing they all leave open: the go/no-go decision, and the evidence that makes it defensible.

The leader track 🎯 · for the people who sign off · 6 x 45 min · no tools required

Judgement mode: why bots fail after a good demo, what "good" honestly looks like per archetype, how to read an eval report without being fooled by a definition, what 90% confident actually means, the risk tiers and the gate - ending with the sign-off you can defend and the rollback triggers you set before you need them.

📉 Leader 1 · easy

Why bots fail after launch

The gap between a demo and production, what a wrong launch actually costs, and why the most confident launch stories in the industry were quietly revised a year later.

▶ Leaders start here 🧭 Leader 2 · easy

Three bot archetypes

Chatbot, text-to-query, analytics - one slide that shows why they need different criteria, and what an honestly good number looks like for each. Hint: it is not 99%.

The contrast 🔎 Leader 3 · medium

How to read an eval report

The same models score 0.7% or 12% hallucination depending on who is measuring. The questions to ask before you believe a number, and how to spot a cherry-picked set.

The grilling kit 📐 Leader 4 · medium

What "90% confident" really means

Three different claims hide behind that phrase. The interval table, live: what 20, 200 and 1,000 examples each buy you, and why a demo set can never be evidence.

The arithmetic 🚦 Leader 5 · deciding

Risk tiers and the launch gate

Not every bot needs the same bar. Blast-radius tiering, the six conjunctive gates, the staged rollout ladder, who signs what - and the EU rule that lands on every chatbot in Aug 2026.

The decision 🔁 Leader 6 · deciding

After GA

The launch number is a snapshot. Drift, online evals, the rollback triggers you set in advance, and the loop that turns every incident into a test that can never fail twice.

The long game

The builder track 🛠️ · for AI engineers, ML engineers & technical PMs · 10 x 45 min · hands-on

Craft mode: the anatomy of a defensible eval, quality and operational metrics with their real formulas, LLM judges and their documented biases, then three full gating exercises - one per archetype - converging on the launch scorecard and the CI pipeline that keeps it honest after you ship.

🧱 Builder 1 · easy

Eval anatomy

The four parts every defensible number has, the grading ladder that says when a rule beats a judge, and why capability and regression suites must never be the same file.

▶ Builders start here 🎯 Builder 2 · easy

Quality metrics

Groundedness, faithfulness, answer relevancy, hallucination rate - the actual formulas, and the uncomfortable fact that changing the definition moves the number tenfold.

What "right" means ⏱️ Builder 3 · medium

Operational metrics

Time-to-first-token, p95 versus the average that hides your unhappiest users, cost per task, and conciseness as the rare metric that is a UX win and a cost win at once.

Speed and money ⚖️ Builder 4 · medium

LLM-as-judge

The four documented biases - position, verbosity, self-enhancement, weak reasoning - and the calibration procedure that turns a judge from an opinion into an instrument.

Grading the grader 💬 Builder 5 · medium

Gate the chatbot

No ground truth, so outcome metrics carry the weight. Containment versus resolution, the multi-turn collapse single-turn evals never see, and escalation as a gate.

Archetype 1 🗄️ Builder 6 · hands-on

Gate the text-to-query bot

Executable ground truth changes everything. Execution accuracy, why schema linking causes most failures, abstention as a metric, and read-only rails that beat any score.

Archetype 2 📊 Builder 7 · hands-on

Gate the analytics bot

A pipeline of gates: does the chart render, does it encode the right thing, does the takeaway follow from the numbers - and does the same question give the same answer twice.

Archetype 3 🧩 Builder 8 · hands-on

Agent-specific evals

When the bot takes steps, the answer is not the only thing to grade. Trajectory matching, tool-call accuracy, and pass^k - the metric that says capability is not reliability.

Grading the path 🚀 Builder 9 · hardest

The launch gate

Everything converges. Run all three bots through one live scorecard, watch confidence climb 76.4% to 92.5%, and find out which levers fixed the bot and which only fixed the measurement.

★ The live gate 🔧 Builder 10 · hardest

Eval as CI/CD and in production

Blocking regression gates, a safety canary that must pass at 100%, online evals on sampled traffic, drift detection, and the refresh cadence that keeps the gate meaningful.

Keeping it true
easy - everyone medium hands-on / deciding hardest

Choose your path 🗺️

Four doors into the same conviction: the number you may promise is the lower bound, not the one you observed.

🎯 I sign off on launches a1→ a2→ a3→ a4→ a5→ a6
🛠️ I build the bots b1→ b2→ b3→ b4→ b5→ b6→ b7→ b8→ b9→ b10
⚡ I ship on Friday and need the gate now a4→ a5→ b9
🚀 Full journey (build AND decide) a1-a6→ b1-b10

The knowledge map 🧠

Both tracks at a glance - hover a session to spotlight its concepts, click any node to jump in.