Two moments, two very different jobs
Think of evaluation as two safety systems on a building. One is the inspection before anyone moves in - the gate that decides whether the building is fit to open. The other is the smoke detector that runs every hour after, catching the fire that starts long after the inspector left. Both are evaluation. Skip the inspection and you open something unsafe. Skip the smoke detector and you never find out when it goes wrong in use. AI systems need both, and most teams have only installed the inspection.
Two kinds of eval 9 min live
Evaluation happens at two moments in a system's life. Offline eval runs before launch and again on every change, against a fixed set of questions with reference answers. Online eval runs continuously after launch, on real production traffic, with no reference answer to compare against. Same discipline, two entirely different setups.
LiveOffline eval - the launch gate you build once and re-run forever3 min▶
Offline eval runs before you ship. You assemble a fixed set of representative questions, each paired with a known-good reference answer, and you grade the system against that answer key. The score is the gate: high enough, you launch; not high enough, you do not.
- The answer key is the whole point. Because you know the right answer for each question, grading is clean and repeatable. The same test run today and next month is directly comparable.
- It re-runs on every change. Swap a model, tweak a prompt, change how documents are fetched - re-run the exact same set. If the score drops, the change broke something. That is a regression, caught before a user ever sees it.
- Its blind spot: it only ever asks the questions you thought to put in the set. It cannot see the questions real users will actually ask tomorrow.
LiveOnline eval - the smoke detector on live traffic3 min▶
Online eval runs after you ship, continuously, on the real questions real users are asking right now. The defining difference: there is no answer key. Nobody wrote a reference answer for a question that was asked thirty seconds ago, so online eval grades differently - it checks the answer against the material it used, watches for signals like low groundedness, and folds in what users tell you.
- No reference answers, ever. Live questions arrive without a known-good answer. So online grading leans on reference-free checks and on user feedback rather than an answer key.
- You sample, you do not grade everything. Grading every single production answer costs real money, so teams grade a slice - a sampling rate. A rate of 0.1 means one in ten answers gets evaluated: enough to spot a trend, cheap enough to run always.
- You can aim the sampling. Teams often focus grading on the answers users flagged as bad - a thumbs-down - so the budget goes where the trouble is.
LiveSide by side - the same discipline, two setups3 min▶
The cleanest way to hold the difference is a table. Read it across each row and the split is obvious.
| Offline eval | Online eval | |
|---|---|---|
| When | Before ship, and on every change | Continuously, after ship |
| On what data | Fixed set of chosen questions | Real, live production traffic |
| How graded | Against known reference answers | No reference - reference-free checks + user feedback |
| Catches what | Regressions between versions | Live-quality issues and drift |
Why you need both 8 min live
Offline is the gate before launch; online is the smoke detector after. Neither does the other's job. Offline cannot see the real world coming, and online cannot gate a launch that has not happened. A team with only one has a blind spot exactly where the other would have been looking.
LiveWhy offline alone leaves you blind3 min▶
Offline eval is frozen in time. Your test set was written on a particular day, reflecting the questions and the world of that day. But the world does not hold still after you ship, and offline eval never notices.
- The documents drift. A policy changes, a price updates, a product is discontinued - and the system keeps confidently quoting the old material. Your offline set, with its old answers, still passes.
- Users ask new things. Real traffic is full of questions nobody put in the test set. Offline eval literally cannot grade a question it does not contain.
- The model changes under you. When a provider updates the underlying model, behavior can shift without a single line of your own changing. Offline catches this only if you happen to re-run - online catches it live.
LiveWhy online alone cannot gate a launch3 min▶
The reverse is just as true. Online eval is powerful, but it only exists once you are live - which is too late to decide whether you should have gone live.
- No answer key yet. Before launch there is no production traffic and no way to know the right answers at scale. Online eval has nothing to grade until real users arrive.
- You cannot regression-test a change safely. Deciding whether a new prompt is better belongs in a controlled offline run against known answers - not by shipping it to customers and watching complaints.
- Feedback is a signal, not a gate. User thumbs up and down are gold for online monitoring, but you cannot ask users to grade a product that has not shipped.
The bot that aged badly. A team (anonymized) shipped a support bot that passed its offline eval cleanly - every reference answer matched. Three months later a refund policy changed. The offline set still had the old questions and old answers, so it kept passing. Meanwhile the live bot confidently quoted the retired rule to real customers for weeks. The only thing that could have caught it was online eval watching live traffic - and the team had not installed one. The first signal was a spike in complaints.
LiveUser feedback - your cheapest online signal3 min▶
The simplest online eval is one you already have the ingredients for: the thumbs up and thumbs down next to an answer. Every user who reacts is grading your system for free, in the real world, on a real question.
- It becomes a number. A thumbs up counts as 1, a thumbs down as 0. Averaged over traffic, that is a live quality signal you can chart and watch for dips - and it can be attached to any step of the process, not just the final answer.
- It aims the expensive grading. Rather than sampling answers at random, focus the deeper evaluation on the ones users flagged as bad. The complaints tell you where to look.
Offline catches this, online catches that ★ 12 min · pen and paper
Pick one AI use case you own or care about. For that case, sort what each kind of eval would catch - and, crucially, what only online would ever see.
Name the use case in one line, and the fixed set of questions you would put in its offline test - the handful you are sure it must get right.
List two things offline eval would catch for this case - for example, a prompt change that quietly made answers worse, or a model swap that dropped quality.
List two things only online eval would catch - for example, a source document changing after launch, or users asking a whole category of question you never anticipated.
Ask the hard one: if the most likely failure for this case is drift after launch, and you only have offline eval, how long would you stay blind? That gap is the case for building online.
Design your online signal ★ 8 min · your next AI review
Online eval only works if you decide, in advance, what to watch and how much of it. Sketch the online signal for your use case in four decisions.
What would you sample? You will not grade every answer - it costs too much. Decide the slice: a random one in ten, or every answer on a high-stakes topic? Name your sampling rate in plain terms.
What feedback would you capture? Thumbs up/down is the minimum. Would you also capture a reason, a follow-up "did this solve it?", or an escalation-to-human signal? Each is a live grade.
What would trigger a deeper look? Decide the rule: "grade every thumbs-down answer", or "when the weekly thumbs-up average drops below X, investigate". Aim the expensive grading at the trouble.
Who watches the trend, and how often? A signal nobody looks at is not a smoke detector, it is a disconnected one. Name the owner and the cadence.
Before session a4 ◐ 30 min total
- Finish the offline-vs-online sort for your top use case and keep it - a4 folds both into a single ship scorecard.
- For a real AI product you use, hunt for the thumbs up/down - and ask whether anyone charts the trend, or just reads individual complaints.
- Write down your best guess at how long your team would stay blind to a post-launch quality drop today. Bring the number to a review.
- Optional: skim the builder track's b3 to see a sampling rate and a feedback score in action - you do not need the code, just the shape.
- Do we have an offline eval - a fixed set of questions with known answers we re-run before every change?
- Do we have any online eval on live traffic, and if so, what sampling rate do we grade at?
- If a source document changed after launch and the bot kept quoting the old one, how would we find out - and how fast?
- Do we capture user feedback like thumbs up/down, and does anyone watch the trend rather than just individual complaints?
- How long, honestly, would we stay blind to a quality drop in production - and what would shorten it?
Official sources covered
The leader track teaches from published engineering guidance and course curricula - no vendor marketing. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · The defining difference between offline and online evaluation is...
Offline runs before ship against a fixed dataset with reference answers and catches regressions. Online runs continuously on live traffic where no reference answer exists, using reference-free checks and user feedback.
2 · A support bot passes its offline eval every week but starts giving wrong answers in production after a policy changes. What happened?
Offline eval is frozen to the day it was written. When documents change under a live system, the old test set keeps passing. Only online eval, watching real traffic, sees the drift.
3 · A sampling rate of 0.1 on online eval means...
Grading every live answer is expensive, so teams grade a slice. A rate of 0.1 grades one in ten - the sampling rate is the budget dial that lets you watch quality every day. Grading can be aimed at feedback-flagged runs.