The archetype that inherits both problems
Beacon's Insight Bot takes a question from an executive, writes a query, computes a figure, draws a chart, and writes a paragraph explaining what it means. Sessions b5 and b6 gated the two halves of that separately. This session gates them together, which is a harder problem than the sum of the two - because a single end-to-end pass rate tells you the pipeline broke without telling you where.
Four stages, four graders, and multiplication 6 min live
The instinct is to score the finished answer, because that is what the user sees. The instinct is wrong on this archetype for one arithmetic reason: an end-to-end number is a product of four stage numbers, and a product tells you nothing about its factors.
LiveWhy one end-to-end number is the wrong instrument here4 min▶
Suppose Beacon's Insight Bot reports 81% end to end. That single figure is compatible with wildly different products:
- 95 / 95 / 95 / 95 - a balanced pipeline with no obvious weak spot, where the next improvement is expensive and spread across four teams.
- 99 / 99 / 84 / 99 - a chart-encoding problem and nothing else. One afternoon of work on the chart-spec validator, and the number moves nine points.
- 99 / 99 / 99 / 84 - the narrative stage inventing causes. A rail on the takeaway prompt fixes most of it, and no amount of query tuning touches it.
The end-to-end number is identical in all three cases. This is the whole argument for staged grading: the aggregate is the thing you report and the per-stage numbers are the thing you act on. Beacon's team ran the aggregate for two quarters and spent both of them optimising the query stage, because that was the stage they understood.
Self-studyThe one thing this archetype inherits from neither parent2 min read▶
The text-to-query bot returns rows. The chatbot returns prose. The analytics bot returns rows, a picture and prose, and the failure that matters most is a mismatch between them. The chart shows a 4% dip; the takeaway says "sharp decline driven by pricing". Both halves pass their own stage gate. The pair is a lie.
You cannot catch that by grading the chart or by grading the sentence. You catch it by grading the sentence against the numbers actually plotted, which is a groundedness question with the chart data as the source passage. That is exactly the faithfulness metric from b2, pointed at a table instead of a document, and it is the single highest-value judge call on this archetype.
Validity, then legality, then readability 7 min live
VisEval's contribution is an ordering, not a score. A chart that does not render cannot be illegal. A chart that encodes the wrong thing does not deserve a readability review. Ask the three questions in order and you stop wasting judge calls on outputs that a rule already disqualified.
Legality is the failure nobody photographs. Beacon's Insight Bot drew a legal-looking line chart of a categorical breakdown, with the axis unsorted. It rendered. It looked professional. It connected six unordered product categories with a line, implying a trend across things that have no order. The validity gate passed it, the readability judge scored it well because the labels were clean, and it went into a quarterly review. An illegal rate of 21.44% on the safest library is the published figure, and the reason a rule-based spec check belongs between the two.
Self-studyWhy the library gap is the most actionable number on this page3 min read▶
The same model, the same tasks, the same prompts: 3.29% invalid code with matplotlib, 25.41% with seaborn. Nearly an eightfold difference in the raw failure rate, produced by a choice your team makes for reasons that have nothing to do with evals - somebody preferred seaborn's defaults.
Two consequences. First, published validity figures do not transfer: a paper's number is a number about the library that paper used, and if your stack is plotly or ECharts or a Vega-Lite spec you know nothing yet. Second, and more usefully, the chart library is a lever. It sits in the same family as switching an LLM judge for a rule: a change that removes failures rather than measuring them, available for the cost of a refactor, and never on anyone's eval roadmap because it does not look like eval work.
Numbers, insights, refusals, and the same answer twice 6 min live
The chart cascade covers one stage. Three more criteria sit around it, and the fourth of them is unique to this archetype: an executive who gets two different numbers for the same question on two different days stops using the bot, whatever your accuracy figure says.
| Criterion | Grading rung | What a miss looks like in production |
|---|---|---|
| Numeric correctness | Rule: recompute deterministically, format-tolerant match | The revenue figure in the deck is off by the wrong denominator at step two |
| Chart validity | Execution: render the spec, catch the exception | A broken image in a board pack. Loud, cheap, embarrassing, harmless |
| Chart legality | Rule on the spec, before any judge sees it | A line chart through six unordered categories, implying a trend that cannot exist |
| Insight faithfulness | Judge, calibrated against human labels (b4) | "Sharp decline driven by pricing" under a chart that shows a 4% dip and says nothing about pricing |
| Refusal correctness | Rule on the expected behaviour: abstain or answer | Three data points, an arrow, and the word "trend" |
| Reproducibility | Rule: N runs, identical numbers required | Two executives quote two different figures for the same question in the same meeting |
Build the Insight Bot's staged harness ★ 12 min · everyone builds
Instrument the pipeline so every stage reports its own number. This is more plumbing than modelling, and it is the work that makes every later decision cheap.
Split the run into four recorded artifacts per case: the query, the computed figures, the chart spec, and the takeaway text. Store all four, always, including on passes. You cannot grade a stage whose output you threw away, and you will want last month's artifacts the first time a number is disputed.
Grade stages 1 and 2 with rules. Execution accuracy on the query as in b6, then recompute the figures independently in SQL or pandas and compare with format-tolerant matching, following DABstep's binary-factoid pattern so that "1,240" and "1240" are the same answer.
Grade stage 3 as a cascade, in this order: render the spec and catch exceptions (validity), assert mark and encoded fields against what the question requires (legality), and only then send survivors to a judge for readability. Measure validity in your own chart library - a published rate of 3.29% belongs to matplotlib, and seaborn's was 25.41%.
Grade stage 4 with a calibrated judge, using the plotted numbers as the source passage. The question is faithfulness, not fluency: does every claim in the takeaway trace to a figure that is actually on the chart? InsightBench's pattern of judging the generated insight against a ground-truth insight is the reference to copy.
Add the insufficient-data slice by hand: three-point series, a cohort with n=4, a question about a segment your warehouse does not track. Expected behaviour is a refusal with a stated reason. Score abstain against fabricate, and treat a fabricated trend exactly as you would treat a hallucinated policy in b5.
Finally, run twenty cases twenty times each and diff the numbers across runs. Prose is allowed to vary. Numbers are not. Any case where the figures move is a case where computation is still happening inside the model, and it is a bug you can locate exactly.
Take the Insight Bot to all green ★ 10 min · everyone builds
The compound archetype on the same board. Work it to six green lights and count the levers it took, then compare that count with what the Analyst Bot needed in b6.
Switch on the quality levers one at a time and watch which category each one rescues. Output verification takes the chart category; retrieval context takes the multi-step cohort category; abstention takes the three-points-is-not-a-trend category; grounding takes the written takeaway; action rails take the instruction sitting in a spreadsheet cell.
Now the point of the exercise: this is the only bot on the board that needs all five quality levers to clear its bar. The chatbot did not. The Analyst Bot did not. That is what "the other two stacked" means as a number rather than a phrase - every failure class from both parents is present, so every defence has to be present too.
With everything on, drop the golden set to n=100. It still clears the T2 bar of 85%. Now imagine the same bot at T3, where the bar is 90%: the same evidence misses. Same bot, same run, same number - a different tier gives a different answer, which is why the tier is decided before the run and not after.
Last, switch latency and cost engineering off with every quality lever still on. Performance and cost go red together, and harder here than on either other archetype, because verification on this pipeline means running a query and rendering a chart before you answer.
Try it yourself - this week ◐ 60 min total
- Take your analytics bot's end-to-end pass rate and decompose it into per-stage rates. If you cannot, that is the finding: you are reporting a product of four numbers and acting as if it were one.
- Measure your invalid-code rate in the chart library you actually ship. The published spread runs from 3.29% on matplotlib to 25.41% on seaborn for the same model, so a number from a paper tells you nothing about your stack.
- Write a legality rule for your three most common question types: which marks are allowed, which fields must be encoded on which axis. Run it over last month's generated charts and count how many rendered perfectly while encoding the wrong thing.
- Add ten insufficient-data cases where the correct answer is a refusal. Score abstain against fabricate. A bot that puts an arrow on three points is doing the analytics version of inventing a refund policy.
- Run one real question twenty times and diff the numbers. If they move, find which stage is still computing inside the model and move that computation into SQL or pandas. Then re-run the test as proof rather than as hope.
Official sources covered
Every figure on this page traces to a primary source - peer-reviewed benchmarks and vendor documentation catalogued in this course's source map, dated 2026-08-13. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · The same model produces 3.29% invalid chart code with matplotlib and 25.41% with seaborn. What follows for your eval?
A near eightfold difference in failure rate produced by a library choice means the number is a property of the pair, not of the model. Option A is a reasonable engineering move and the library genuinely is a lever - but it is a fix, not a finding, and it does not excuse you from measuring your own stack. Option B is the trap: validity is indeed cheap to catch, which is exactly why the interesting failure is the next gate down, legality, where the chart renders perfectly and encodes the wrong thing 21.44% of the time.
2 · Your analytics bot reports 81% end to end. Why is that number the wrong instrument for deciding what to fix?
Errors compound across query, compute, chart and takeaway rather than averaging, so the aggregate is a product and a product hides its factors. A balanced pipeline and a single broken chart stage look identical from outside. The aggregate is what you report; the per-stage rates are what you act on. Beacon's team spent two quarters optimising the query stage on the strength of an aggregate, because the query stage was the one they understood.
3 · You set temperature to 0 and the bot still returns different numbers for the same question. What is going on?
Determinism is not something the sampler can give you. Floating-point addition on a GPU is not associative, so changes in batching and expert routing shift the logits run to run; OpenAI documents its seed parameter as best-effort and vLLM documents the same caveat. Option C is worth ruling out with a frozen snapshot, but it is not the cause here. The fix is architectural rather than parametric: push every quotable number into deterministic SQL or pandas, keep the model on orchestration and prose, then run each case N times and require identical numbers while letting the prose vary.