What the bench measures
Coverage is the number everybody reports and the least useful of the three. A set that catches every incident while paging somebody four times a week has not bought detection, it has bought a muted channel. So the bench prints caught before the stakeholder, false pages and mean hours to detect side by side, and the ladder below is worth walking in order because the interesting rungs are the ones where two numbers move in opposite directions.
★ What you walk out with tonight. A monitor set you have actually scored rather than assumed. The measured difference between a naive threshold and a learned one, which turns out to be eight false pages for the same two catches. And the number that ends the argument about tightening thresholds after an incident: seven monitors at hair-trigger settings catch 2 of 5 and detect no faster than having almost nothing on.
LiveWhy three numbers and not a score3 min▶
Any single score hides the trade. Coverage alone rewards firing on everything. Precision alone rewards firing on nothing. A weighted blend of the two rewards whoever picked the weights, which is usually the person presenting the slide.
Three numbers force the conversation to be about the exchange rate: what did this monitor cost in pages, and what did it buy in hours. That is a decision a team can actually make, and it is the same shape as an error budget in service reliability, which is where the idea comes from.
Bellwether ran a review after the finance incident and added six monitors in an afternoon, all at aggressive thresholds. Detection for the next real problem took 26 hours, exactly as long as before, because the on-call had stopped opening the channel by day nine. The bench reproduces this: it is the last row of the table below.
Self-studyWhat is real here and what is modelled3 min▶
Worth being precise about, because a widget that quietly scripts its own conclusion teaches nothing.
- Real. The 30 nights of metadata, generated once from a fixed seed and identical for every learner. The five defects, injected into specific nights. Every monitor's arithmetic: trailing percentiles, same-weekday medians, duplicate rates, column-set diffs, standard deviations. The counts, the false pages and the detection times all fall out of that arithmetic.
- Modelled. The human. Six or more false pages in a trailing week is treated as a muted channel: the page still fires, nobody reads it, and the 09:00 stakeholder gets there first. That threshold is a teaching choice, not a measurement, and the widget says so on its face.
- Deliberately absent. Any notion of severity weighting. All five defects count the same here, which is not true in a real estate and is the first thing you would add.
The ladder, with the numbers it actually produces
10 min liveEvery row below is computed by the bench on this page, not asserted. Walk it in order and watch rungs three and four, where the interesting thing happens: swapping two naive monitors for two learned ones keeps the catches and removes every false page.
| Monitors on | Caught | False pages | Mean hours to detect |
|---|---|---|---|
| Nothing | 0 of 5 | 0 | 26.0 |
| Freshness, fixed deadline | 1 of 5 | 0 | 20.9 |
| + volume, fixed band | 2 of 5 | 8 | 15.8 |
| Learned freshness + same-weekday volume | 2 of 5 | 0 | 15.8 |
| + uniqueness | 3 of 5 | 0 | 10.7 |
| + schema diff | 4 of 5 | 1 | 5.6 |
| + null rate, all five signals | 5 of 5 | 2 | 0.5 |
| All seven monitors, every threshold tightened | 2 of 5 | 25 | 15.8 |
Rungs three and four are the whole course in two rows. Same two catches, same detection time, eight false pages against zero. The difference is not effort or tooling, it is that one pair of thresholds was set from a guess and the other from the table's own history.
Run the ladder yourself
★ 25 min · everyone buildsThe bench is live below. Every number updates as you toggle, and hovering a night says what fired and why. Work the steps in order; the last two are the ones worth arguing about.
Start from nothing. Read the strip: 30 quiet-looking nights, five of them red. Note that no monitor is watching any of them and the mean detection time is 26 hours, which is the time until somebody in finance opens a dashboard the next morning.
Press "the naive pair". Fixed deadline plus fixed volume band. Two catches and eight false pages, all eight on weekends. Hover one to confirm: the count is fine, the band is wrong.
Press "the learned pair". Same two catches. Zero false pages. This is the single highest-value swap in the whole track and it involves no new tooling, only comparing against the right history.
Add uniqueness, then schema, then null rate. Watch which night lights up at each step, and notice that night 19 is invisible to everything except the duplicate rate. Row count on night 19 is normal. A pipeline-success check, a freshness check and a volume check all pass on a night when every downstream sum is wrong.
Look at the two false pages you are left with. Night 8 is a null-rate wobble inside a genuinely tiny standard deviation. Night 24 is the schema monitor firing on the revert, which is a real change and arguably a real page. Decide which of the two you would tune away and what that costs you.
Now tick "hair-trigger every threshold". All seven monitors, every threshold aggressive. Coverage in the sense people usually mean it is perfect: every defect fires a page. Detection collapses to 15.8 hours and 25 nights out of 30 carry a false page. Three of the five incidents fire into a channel that the model treats as muted.
Write down the set you would actually run on Monday and the one number you would report to a stakeholder. If you pick coverage, the last rung is your set and it is worse than five monitors.
The honest caveat, stated plainly. The muting rule is a model, not a measurement. What is not a model is the arithmetic that produces 25 false pages from those thresholds against that history: you can hover every night and check it. If you think a real team tolerates more noise than this bench assumes, raise the number and see what happens to the ladder. The direction does not change.
Closing the track
40 min- Score your own set the way the bench does. Pull your metrics table, list every incident of the last quarter that anybody remembers, and mark which of them your monitors found first. Most teams discover the number is lower than they would have guessed.
- Count your false pages for the same period. If the channel history is unreadable because nobody triaged it, that is your finding.
- Delete or retune every monitor that fired more than four times without a real cause. Deleting is allowed and is usually right.
- Write the one-page version for whoever funds this: three numbers, before and after, plus the tier list from leader session 5. Not a tool comparison.
Three questions
Two rungs of the ladder both catch 2 of 5 with the same detection time. What separates them?
Identical catches, identical hours, eight false pages against zero. The thresholds were set from a guess rather than from the table's own history, and the whole difference is the noise.
Night 19 is a silent dedup failure: the duplicate rate goes from 1.0 to 8.4 percent while the row count stays normal. Which checks pass on that night?
A merge key change can duplicate some rows and drop others, landing within a percent of normal. This is the failure class that makes uniqueness worth its own monitor.
Tightening all seven thresholds makes every defect fire a page, yet mean detection time gets worse than the five-monitor set by a factor of 30. Why?
A page that nobody reads is not a detection. The bench models a muted channel above six false pages in a trailing week, which is the documented failure mode of noisy alerting rather than anything exotic.