What month two looks like
A monitoring programme has two failure modes and only one of them is visible. The visible one is not enough coverage: something broke and nothing was watching. The invisible one is too much noise: everything was watching, everything fired, and the channel became wallpaper. The second failure looks identical to success on a status report, which is why it is the one that survives to month six. Tonight puts numbers against both, taken from the same 30 nights of Bellwether history that the builder track scores its monitor set against.
★ What you walk out with tonight. The three numbers to ask for whenever somebody reports on monitoring, and the reason one of them alone is worthless. A measured pair of monitor sets with identical catches and identical detection time, where one spent eight false pages and the other spent none. And the number that settles the argument after an incident: seven monitors at tightened thresholds catch 2 of 5 and detect no faster than having almost nothing on.
LiveWhy the second month is the one that decides it3 min▶
In the first weeks, alerts are novel and somebody investigates every one. That is what makes the early data misleading: the response rate is high because attention is abundant, not because the monitors are good. By week five the same person has been woken for a weekend volume dip four times, has learned that this particular alert is never real, and has started to skim. Skimming is not laziness, it is a rational response to a channel with a low hit rate, and it is where detection quietly goes to zero while the dashboard still says every monitor is green and firing.
The consequence for anybody funding this is uncomfortable. The monitors that get switched off are rarely switched off; they get ignored, which is worse, because the line item stays in the budget and the reporting still counts them as coverage. A programme that would have been improved by deleting three monitors instead reports seven.
Bellwether ran a review after the finance incident and added six monitors in an afternoon, all at aggressive thresholds. Detection for the next real problem took 26 hours, exactly as long as before the review, because the person on call had stopped opening the channel by day nine. Nothing was misconfigured. Every monitor fired correctly. The channel had simply stopped being a place where anybody looked.
Self-studyThe four terms this session runs on3 min▶
- False page - an alert on a night when nothing was actually wrong. Not a bug. It is the currency you spend to buy detection, and the whole session is about the exchange rate.
- Caught before a stakeholder - an incident where a monitor told somebody on the team before somebody outside the team noticed. The only version of coverage that means anything, because it is the outcome you were buying.
- Mean hours to detect - average hours from the moment a table went wrong to the moment a human knew. It is the number that turns into money, since it multiplies by the number of decisions made on bad data in the interval.
- A muted channel - a destination that still receives every alert and that nobody reads. Indistinguishable from having no monitors, except that it costs money and appears on the report as coverage.
The second month is where a monitoring programme dies
10 min liveEvery monitor buys detection with false pages. There is no threshold setting that avoids the trade, only settings that get a better rate for it, and the difference between a good rate and a poor one is usually not tooling. The chart below is one measured pair from Bellwether's history: two monitor sets, the same two incidents caught, the same detection time, and eight false pages against zero.
LiveThe exchange rate, and why it is never zero3 min▶
A monitor sets a line and calls anything past it wrong. Real data wanders for reasons that are not incidents: a bank holiday, a marketing push, a genuinely slow but successful load. Move the line closer to normal and you catch more real problems and more of the wandering. Move it further out and you catch fewer of both. There is no line that catches every incident and nothing else, so the only question worth asking about a threshold is what rate it is getting you.
The good news in that chart is where the improvement comes from. Both sets watch the same two signals on the same table with the same query cost. One compares against a number somebody picked in a meeting, the other against the table's own recent history. That single change removes every false page while keeping both catches, and it needs no product, no headcount and no procurement cycle. When somebody proposes buying a tool to reduce alert noise, this is the question to ask first: are the current thresholds set from history or from a guess?
LiveAlert fatigue is a documented failure mode4 min▶
This is not a data problem, and that is what makes it credible. Software operations has written about noisy paging for decades, and the fix in every published account is the same shape: fewer alerts, each one meaning something specific and actionable, routed to somebody who has agreed to answer it. Clinical alarm research reaches the same conclusion from a different direction, with monitoring equipment that alarms so often that staff response degrades. Different domains, same result, same remedy. The recommendation is never "add more alarms".
Where honesty is required is the number. There is no clean published threshold for how many false pages a specific team tolerates before it tips over, and anybody quoting one as a law is overstating it. It depends on how the page arrives, what hour it arrives at, how long triage takes and how much the reader trusts the sender. The direction, though, is not in doubt in any of that literature: response degrades as the proportion of false alerts rises, and it does not recover on its own.
What to do with a well-evidenced direction and no exact number. Do not invent a threshold and present it as research. Measure your own instead. Count your false pages per week now, agree with the on-call rota what number would be intolerable, and treat that agreed number as the budget. It is a local number, it is defensible because it is yours, and it does what a published constant would have done anyway.
Self-studyA muted channel is worse than no monitors4 min▶
Consider two teams. The first has no monitoring on a table at all. The second has seven monitors on it firing into a channel that nobody has opened in three weeks. Both find out about incidents when a stakeholder complains, so their detection times are identical. On every practical measure they are the same team, with three differences, all in favour of the first:
- The second team is paying. Compute, licence, and the hours that went into building and tuning seven monitors are all spent and still being spent.
- The second team believes it is covered. Nobody proposes fixing a problem they think is solved, so the muted channel is self-sustaining in a way that having nothing is not. An honest gap gets funded. An invisible one does not.
- The second team's next incident is harder to explain. "We were not watching that table" is a resourcing conversation. "It alerted correctly and nobody read it" is a trust conversation, and the trust conversation is much more expensive.
Which is why deleting a monitor counts as progress rather than retreat. Every noisy monitor you remove raises the hit rate of the channel and therefore the response rate of everything left in it. The instinct to keep a monitor because it might catch something one day ignores what it costs the other monitors while it waits.
Reading a coverage number that was bought with noise
8 min liveOne number cannot describe a monitoring set, because any single number can be maximised by doing something stupid. Coverage alone rewards firing on everything. Precision alone rewards firing on nothing. So the habit worth building is to ask for three together, and to refuse the summary that offers one. The table below is computed against 30 nights of Bellwether load metadata with five real defects in it, and every row of it is arithmetic rather than assertion.
| Monitors on | Caught | False pages | Mean hours to detect |
|---|---|---|---|
| Nothing | 0 of 5 | 0 | 26.0 |
| Freshness, fixed deadline | 1 of 5 | 0 | 20.9 |
| + volume, fixed band | 2 of 5 | 8 | 15.8 |
| Learned freshness + same-weekday volume | 2 of 5 | 0 | 15.8 |
| + uniqueness | 3 of 5 | 0 | 10.7 |
| + schema diff | 4 of 5 | 1 | 5.6 |
| + null rate, all five signals | 5 of 5 | 2 | 0.5 |
| All seven monitors, every threshold tightened | 2 of 5 | 25 | 15.8 |
Read the last two rows together, because that pair is the session. Five signals at thresholds learned from history: 5 of 5 caught, 2 false pages, 0.5 hours to detect. The same seven monitors with every threshold tightened: 2 of 5 caught, 25 false pages, 15.8 hours. More monitoring, more alerts, more diligence, and a worse result on the two numbers anybody actually cares about.
LiveThe three-number habit3 min▶
Three numbers, reported together, every time. They are cheap to produce once the alert channel has been triaged for a quarter, and they are the only shape of report that cannot be gamed by doing something silly to one of them.
- Incidents caught before a stakeholder noticed, as a count out of the total number of incidents. Out of the total, not as a percentage of monitors, and "before a stakeholder" is the load-bearing part of the phrase.
- False pages per week. A rate, not a total, so it stays comparable as the estate grows. This is the number a report will not volunteer, and it is the one to ask for first.
- Mean hours to detect. Hours from the data going wrong to a human knowing. It is the number that converts into business consequence, because the interval is exactly how long decisions were being made on bad data.
LiveThe after-an-incident reflex, and what it buys3 min▶
The reflex is universal and well-intentioned. Something got missed, so every threshold gets tightened and a few new monitors go on, usually in the same afternoon as the review. It feels like the responsible response, and it reliably makes the system worse, because it spends the whole noise budget in one go and buys detection that lands in a channel that stops being read a week later.
The measured version is in the table: all seven monitors with every threshold tightened produce 25 false pages across 30 nights while catching 2 of 5. Every defect does fire a page, so the version of coverage most people mean is perfect. What collapses is the part that was never on the report, and detection ends up at 15.8 hours, which is the same as running two learned monitors and materially worse than running five.
The better response to a missed incident is narrower and slower. Bellwether's night 19 was a duplicate failure with a normal row count, invisible to every monitor that was running. The fix was one new monitor on duplicate rate at a threshold read from the table's own history, and no change at all to the six existing thresholds. That single addition moved catches from 2 of 5 to 3 of 5 and detection from 15.8 hours to 10.7, and it cost zero false pages.
Self-studyDeleting a monitor is progress, and how to say so4 min▶
Removal is the hardest thing to get agreed, because a deleted monitor sounds like reduced coverage and nobody wants to be the person who removed the check on the night before the incident. Three framings make it survivable in a governance meeting, and all three rest on the same fact: attention is the scarce resource, not compute.
- A monitor that has never been right has no track record to defend. "This has fired 34 times in a quarter and been a real incident zero times" is not a controversial statement, it is a measurement. Bring the count.
- Deletion is not the only option, and retuning is the usual answer. Most noisy monitors are noisy because the threshold came from a guess rather than from history, which means the same monitor at a learned threshold keeps the catches and drops the noise. Offer both and let the room pick.
- Name what the noise is costing the rest. A monitor firing four times a week is not costing you compute, it is costing the response rate of every other monitor in the same channel. That is the sentence that changes the decision, because it reframes deletion as protecting coverage rather than reducing it.
One useful practice to adopt alongside it: a review of the alert channel on a fixed cadence, quarterly is enough, with the specific brief of removing or retuning the noisiest monitor. Not adding. A programme with no removal mechanism only ever accumulates, and accumulation is precisely how a channel becomes wallpaper.
Audit your channel
★ 18 min · everyone buildsThis is an afternoon of work at most, and it produces the baseline you will be reporting against for the rest of the year. It needs no tooling and no access you do not already have: the alert channel, whoever is on the rota, and a willingness to count things that nobody has counted before.
Pull the last 90 days of the alert channel. Whatever it is: a chat channel, a mailbox, a paging tool's history. Ninety days rather than a week, because a week is short enough to be an unusually quiet or unusually awful one and you will be quoting this number for months.
Count three things: alerts, real incidents, and alerts nobody replied to. The third count is the one people skip and it is the most informative, because a long run of unanswered alerts is the signature of a muted channel with a date on it. Note the date where the replies stop; that is the day your detection quietly reverted to stakeholder discovery.
Compute false pages per week. Total alerts minus real incidents, divided by roughly thirteen weeks. Say the number out loud in the room. If it is above one a week, you have found the constraint on your entire programme, and no additional monitor will improve detection until it comes down.
Find the noisiest single monitor and decide, tonight, retune or delete. In most estates one or two monitors produce the majority of the noise, and they are usually fixed-band checks on something seasonal. Retune means setting the threshold from that table's own trailing history. Delete means delete. Both are decisions; leaving it to be looked at later is not.
Write the three numbers down as your baseline, with today's date. Caught before a stakeholder, false pages per week, mean hours to detect. Approximate is fine and honestly labelled approximate is better. Without a dated baseline, every improvement you make for the next year is unprovable, and unprovable improvements do not get funded twice.
Expect the first audit to be embarrassing, and say so before you start. Almost every team discovers that the caught-before-a-stakeholder count is lower than anyone would have guessed and that a large share of alerts were never triaged. That is the finding, not a failure of the audit, and a leader who frames it as a baseline rather than a verdict gets a team that helps count. A leader who frames it as an assessment gets a channel that mysteriously looks tidy by Friday.
Before session 5
25 min- Agree a false-page budget with whoever carries the rota. One number, per week, that you both accept as the ceiling. It is a local number rather than a published one, and that is exactly what makes it usable.
- Retune or delete the noisiest monitor you found, and record the before and after count. This is the smallest possible piece of evidence that the practice works, and it is worth having before session 6 asks you to report on progress.
- List every table currently carrying a monitor, and mark which ones a decision actually depends on. Session 5 is about coverage without headcount, and it starts from the observation that most estates are monitoring the wrong twenty percent.
- Ask the person on call one question and write down the answer verbatim: what do you do when a page arrives at 07:00? If the answer describes checking whether it is real before acting, your channel already has a trust problem and now you have it in their words.
Three questions
Two monitor sets both catch 2 of 5 incidents with a mean detection time of 15.8 hours. What separates them?
Identical catches, identical hours, 8 false pages against 0. The whole difference is that one pair of thresholds came from a guess and the other from the table's own trailing history, which costs nothing to change.
All seven monitors are tightened after an incident. Every defect now fires a page, yet catches fall to 2 of 5 and detection goes to 15.8 hours. Why?
A page nobody reads is not a detection. The exchange rate went bad enough that the channel stopped being a place anybody looked, which is the documented failure mode of noisy alerting rather than anything exotic.
Team A has no monitors on a table. Team B has seven firing into a channel nobody has opened for three weeks. How do they compare?
Both find out when a stakeholder complains. An honest gap gets funded; an invisible one does not, and "it alerted and nobody read it" is a much more expensive conversation than "we were not watching that table".