From firefighting to a trust score

An observability practice has an unusual failure mode: it succeeds invisibly. When it works, incidents get caught early, fixed quietly and never reach anybody who signs a budget. The absence of pain reads as the absence of need, and the line item looks like a candidate for the next round of cuts. The only defence is a number that was being reported before anybody asked for it.

Monthly is the right cadence for three reasons. It is frequent enough that a bad month cannot be hidden, slow enough that the numbers are not dominated by noise, and it matches the rhythm at which most organisations reallocate money. Weekly reporting on these three numbers produces meaningless swings, because a month with one incident and a month with four are both normal.

• Incident - an occasion on which a table was wrong, late or missing while people were using it. Counted whoever found it, which is the point.
• Caught before a stakeholder - the share of incidents where a monitor was first and a human outside the team was second. The numerator is monitors, the denominator is all incidents including the ones you missed entirely.
• Mean hours to detect - from the moment the table went wrong to the moment a human knew, averaged across every incident in the period. Incidents nobody caught carry the full time until a stakeholder noticed.
• False page per week - an alert on a period when nothing was actually wrong, expressed as a weekly rate so it can be compared against what a rota tolerates.
• Baseline - the first honest measurement of all three, taken before the work, with its guesses labelled as guesses.
• Denominator - the word to listen for. Almost every dispute about a data quality metric is a dispute about the denominator, and a single score lets whoever reports it pick.

A single trust score invites the team that reports it to define its own denominator, and the definition drifts in exactly one direction. Divide passing checks by all checks and you improve the score by adding easy checks. Divide clean tables by monitored tables and you improve it by monitoring fewer, cleaner tables. Weight the dimensions and you improve it by reweighting. None of that is dishonesty; it is what happens to any composite metric that one team both defines and is measured on.

The three numbers resist this because their denominators are outside the team's control. All incidents, including the ones nobody caught, is a denominator that a stakeholder can contest and often will. Hours are hours. Pages are counted by the people receiving them. And critically, the three cannot all be improved at once by any single lazy action: adding monitors moves the first up and the third the wrong way, tightening thresholds moves the third up and, past a point, the first two backwards.

• Report all three or none. A month where catches rose and pages doubled is a month you need to talk about, and reporting only the catch rate removes the conversation.
• Keep the definitions written down and dated. Any change to a definition gets reported alongside the number that month, because a metric that quietly changes shape is worse than no metric.
• Show the count, not just the rate. Three of four incidents caught and 30 of 40 are the same percentage and completely different months.

A reported number is an instruction to the team, whatever the accompanying words say. So it is worth being deliberate about the instruction each of these three sends.

• Caught before a stakeholder sends: find it first. It rewards putting monitors where damage is rather than where monitoring is easy, and it cannot be improved by adding monitors to tables that never break, because those tables contribute nothing to the numerator or the denominator. Its failure mode is that a team stops recording incidents it did not catch, which is why the denominator has to be built from the business's complaints rather than from the alert log.
• Mean hours to detect sends: shorten the incident. This is the number a stakeholder feels directly, because it is the length of time they were making decisions on wrong data. It also correctly punishes a monitor that fires into a channel nobody reads, since a page nobody opened is not a detection and the hours keep accumulating.
• False pages per week sends: deleting a monitor is progress. This is the only one of the three that gets better when you remove work, and having it on the page is what makes retuning and deletion a reportable achievement rather than an admission. Without it, every incident review adds monitors and none ever removes them.

This is the case worth memorising, because it is the argument you will have after your first serious incident. The instinct after being caught out is to tighten every threshold, and the instinct is wrong in a way that only a three-number report can show.

Bellwether's 30 nights of load metadata contain five real defects, one visible to each signal. Two monitor sets were scored against exactly that history:

In the second set every one of the five defects fired a page. Every single one. A report whose only number is coverage in the sense people usually mean it - did an alert exist for this failure - would have called that month a complete success, and it would have been describing a month in which three of the five incidents were found by a stakeholder anyway, because 25 false pages in 30 nights had turned the channel into wallpaper. Detection went from half an hour to nearly sixteen.

Notice what the middle column is doing. It is not a hygiene metric or a nice-to-have; it is the mechanism that explains why the first column collapsed. That is the argument for carrying all three numbers even when only one of them gets asked for.

The hard part of the first number is not the monitors, it is knowing how many incidents there were in total, including the ones no monitor saw. A team that builds this denominator from its own alert log will report a catch rate near 100 percent while stakeholders quietly lose faith, and the gap between those two things is the whole reason the practice exists.

• Take the denominator from the business. Every complaint, ticket, "is this number right" message and hallway question about a table counts as a candidate incident until somebody checks it and rules it out. Rule-outs are cheap; missing incidents are not.
• Keep one list, in one place, with one owner. Two lists diverge within a quarter and then the monthly report becomes an argument about which list is right. The list is more valuable than any tooling you will buy this year.
• Record who found it and when it started. Two fields, and they are the two that produce both the catch rate and the hours. Everything else on the record is useful for the retrospective and not for the report.
• Count the near misses separately. An alert that fired and was correct but where nothing downstream was affected yet is a good outcome and a bad incident record. Keeping them apart stops the catch rate from being flattered by easy wins.

Never promise fewer incidents. It is the promise everybody wants to hear, it is the one that gets the budget approved, and it will be held against you in the third quarter for a reason that has nothing to do with how good your monitoring is.

Incident count is mostly upstream of you. A vendor changes an API, a source system ships a schema migration on a Thursday, a merge key changes because a business process changed, a region gets added. Monitoring does not prevent any of that. What it changes is how long the wrongness lasts and who finds out first, and those are exactly the two things the three numbers measure.

There is a second-order problem too. A team promising fewer incidents is incentivised to record fewer incidents, and the recording is the fragile part of the whole practice. Promise detection and the incentive points at finding more of them, which is what you want.

Monitors accumulate and nothing removes them unless removal is somebody's job. Every incident review adds one, every new table brings its own, and after a year the set contains monitors whose original author has left and whose threshold nobody can explain. That set is not more protective than a smaller one; it is less, because the noise it makes is subtracted from the attention available for real alerts.

• Review the alert log monthly, not the monitor list. Sort by how often each monitor fired and how often that firing was real. The bottom of that list is your deletion candidates, and it is usually obvious.
• Anything that fired repeatedly without a real cause gets retuned once, then deleted. Not muted. A muted monitor is worse than a deleted one because it looks like coverage in every report and provides none.
• Delete tables too. The heartbeat from the tier 3 layer will show tables that stopped loading months ago and nobody noticed. Removing them shrinks the estate you are responsible for, which is the only permanent improvement available.
• Report the deletions. "Eleven monitors, down from nineteen, with a lower page rate and a higher catch rate" is a sentence that teaches your stakeholders how this practice actually works.

There is a category of commercial data observability platform, and the honest position is that the buy-or-build question is decided by your estate and your team rather than by any product's feature list. What follows is the set of questions worth asking, in the order that matters. A scorecard comparing vendors would be out of date before you finished reading it; these questions will not be.

• How many tables need more than a heartbeat? Below a few dozen, scheduled queries plus your existing alerting will hold, and the effort goes into thresholds rather than tooling. In the hundreds, the maintenance of hand-written monitors becomes the dominant cost and buying starts to make sense.
• Where does your lineage come from, and does the platform read it or invent it? Automatic lineage is the feature with the widest gap between demo and estate. Ask specifically about the paths that matter: your reverse-ETL jobs, your BI layer, your exports. A tool that stops at the warehouse boundary has the same blind spot you already have.
• How does it set thresholds, and can you see the arithmetic? Learned thresholds are the main thing you are buying. If you cannot inspect why a monitor fired or explain it to an on-call engineer at 07:00, the alerts will be muted regardless of how good the model is.
• What does it cost per table per month, and does that scale with your long tail? Pricing shaped around monitored tables punishes exactly the tier 3 coverage that is cheap to do yourself. Model it against your real table count, not the tier 1 list.
• Who carries the pager, and does the tool fit their existing rota? If the answer requires a new channel that only the data team reads, the tool has not solved the problem the practice is about.
• Can you leave? If the metrics history lives only in the vendor's store, your baseline and everything you have reported for two years leaves with them. Keeping your own metrics table alongside any platform is cheap insurance and takes an afternoon.

One structural point worth making regardless of the answer: build the metrics table and the incident list yourself either way. They are the two assets that make a platform evaluation possible, they are the two that survive a change of vendor, and neither costs more than a week.