Where tonight sits
Session 1 put a price on wrongness and named the three failure shapes: late, short, and wrong-but-complete. Tonight each of those shapes gets matched to the thing that can see it, and you find out that your incident list from last week is really a shopping list. Session 3 then asks which quality dimensions are worth measuring at all, session 4 handles the noise those measurements generate, session 5 decides which tables get any of it, and session 6 turns the lot into three reportable numbers. The table is still orders_daily at Bellwether, and one of its five buried defects shows up tonight in a way that should make you uncomfortable.
★ What you walk out with tonight. Five signals, each with the failure it catches and the failure it cannot see, in language you can use with a vendor and with a board. A defensible purchase order: which two to start with because they cost nothing, which one catches the class of failure that all the cheap checks miss, and which one to leave until last no matter how good the demo was. And your own incident list marked up against all five, including the one that none of them would have caught.
LiveWhy five, and why these five3 min▶
Worth saying plainly at the start, because it changes how you should read any product page. The five-signal carve-up of freshness, volume, schema, distribution and lineage is vendor-originated. It came out of the data observability tool market rather than out of a standards body, and it has since become close to universal: analysts use it, competing vendors use it, and job descriptions use it. That makes it a genuinely useful shared vocabulary and it does not make it a standard.
The practical consequence is that nobody owes you these five specifically. There is no compliance argument for having exactly five monitors, and no auditor will ask. What holds up is the underlying logic: the first four answer "what moved", and lineage answers "why, and what else". If a vendor offers you a sixth signal, ask which of those two questions it answers better than what you already have.
Bellwether's team went into a vendor evaluation with a checklist of five pillars and came out having bought lineage first, because it was the only part of the demo that looked impressive on a projector. Nine weeks later the graph was 60 percent stale, no incident had been detected by anything, and the two checks that would have caught three of their five defects were a scheduled query away and still not written.
Self-studyThe two questions any signal is answering3 min▶
Strip away the naming and every monitor in the practice is doing one of two jobs. Keeping them separate saves you from buying the same thing twice.
- "What moved?" A measurement of the table itself, compared against its own past. Did it arrive on time, is it the usual size, does it have the usual columns, do the values look like they usually do. Cheap, mechanical, runs on a schedule, and blind to consequences.
- "Why, and what else?" A map of relationships. Where did this table come from, what reads it, who needs telling. Expensive, needs maintaining, and useless for detection on its own because a map does not notice anything.
Detection comes entirely from the first job. Response quality, blast radius and the message you send to stakeholders come from the second. A team that buys only the first has fast detection and chaotic incident handling. A team that buys only the second has beautiful diagrams of a problem that finance found on Monday.
The five signals, and what each one is blind to
12 min liveEvery signal below is defined by two things: the failure it catches, and the failure it walks straight past. The second half is the half that gets left out of product pages, and it is the half that decides how many of these you need.
LiveFreshness: did it arrive when it was promised2 min▶
What it measures. The gap between when the data was supposed to be current and when it actually was. Every table your business reads has an implied promise attached to it, usually something like "by 06:00 on a working day", and freshness is the check that the promise was kept.
The failure it catches. Late and missing arrivals. A source that stopped sending, a job that queued behind something else, a weekend backlog that never cleared. In Bellwether's history one night landed 230 minutes past the 06:00 promise when the usual worst case is around 22 minutes, and freshness is the only signal that sees it.
What it cannot see. Anything about the contents. A table can be perfectly punctual and completely wrong, and freshness will report a clean night forever. There is also a specific trap: a table rebuilt in full every night looks fresh even when the source stopped producing events a week ago, because the timestamp records the write rather than the event.
The leadership question. Not "is it fresh" but "what is the promise, who made it, and does the business know what it is?" Half of all freshness disputes turn out to be an undocumented expectation rather than a late table.
LiveVolume: is it the usual size for this kind of day2 min▶
What it measures. How many rows landed, compared against how many usually land on a day like today. The comparison is the whole signal; a row count on its own is a number with no meaning.
The failure it catches. Partial loads. A missing regional shard, a truncated file, a filter that got tighter upstream, a duplicate load that doubled everything. This is the shape behind Bellwether's finance incident: 41 percent of a normal same-weekday count, for two business days, with nothing failing.
What it cannot see. A normal-sized load of wrong rows. If something upstream replaced every value in a column but kept the row count, volume reports a textbook night. It is also the signal most likely to be switched off in its first month, because grocery orders on a Saturday are not grocery orders on a Tuesday and a threshold naive enough to ignore that fires every weekend.
LiveSchema: are the columns still the columns2 min▶
What it measures. The set of columns, their names and their types, compared against yesterday. The warehouse already keeps this list, so the check is a comparison rather than a computation.
The failure it catches. Structural change without notice. A column dropped, renamed, or retyped from integer to string by an upstream team who had no idea anybody was reading it. In Bellwether's history one night lost ship_region and gained promo_code, which broke a regional report while leaving every count and timestamp perfectly normal.
What it cannot see. Contents. The same columns can carry entirely wrong values and schema will not blink. It also has a governance quirk worth knowing before you approve an alerting policy: a schema monitor fires on the revert too, because rolling a change back is itself a change. Whether that counts as an alert is a decision somebody has to make in advance rather than at 07:00.
LiveDistribution: do the values look like themselves3 min▶
What it measures. The shape of the values rather than the values themselves. Null rate per column, how many distinct values a key has, minimum and maximum, and the share each segment usually holds. All of it computable in a single pass over the table.
The failure it catches. Silent corruption inside a healthy-looking table. This is the only family that sees the wrong-but-complete shape from session 1. Bellwether has a night where customer_id null rate goes from 0.5 percent to 11.4 percent: on time, right size, right columns, and one in nine orders no longer attributable to a customer.
What it cannot see. A shift that stays inside the normal range. If a price column that usually sits between 2 and 90 euro gets systematically inflated by 10 percent, nothing about its shape looks unusual and no metadata check will notice. That failure needs either a business rule written down as an assertion or genuine statistical anomaly detection, and both are separate disciplines with their own tooling.
Self-studyLineage: why, and what else is wrong4 min▶
What it measures. Nothing. This is the point most easily missed in a demo. Lineage is a graph of what feeds what, and a graph does not detect anything, because detection requires something to be measured over time and compared.
What it buys instead. Two things, both valuable and neither of them detection.
- Cause. When a table breaks, the graph shortens the hunt for which upstream thing changed from an afternoon of asking people to a couple of clicks. That is resolution time, not detection time.
- Blast radius. When a table breaks, the graph tells you the eleven dashboards, four models, two exports and one undocumented push into the CRM that read it. That is who you have to warn, and it is the difference between one honest message and a fortnight of trust damage.
What it costs. More than the other four combined, and the cost is ongoing rather than one-off. A dependency graph parsed from your transform code is achievable. A graph that also covers reports built by three teams in two BI tools, plus spreadsheets, plus the reverse-ETL job, is a maintenance commitment. A lineage graph that is 60 percent current is worse than none, because people trust it and then act on the missing 40 percent.
Why it still matters to you specifically. Lineage is the signal that improves your message. Session 1 argued that the real cost of an incident is the trust damage, and trust damage is mostly caused by stakeholders discovering for themselves that they were affected. The graph is how you get to them first.
Signals are not substitutes
9 min liveThe tempting simplification is that with limited budget you pick the best one or two and get most of the value. It is a reasonable instinct and the arithmetic does not support it, because each signal is the only thing that can see a whole class of failure. Here is the night that makes the case.
LiveWhy each signal earns its slot3 min▶
Bellwether has five defects buried in one month of history. What makes them a good teaching set is that each one is visible to exactly one signal, which is not a coincidence of the example so much as the reason the five-signal carve-up caught on in the first place.
| What happened | Seen only by | Everything else reported |
|---|---|---|
| Landed 230 minutes past the 06:00 promise | Freshness | A clean night |
| One of three shards missing, 41 percent of normal | Volume | A clean night |
| Merge key changed, duplicate rate 1.0 to 8.4 percent | Uniqueness | A clean night |
| A column dropped, another added | Schema diff | A clean night |
| Customer id null rate 0.5 to 11.4 percent | Null rate | A clean night |
Read the third column. Every row of it says that the other checks were not silent, they were reassuring. That is the specific danger of a partial programme: it does not merely fail to catch things, it actively produces evidence that nothing is wrong, and people quite reasonably believe it.
LiveWhat a single-signal programme misses3 min▶
Most programmes start and stop with freshness, because it is the easiest to explain and the easiest to buy. Follow that through against those five defects and the result is one caught out of five, and a dashboard that says the pipeline is healthy on the other four nights.
- Freshness only. Catches the late night. Reports clean on the short night, the duplicate night, the schema night and the null night. Your discovery gap barely moves, because the four it misses are the four that stakeholders find.
- Freshness plus volume. Catches two of five, and these two are genuinely the most common shapes in most estates, so this is a reasonable starting position rather than a token one.
- Everything except uniqueness. The trap. Feels comprehensive, misses the entire silent class where the row count stays normal, and those are the incidents with the longest detection times because nothing about them looks odd to anybody.
- Lineage only. Catches nothing, by construction. Excellent incident response, no incidents detected. Worth stating in exactly those words when a proposal arrives that leads with a graph.
Self-studyThe order to buy them in4 min▶
Cost and coverage do not line up, which is lucky, because the cheapest signals cover the most common failures. That gives a defensible sequence, and it is very close to the reverse of the order a product demo presents them in.
| Order | Signal | Why here |
|---|---|---|
| 1 | Freshness | Free, one number the warehouse already keeps, and covers the single most common complaint |
| 2 | Volume | Free, catches partial loads, and the same-day-of-week comparison is arithmetic rather than tooling |
| 3 | Uniqueness | Still cheap, and it is the only thing that sees the silent class where counts look normal |
| 4 | Schema | Cheap to compute, but needs a policy decision first about whether additive changes and reverts page anybody |
| 5 | Distribution | Cheap per column, expensive in judgement: which columns, which baseline, how much noise you will absorb |
| 6 | Lineage | Highest cost and highest ongoing maintenance, and it detects nothing. Buy it when your detection already works |
Two caveats before you use this as a plan. The order assumes you have the tables tiered, which is session 5, because running six signals across four hundred tables is how you end up with the noise problem in session 4. And steps 1 to 3 are days of work rather than a procurement cycle, which means the honest first ask is often a fortnight of somebody's time rather than a licence.
Which signals would have caught yours
★ 16 min · everyone buildsGet out the incident log from session 1. Five signals, five new columns, and by the end of this you have your own version of the table in part 2 with your own incidents in it. This is the exercise that turns "we should invest in observability" into a named, ordered, costed list.
Add five columns and tick the ones that would have caught each incident. Freshness, volume, uniqueness or schema, distribution, lineage. Be strict: the question is not whether the signal is related to the incident, it is whether a check on that signal, with a sensible threshold, would have fired on that night. If it needed an unreasonably tight threshold to fire, it does not count, and session 4 explains why.
Count the ones that needed uniqueness or schema. These are the incidents that the two cheap and popular signals cannot see. In most groups the count is between a quarter and a third of the list, which is the number that justifies steps 3 and 4 of the buy order rather than stopping at freshness and volume. If your count is zero, check whether you are missing that class of incident rather than not having it.
Mark the hours column against the ticks. Take the average hours-wrong for incidents that one of your first three signals would have caught, and compare it with the average for the rest. The second group is almost always several times worse, because a defect that no cheap check can see stays live until a human trips over it.
Find the one that no signal would have caught. There is usually exactly one, and it is worth its own conversation. Typically it is a business-rule violation: a value that was inside the normal range and still wrong, a total that reconciled against itself and not against the source system, a rule that only one person in finance knows. No amount of metadata monitoring sees these.
Write the ask. Three lines: how many of your incidents the first two signals would have caught, how many more the third and fourth add, and what each stage costs in time rather than licences. Bring it to session 5, where it meets the list of tables and becomes a plan.
The one that got away belongs to a different discipline. A rule like "every order must have a delivery slot in the future" is not a property of the table's shape, it is a statement about the business, and it lives as a row-level assertion in whichever project builds the table. Row-level assertions at build time and metadata monitors on a schedule are complementary layers, and a team that tries to make one do the other's job ends up with both badly done. Point that incident at whoever owns the transform code, and keep it out of the monitoring plan.
Before session 3
20 min- Finish the five-column markup on your whole incident log, then show it to whoever would build the checks. They will disagree with at least two of your ticks, and the disagreements are the useful part.
- For your top table, write down the promise: by what time, on which days, at roughly what size. If nobody can state it, that is your first finding, and it is a more common one than a late table.
- Ask whether anything already maintains a dependency graph in your estate, and if so, when it was last accurate. Transform tools often generate one that nobody has looked at. A stale graph is a liability rather than an asset.
- Write one sentence for each of the five signals, in your own words, that you could say to a vendor without reading it off a slide. Session 3 assumes you can.
Three questions
On night 19 the merge key changed, the duplicate rate went from 1.0 to 8.4 percent, and the row count stayed normal. Which checks reported a clean night?
Duplicates and dropped rows nearly cancelled in the total, so the count landed within a percent of the median. Three checks reported reassuringly while every downstream sum was wrong, which is the whole argument for uniqueness having its own monitor.
A proposal arrives that leads with column-level lineage and would take a quarter to deliver. What is the strongest objection?
Lineage is a map, and a map notices nothing. It shortens the hunt for cause and it tells you who to warn, both of which are genuinely valuable, and neither of which reduces the hours a table was wrong before anybody knew.
Why is the five-signal framing worth using even though it came out of the vendor market rather than a standards body?
No standards body publishes these five and no auditor asks for them. What makes the carve-up useful is that everyone in the market uses the same words, and that the two questions underneath it are real and genuinely different.