How the leader track works
Six sessions, 45 minutes each, no SQL. The artifact you build across them is a one-page trust report you could hand to a CFO. Tonight it is the cost: what a wrong table does to a business and who currently discovers it. Session 2 takes the five signals and what each one is blind to. Session 3 picks which quality dimensions are worth measuring at all. Session 4 is the noise problem, which is where most programmes quietly die. Session 5 is tiering, because you will never instrument everything. Session 6 turns the whole thing into the three numbers you actually report. The running example throughout is orders_daily at Bellwether, a grocery delivery company with five real defects sitting in its warehouse that nobody has noticed yet.
★ What you walk out with tonight. A list of your last six months of data incidents with two columns filled in: hours wrong, and who found it. The share of that list found by a stakeholder rather than by your team, which is the single number that changes how a leadership meeting goes. And one euro figure on your worst incident, because a business case built on hours is an argument and a business case built on money is a decision.
LiveWhy the cost comes before the tooling3 min▶
The usual order is backwards. A team gets burned by an incident, evaluates three vendors, buys one, instruments four hundred tables, and six months later cannot answer whether anything improved. The reason is that nobody wrote down the starting position, so there is nothing to compare against.
The starting position is two numbers and both are free to collect: how long your tables are wrong before anybody knows, and what share of that discovery comes from outside the data team. Everything after that is a question of how much of the gap you want to buy, and at what price in interruptions.
Bellwether's finance team found a broken revenue number on a Monday morning. The pipeline had not failed, so nothing alerted. One of three regional shards had stopped arriving on the Thursday before, and the table had been at 41 percent of a normal day's rows for two business days. Three decisions were made on that number before anybody knew, including a replenishment order for the wrong region.
Self-studyThe vocabulary the rest of the track uses4 min▶
- Data downtime - the hours in which a table was late, short or wrong while people were relying on it. Not the hours it was broken in the abstract, which is why a defect that lands on a Saturday and is fixed on Sunday costs almost nothing.
- Time to detect - from the moment the table went wrong to the moment a human knew. The part of downtime you can shorten with money and effort.
- Time to resolve - from knowing to fixed. Mostly a function of your team's capacity and the state of the upstream system, and much harder to buy.
- Discovery gap - the share of incidents reported to you rather than by you. A programme is working when this falls, regardless of what the coverage dashboard says.
- Blast radius - everything downstream of a broken table: models, dashboards, exports, and the one job that pushes into the CRM that nobody has documented.
- False page - an alert on a night when nothing was actually wrong. The currency you spend to buy detection, and the subject of session 4.
A green pipeline is not evidence
9 min liveThere is a comfortable belief that if the jobs ran, the data is fine. It survives because most of the time it is true, and it fails on exactly the incidents that cost the most. Three failure shapes account for almost all of them, and none of the three raises an error anywhere in your stack.
LiveWhy orchestration success is a weak signal3 min▶
An orchestrator reports on itself. It knows whether the task it launched finished, how long it took, and whether the process returned an error code. It does not know how many rows landed, whether the third shard was in the batch, or whether the customer id column went 90 percent null overnight. It was never designed to know any of that, and asking it to is a category error.
So the green tick answers a real question, just a narrower one than people read it as: did the code we wrote run to completion. It says nothing about whether the upstream system it read from behaved. In practice most of your painful incidents come from upstream behaviour, because that is the part you do not control and do not deploy.
A useful test for any status board. Ask what it would show on a night when the source system silently sent two thirds of the usual data. If the answer is a green tick, then the board is reporting on your code and somebody still has to report on your data.
LiveA pipeline error and a data defect are different animals3 min▶
They arrive differently, they are found differently, and they cost differently. Worth separating out loud, because the two get merged into one line item called "data issues" in most incident reporting, and the merged number cannot be acted on.
| Pipeline error | Data defect | |
|---|---|---|
| How you learn | The tool tells you, usually within minutes | Somebody tells you, usually on a Monday |
| Typical detection | Minutes | Hours to days |
| Who is annoyed | Your engineers | Your stakeholders, and their stakeholders |
| What is downstream | Nothing yet, the build stopped | Everything, because the build succeeded |
| Reputational cost | Low. Failure handled | High. A number was believed and used |
The last row is the one that matters to you. A pipeline that fails loudly is a pipeline doing its job, and stale data is often better than wrong data because at least it looks stale. The expensive failure is the one that delivered a confident, complete, plausible, incorrect answer, and it did so on time.
Self-studyWho currently finds your incidents4 min▶
Ask five people on your team to name the last three data incidents. Then ask who reported each one. The answers cluster into four groups, and the mix tells you almost everything about the maturity of the practice without a single tool being installed.
- A monitor found it. Detection in minutes, no audience involved, the incident stays internal. This is the only group where downtime is cheap.
- An engineer found it by accident while working on something else. Detection in hours, usually because the shape of a query looked odd. Real, but you cannot staff a practice on luck.
- An analyst found it mid-analysis, which is the most common answer in most organisations. Detection in a day or two, and the cost includes their wasted afternoon.
- A business stakeholder found it and told you. Detection in days, plus a trust cost that outlives the fix by months. Every one of these is a bill you already paid.
Count the fourth group as a proportion of the whole. If it is above about a third, the programme you need is not a smarter detector, it is any detector at all on the tables that matter, which is session 5.
Data downtime, as a number
8 min live"We have data quality problems" is not fundable. "Our top twelve tables were wrong for 340 hours of business time last quarter, and stakeholders found 6 of the 9 incidents before we did" is fundable, and it is the same information. The unit that carries the argument is hours wrong while in use.
LiveThe four inputs to a downtime number4 min▶
Four inputs, and you can estimate all four from memory well enough to be useful. Precision is not the goal on the first pass; getting the order of magnitude on the table in front of the person who funds things is.
| Input | What it is | Can you change it? |
|---|---|---|
| Detection time | Hours from wrong to somebody knowing | Yes, directly. This is what monitors buy |
| Resolution time | Hours from knowing to fixed | Slowly. Capacity, runbooks, upstream owners |
| Blast radius | How many reports, models and exports read it | Yes, by simplifying the estate. Not quickly |
| Decisions taken while wrong | What was actually acted on before the fix | Only by shortening detection |
Multiply the first two by the third and you get exposure in report-hours, which is a reasonable proxy for how much wrongness was available to be believed. The fourth input is the one that turns exposure into money, and it is the only one a business stakeholder can help you fill in.
Bellwether costed the finance incident at about 38,000 euro: 24,000 in a replenishment order sized against the short table, 9,000 in a promotional spend decision that would have gone the other way, and 5,000 in analyst and engineer time spent reconciling. The number that got the programme funded was the 24,000, because it was a real invoice with a real signature on it.
LiveThe discovery gap, and why it changes behaviour4 min▶
Of all the numbers available to you, one moves the room: the percentage of incidents that a stakeholder reported to you rather than your team reporting to them. It works for three reasons and they are worth knowing before you present it.
- Everybody can check it. The people in the meeting were the ones who reported those incidents. There is no methodology to argue about, which is not true of a coverage percentage.
- It cannot be gamed by adding monitors. Coverage goes up when you instrument more tables, whether or not anything improves. The discovery gap only falls when detection genuinely beats the audience.
- It is about trust, not about tables. The cost of a stakeholder finding your defect is not the defect, it is that the next correct number also gets questioned. That cost is invisible in every technical metric you have.
Set it as the headline goal and the follow-on decisions get easier. Which tables to instrument first becomes "the ones stakeholders keep finding problems in". Whether a monitor is worth keeping becomes "did it ever beat a human to something real". Session 6 makes this one of the three numbers you report on a page.
Self-studyTime to detect is the part you can fund4 min▶
Split any incident into detect, then resolve. Both are hours and they feel equivalent on a slide, but they behave completely differently when you try to spend money on them.
Resolution depends on the upstream system, whoever owns it, whether they are in your organisation, whether a backfill is even possible, and how many people you have. Halving it is a reorganisation. It is real work and it is not a project you can start this quarter with a modest budget.
Detection depends on whether anything is watching. Going from nothing to a scheduled check on your twelve most-read tables is days of work and no new licences, and it typically takes detection from days to under an hour on the failure shapes those checks can see. There is no other lever in the practice with that ratio.
Which is why the honest version of the first business case is narrow: no promise about fewer incidents, no promise about faster fixes, one promise about hours of exposure. Overpromising here is the most common way a programme loses its sponsor in month four, and session 6 has the list of things not to promise.
Your own downtime audit
★ 20 min · everyone buildsNo tooling, no access, no data pull. Six months of incidents that people already remember, five columns, twenty minutes. This is the artifact you extend in every later session, and it is deliberately built from memory because memory is what your stakeholders are working from too.
List the last six months of incidents anybody remembers. Ask the room, not the ticket system. If it never became a ticket but three people still bring it up, it counts, and if it was ticketed and nobody remembers it, it probably did not cost much. Expect somewhere between five and fifteen. Fewer than five usually means the incidents are being absorbed silently rather than not happening.
For each one, write who found it and how many hours it was wrong. Hours of business time, not wall clock: a table that broke on Friday evening and was fixed on Saturday was wrong for very few usable hours. Round to whole hours and argue about it out loud, because the argument is where the useful detail comes out.
Count the share found by a stakeholder rather than by your team. One division. Write the fraction, not just the percentage, because 4 of 9 lands differently than 44 percent in a room of people who remember all nine. This is your discovery gap and it is the baseline you will be measured against, so record the date you took it.
Put one euro or dollar figure on the worst one. Pick the single worst incident and cost only what you can defend: an order placed at the wrong size, a spend decision that would have gone differently, hours of analyst time redone. Skip anything speculative. One defensible number beats a spreadsheet of estimates, and it is the sentence that gets repeated when you are not in the room.
Name the tables involved and count them. Nearly every group is startled here. Ten incidents typically involve four or five distinct tables, often the same two repeatedly, and those are almost always the tables the business actually reads. You have just done the first draft of the tiering exercise from session 5 without meaning to, and you did it from evidence rather than from an inventory.
How you say it decides whether it is heard. The audit is not the deliverable. One sentence with a number, a source and a consequence is the deliverable, and it is worth writing before you leave the room.
Before session 2
25 min- Finish the incident log properly. Send it to two people who were involved in the incidents and ask them to correct the hours. They will, and the corrections are usually upward.
- Ask one business stakeholder a single question: "when did you last not trust a number in a report, and what did you do about it?" Write the answer down verbatim. It is more persuasive than anything you will produce yourself.
- Take the tables from your log and rank them by how many incidents each one appears in. Bring the top five to session 2.
- For your worst incident, write down which of the three failure shapes it was: late, short, or wrong-but-complete. If you cannot tell, that itself is worth bringing, because it usually means nobody recorded what the table looked like at the time.
Three questions
One of three regional shards stops arriving. The load job reads what is there, finishes, and exits cleanly. What does your orchestrator report?
An orchestrator reports on its own tasks: did the process finish, how long did it take, what exit code. Row counts and missing shards are properties of the data, and nothing in that stack was asked to look at them.
Which failure shape is the most expensive, and why?
Stale data usually looks stale, and a missing table announces itself. A table that is punctual, correctly sized and quietly incorrect is the one that ends up in a decision, which is where the cost lives.
You need one number for the leadership meeting. Why is the discovery gap a better choice than a coverage percentage?
Coverage measures your effort. The discovery gap measures the outcome, everybody in the room can verify it from memory, and it cannot be improved by adding monitors that never catch anything.