Why this is the governance session
Session a2 closed on an uncomfortable sentence: models as code makes a good model cheaper to maintain and a bad model faster to spread everywhere, in seconds, on the next build. This session is about the thing that stops the second half of that sentence, and it is the one part of the analytics-engineering story that you can hold a team accountable for numerically.
Most data-quality conversations at leadership level are unfalsifiable. Somebody asserts the data is trustworthy, somebody else asserts it is not, and nothing in the room can settle it. A test suite changes that, because a test is a claim about the data that either holds or does not, recorded every time the estate is rebuilt. Read that way, the suite is a governance artifact: an inventory of what your organisation has committed to being true, plus a record of every time one of those commitments failed.
Everything measured on this page comes from this course's own project: the Bazaar marketplace warehouse, 23 models, 100 tests (95 cheap generic ones and 5 hand-written assertions), 1 snapshot, 8 declared sources. A full build reports PASS=121, WARN=0, ERROR=0 in 1.22 seconds. The same warehouse also exists hand-written, as 8 SQL files plus a Python validator with 27 checks, and both produce settled net revenue of 111,906.48 over 957 orders, AOV 116.93. The six-breakage measurement below was run against the real project, not estimated.
What a suite buys, measured 13 min live
The mechanism is worth thirty seconds even though you will never write one. A test is a query that selects the rows that should not exist. If it returns nothing, the test passes; if it returns rows, the test failed and the build stops. That is the whole idea, and its simplicity is why tests can live beside the models and run every single time rather than when somebody remembers.
Which means the interesting question is not how tests work. It is what a suite is worth, and that is answerable. Six realistic breakages were applied to this project as real damage - a retried load creating duplicates, a widened column, a key that stops resolving, a source edited after the build, a metric definition changed without a rename, an aggregate that quietly disagrees with the fact it summarises - and the suite was re-run against each one at three levels of coverage.
LiveHow to read a suite as a governance artifact4 min▶
Stop thinking of the suite as quality-assurance work and start thinking of it as a register of commitments. Every test in it is a sentence of the form "this must be true about our data, and if it stops being true we want the build to stop". Read that way, three properties of a suite matter to you and none of them require SQL:
- Coverage tells you where you are exposed. Not a percentage, which is a meaningless number here. A list: which models have zero tests. A model with no tests is a model your organisation has made no claims about, and if a reported figure depends on it, that is an unhedged position rather than an oversight.
- Composition tells you what class of failure you are protected from. A suite that is entirely cheap generic tests protects you from broken shapes and not at all from wrong meaning. This project's ratio is 95 generic to 5 hand-written, and the 5 are what took it from 4 of 6 to 6 of 6.
- Failure history tells you whether the suite is real. A suite that has never failed is either protecting a very stable estate or is not being run, and the second is far more common. A suite that fails constantly is being ignored, which is worse.
LiveWhy 100 tests is not a target and 23 models is not a denominator4 min▶
The moment you learn that a suite can be counted, the temptation is to set a target. Resist it, and resist a coverage percentage even harder, because both are trivially satisfiable in ways that make the estate worse.
Tests are cheap to add and nearly free to add badly. A team asked to raise a coverage number will add a not-null test on a column that is already part of a unique key, an accepted-values test on a field the business changes monthly, and a relationships test between two models one person owns end to end. All three pass, all three raise the number, none of them would ever catch anything you care about, and the third will start failing for legitimate business reasons and train everybody to ignore failures.
The useful question is per model and it is judgement, not arithmetic: for this model, can somebody name the failure that would embarrass us most, and is there a test for that specific failure? Ask it of your ten most consequential models and you will learn more than any coverage figure could tell you. This project arrived at 100 tests over 23 models without anybody setting a target, which is the only way that ratio means anything.
Self-studyWhy the hand-written estate scored zero3 min read▶
It is worth being fair about this, because "0 of 6" sounds like an indictment of the people who wrote the hand-written version, and it is not. That version has a Python validator with 27 checks in it, and those checks are good checks. Several of them would catch several of the six breakages.
The zero comes from one structural fact: the validator is not attached to the build. It is a separate program a person runs. So the honest question is not whether the checks are good but how often they run, and in every organisation the answer is: reliably at first, then when something feels wrong, then almost never. Nobody decides to stop. It simply stops being the thing that happens under deadline pressure, and the checks that matter most are the ones you skip on the day you are in a hurry, which is also the day you are most likely to break something.
That is the actual difference between a validator and a test suite, and it is a process property rather than a technical one. A test that runs on every build has no opportunity to be skipped. Session a2's environments practice is what makes that possible: if the estate can be rebuilt on demand, tests can be attached to the rebuild.
Shape errors, meaning errors, and who owns which 9 min live
There are exactly two classes of data failure that a suite can be aimed at, they need completely different tests, and they should be owned by different people. Most organisations protect themselves thoroughly against the cheap class and not at all against the expensive one, and the reason is entirely understandable: the cheap class is easy to test and easy to explain.
| Failure class | What it looks like | What catches it | Who owns it |
|---|---|---|---|
| Shape errors (the cheap class) |
Duplicate rows after a retried load. A key that is suddenly missing. A lookup that stops resolving, so an inner join quietly loses rows. A new status value appearing upstream while every dashboard filters on the old list. | One-line generic tests, declared once per column and reused everywhere: keys unique, keys present, foreign keys resolve, values within a declared set. 95 of this project's 100 tests are these, and they caught 4 of the 6 breakages. | The data team. No business input needed. Nobody has to be consulted about whether an order id should be unique, so there is no excuse for these being absent. |
| Meaning errors (the expensive class) |
A definition changes without a rename, so revenue is 3.5% higher and the column is still called revenue. Somebody edits a source after the build, so the warehouse and the system of record disagree. An aggregate quietly stops matching the fact it summarises, creating a second version of the truth. | An assertion a human thought about: this total must still equal the source to the cent, this derived measure must still equal its own parts, this summary must still agree with the detail. Five of these took the project from 4 of 6 to 6 of 6. | The business, with the data team. Only a business owner can say what the number must reconcile to, and a reconciliation assertion needs somebody who will answer the phone when it fires. |
LiveWhy meaning errors are the ones that cost you4 min▶
Follow one of the six breakages through, because the shape of the incident is what makes the argument. A mart loses its settled-money filter. The column is still called net revenue, every row is well-formed, there are no duplicates, no missing keys, no orphaned references, no undeclared values. Revenue is 3.5% higher than it should be.
Every one of the 95 cheap tests passes, and they are right to pass, because nothing about the shape is wrong. The build is green. The dashboards refresh. The number goes into a weekly pack, and then into a monthly one. There is no incident, no alert and no failure, and there will not be until a human being who knows what the number should look like happens to look at it. In this project's measurement, that breakage and one other are the two that generic tests structurally cannot see.
Two consequences worth naming for a leadership audience:
- The cost of a meaning error scales with how long it goes unnoticed, and there is nothing bounding that duration except somebody's intuition. A shape error is usually caught in hours because something visibly breaks. A meaning error can run for a quarter.
- These are not edge cases. A definition changing without a rename and a source being edited after the build are the two most ordinary ways a warehouse that has worked for years starts producing wrong numbers. They are what happens when an estate is healthy enough to be actively changed.
LiveWho owns which test, and why the phone matters3 min▶
The ownership split is clean, and getting it wrong is the most common reason meaning tests do not exist.
Shape tests belong entirely to the data team. Nobody needs to consult the business about whether an order identifier should be unique or whether a required key can be absent. These are engineering claims about a well-formed table, they are one line each, and there is no legitimate reason for a consequential model to lack them. If your team says the business has not told them what the rules are, they are describing a meaning test and calling it a shape test.
Meaning tests need a business owner, by name. Only somebody accountable for the number can say what it must reconcile to, what tolerance is acceptable, and whether a 0.4% difference is a rounding artifact or a genuine failure. The data team writes it, but the claim is not theirs to make.
And here is the part that is easy to skip. A reconciliation assertion needs a business owner who will answer the phone. When that test fires at 3am, somebody has to decide whether the correct action is to fix the pipeline, restate the number, or accept the difference, and that is not an engineering decision. A reconciliation test with no named owner will fire once, get investigated by whoever is on duty, get no answer, and be downgraded or disabled within a fortnight. That is not a testing failure, it is a governance failure, and it is yours to prevent.
Self-studyThe five assertions, in plain English3 min read▶
Five hand-written checks took this project from 4 of 6 to 6 of 6. None of them is clever, and every one of them started as an incident class somebody had lived through. In plain English:
- Reconcile to the source. Settled revenue in the warehouse must still equal the source system, to the cent. The single most valuable assertion in the project and the one auditors ask for by name.
- A derived number must equal its own parts. If net is defined as gross less discount, then it must still be exactly that. A derived measure that has drifted from its own definition means the model is internally inconsistent, and that inconsistency will surface as an argument between two teams.
- A summary must agree with its detail. An aggregate that disagrees with the fact table it summarises is a second source of truth, which is worse than having no summary at all, because both numbers are now defensible.
- Broken lookups must stay rare. Failed joins are made countable rather than invisible, and this fires if their share climbs past a small threshold. Silent row loss is the failure that makes two correct reports disagree.
- Joining payments must not change revenue. A guard against the classic double-counting mistake, expressed as a test instead of as a warning in a comment nobody reads.
Notice what they have in common: each one is a sentence a business person can read, agree with, and be accountable for. That is the practical test of whether something is a meaning assertion. If a stakeholder cannot understand the claim, they cannot own it, and a claim nobody owns will not survive its first false alarm.
What should block a build, and what should only warn 5 min live
Not every failing test should stop the world. Deciding which ones do is a governance call rather than a technical one, and it is the call most likely to land on your desk. There is one question that settles almost every case, and it is deliberately unsentimental.
LiveThe 3am test, and the case for deleting tests3 min▶
The question is: if this test fails at 3am, would anybody act on it? If yes, it should stop the build, and stopping the build is a genuinely valuable behaviour. Nothing downstream of the failure gets rebuilt, so the dashboards keep yesterday's numbers instead of being refreshed from something known to be wrong. In a hand-written estate the equivalent script would have carried on regardless, because a script runs steps and has no idea what depends on what.
If nobody would act on it, the test is noise, and this is where leaders reliably get the decision backwards. The instinct is that more alerts mean more safety. In practice a suite that cries wolf gets skipped, and once your team has learned to click through failures they will click through the important one too. A skipped suite is worse than no suite, because no suite means you know you are exposed, while an ignored suite means you believe you are covered.
So the honest options for a test nobody would act on are: delete it, or make it a warning that reports without failing the build. Warnings are the right home for things worth watching and not worth waking anybody for. What is not an option is leaving it as a blocking failure that everybody has learned to override, which is the state a great many real suites are in.
LiveReading a test suite as a leader, without reading SQL3 min▶
Three questions, all answerable by your team in a few minutes, none requiring you to look at any code. Together they tell you more about your exposure than any dashboard about data quality.
- How many of our models have zero tests, and which ones? A count and a list. Then ask which of those models a reported number depends on. That intersection is your unhedged exposure, stated as a list of names rather than a feeling, and it is usually shorter and more fixable than people expect.
- Which of the numbers we report has a reconciliation assertion behind it? Ask about your three most-quoted figures by name. This is the question that separates a suite protecting against shape from a suite protecting against meaning, and the answer for most organisations is none of them.
- When did the suite last fail, and what happened next? The best question of the three, because it tests the whole system rather than the artifact. A good answer names a date, a failure, a person and an action. "It does not really fail" means it is not being run, or it is only testing things that cannot break. "It fails most days" means it is being ignored.
Notice that all three are questions about your organisation rather than about the technology, which is why they keep working when the tooling changes.
The trust-budget audit ★ run this with your own team
Two exercises. The first builds the case for the tests you are missing, using your own reported numbers rather than an argument about best practice. The second finds the noise you are already paying for, and it is the one that saves money rather than spending it.
Name your three numbers first, before anybody talks about tests. The three figures your organisation actually acts on. Do this out loud, and be prepared for the list itself to be contested.
Then ask, for each one, what would fail if it silently moved by 3%. Three percent because it is too small to notice by eye and large enough to change a decision.
Now turn it around and hunt for noise. Which current failures would your team skip at 3am? Those are candidates for deletion or downgrade this week.
Leave with two lists and one name. The assertions to write, the alerts to remove, and the person who will answer the phone when a reconciliation fires.
LiveExercise 1 · The three-numbers question6 min · run it next week▶
The prompt, word for word
"Name the three numbers this company actually acts on. For each one, tell me which test would fail if it silently moved by three percent, and who we would call when it did."
What a good answer sounds like
For at least one of the three, a real assertion and a real name: "settled revenue reconciles to the payments system every build, and if it breaks we call Dana in finance." That is a governed number. For the other two, the good answer is a straight admission: "nothing would fail, we would find out from the monthly review." Written down beside the number's name, that admission is the most persuasive funding case you will make this quarter, because it is specific, it is about a number your board sees, and nobody has to be blamed for it.
The failure mode to listen for
Shape tests offered as if they were meaning tests. "We test that the revenue column is never null" sounds like an answer and is not one: a 3% error has no nulls in it. Push once, gently: "would that test fail if the number were wrong but well-formed?" The second failure mode is a technical owner given for a business claim. If the answer to "who would we call" is the data team, the assertion has no business owner, and it will not survive its first false alarm.
LiveExercise 2 · The noise audit6 min · run it next week▶
The prompt, word for word
"How many test or data-quality failures did we see last month, and how many led somebody to do something? Then: which of our current alerts would you personally skip at 3am?"
What a good answer sounds like
Two numbers and an honest list. "About twenty-five failures, three of them mattered, and I would skip the freshness alert on the marketing extract every single time because it is late twice a week and nobody minds." That is a team that can be trusted with its own suite, and the right response is to let them delete or downgrade those alerts this week rather than to ask for a plan. Deleting a noisy test is a real improvement to your protection, and it should feel like one.
The failure mode to listen for
Reluctance to delete anything, usually phrased as caution: "I would not want to remove a check." That instinct is exactly backwards and it is worth naming in the room, because the risk is not the alert you removed deliberately, it is the alert your team learned to ignore silently. Also listen for the failure count being unknown, which means nobody is watching whether the suite works, and that is a bigger finding than any individual test.
The measurement on this project is the most useful single slide in this whole track, and it is worth reproducing exactly. Six realistic breakages, applied to a working 23-model warehouse as real damage, with the suite re-run at three levels. No tests: 0 of 6 caught. Cheap generic tests: 4 of 6. Generic plus five hand-written assertions: 6 of 6.
The middle column is the lesson. Teams add generic tests quickly, because they are one line and they feel like progress, and they genuinely do catch the shape errors: duplicates, missing keys, broken lookups, undeclared values. Then a definition changes without a rename, or somebody backfills a source after the nightly build, and every one of those 95 tests passes while the number is wrong. Those two breakages are not exotic. They are the two most common ways a mature warehouse loses trust, precisely because they only happen to estates healthy enough to still be changing.
And notice what "no tests" really describes. The hand-written version of this warehouse has 27 checks written into a Python validator, and they are competent checks. It scores zero because the validator is a separate program somebody has to remember to run, and under deadline pressure nobody does. The 100 tests in the dbt version run on every build, in 1.22 seconds, whether anybody feels like it or not. That is the difference you are funding, and it is a process property rather than a clever one.
Questions to ask your data team 7 questions
None of these requires you to read SQL, and all of them are answerable by a team that has the situation under control. The last three are the ones to repeat every quarter.
| Ask this | Why it works |
|---|---|
| "How many of our models have zero tests, and which reported numbers depend on those models?" | Turns coverage into an exposure list rather than a percentage. The intersection of untested models and reported numbers is your unhedged position, stated as names you can act on rather than a proportion you cannot. |
| "Which of our three most-quoted numbers has an assertion that it still reconciles to the source system?" | The single highest-value question on this page. It separates a suite that protects shape from one that protects meaning, and for most organisations the honest answer is none of them, which is the funding case in one sentence. |
| "When did the suite last fail, and what happened in the next hour?" | Tests the whole system rather than the artifact. A good answer has a date, a failure, a person and an action. "It does not really fail" means it is not running or not testing anything that can break. |
| "How many failures did we see last month, and how many led to somebody doing something?" | Measures noise, which is the failure mode nobody reports upward. A large gap between the two numbers means your team has learned to click through failures, and they will click through the important one too. |
| "For our most important model, what is the failure that would embarrass us most, and is there a test for exactly that?" | The right substitute for a coverage target. It is a judgement question that cannot be satisfied by adding cheap tests, and asking it per model produces a real backlog instead of a number. |
| "Which of our tests would you delete this week if I told you deleting one was a win?" | Gives explicit permission, which is what the answer needs. Teams keep noisy tests out of caution, so the useful move is to make removal a legitimate outcome rather than an admission of a mistake. |
| "For each reconciliation test we have, name the business person we call when it fires." | Closes the governance gap directly. A reconciliation assertion with no named business owner will fire once, get no answer, and be disabled within a fortnight, and that is a governance failure rather than an engineering one. |
Before session a4 ◐ 30-40 min total, all conversations
- Write down your three most-acted-on numbers and, beside each, whether a reconciliation assertion exists. Take the list to your next leadership meeting. It is a one-page governance document and it took ten minutes.
- Ask for the count of models with zero tests, and for the subset of those that a reported figure depends on. Do not ask for a coverage percentage, which is satisfiable without improving anything.
- Run the noise audit and give your team explicit permission to delete or downgrade three alerts this week. Then check in a month whether the remaining failures are being acted on.
- For your single most important number, name the business person who will answer the phone when a reconciliation assertion fires, and tell them. If you cannot name one, you have found the real gap, and it is not a technical one.
- Bring the count of models and the count of editors to a4, along with your current test position. The buy-versus-build decision, including Core versus Cloud, turns on all three.
Sources covered
This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Your team reports 96% test coverage and every build is green. A monthly figure has been 3% too high for six weeks. What is the most likely explanation?
A definition that moved without a rename produces no duplicates, no missing keys, no orphans and no undeclared values. Every cheap generic test passes correctly. Only an assertion about meaning, such as a reconciliation to the source, catches it, which is why five hand-written checks took this project from 4 of 6 to 6 of 6.
2 · A test has failed most mornings for three months. The team overrides it each time and nothing bad has happened. What should you do?
Once your team has learned to click through failures, they will click through the important one too. No suite means you know you are exposed; an ignored suite means you believe you are covered. Deleting a noisy test is a genuine improvement in protection and should be treated as a win, not an admission.
3 · Your team proposes a reconciliation assertion on reported revenue. What is the one thing you must supply for it to survive?
Whether to fix the pipeline, restate the number or accept a difference is a business decision, not an engineering one. A reconciliation test with no named owner fires once, gets investigated by whoever is on duty, gets no answer, and is disabled within a fortnight. That is a governance failure, and it is the leader's to prevent.