Why this bill is invisible
A bad model never sends an invoice with its own name on it. It bills you through other people's line items: analyst time coded as "reporting", a BI migration that ran three months long, a headcount request to "improve data quality", a decision made on instinct because nobody trusted the dashboard. Session a1 argued that the schema is a business decision. This session prices it.
One honest caveat before the numbers: the figures on this page come from Bazaar, the small marketplace dataset this course is built on - 985 orders over 90 days. They are an illustration of the arithmetic, not a benchmark for your organisation. The percentages are what transfer; the dollars are yours to substitute.
The four cost lines 8 min live
Three of these are money you can count. The fourth is the one that actually hurts, and it compounds.
| Cost line | How it shows up in your world | The signal it is already happening |
|---|---|---|
| 1 · Rework | dashboards rebuilt, pipelines re-pointed, a migration that overruns | an engineer says "we need to redefine what a row means in that table" |
| 2 · Reconciliation | analyst time coded as reporting, recurring "why do these differ" meetings | somebody owns a spreadsheet that bridges two systems, and only they understand it |
| 3 · Lost trust | decisions made on instinct, teams keeping private numbers | a leader says "I do not use that dashboard, I ask my team directly" |
| 4 · Automation cap | self-service stalls, an AI analyst pilot never leaves pilot | a vendor demo works and your own data breaks it in week two |
LiveRework scales with time trusted, not with size of error3 min▶
This is the counterintuitive part, and it is the argument that unlocks budget. A small modeling error found in week one costs a conversation. The same error found after eighteen months costs every dashboard, alert, pipeline, board slide and external commitment built on top of it - plus the credibility of everyone who quoted it.
- The multiplier is downstream count, not error magnitude. A 3% wrong number that forty assets depend on is far more expensive than a 40% wrong number nobody has built on yet.
- Therefore the cheapest moment to fix is always now, and the second cheapest is before the next team builds on it. That second window is the one leaders actually control.
- And it explains the pattern you have seen: the team that raises a modeling concern early gets told it is premature, and the same concern eighteen months later becomes a programme with a steering committee.
Self-studyA worked cost sketch, with the arithmetic visible4 min read▶
Bazaar's dataset is deliberately small, which makes the arithmetic legible. Substitute your own volumes and the shape holds.
- The error: a dashboard counts rows in the order-line table and calls them orders. Bazaar has 985 orders and 1,434 order lines, so June reports 481 orders instead of 327 - a 47% overstatement. Nothing errors. The number simply reads high, consistently, forever.
- Who acts on it: whoever sets acquisition targets from order volume. Every target derived from that number is set against a base that is half again too big, so performance looks like underdelivery when it is not.
- The second error, same table: average order value computed by averaging line values instead of dividing revenue by orders. Bazaar's true AOV is 116.93; the wrong method gives 80.68 - a 31% understatement. Note the direction: one metric reads high while a related one reads low, so the two errors hide each other in any sanity check.
- The third, from joining two tables that should never meet: revenue inflates from 111,906.48 to 121,003, about 8% high. Small enough to survive review, large enough to matter in a forecast.
- What it costs to find: not much, if the model published what one row means. Weeks, if it did not - because the investigation is not a query, it is an archaeology project across everyone who ever touched the table.
The pattern worth taking to your own numbers: modeling errors are usually not wild, and that is exactly why they survive. A number that is 47% wrong in one direction and 31% wrong in the other still passes a smell test, and passes it every month.
Fix it, live with it, or ringfence it 7 min live
Not every known modeling problem deserves a project this quarter. Three options, and three signals that decide between them.
| Option | What it means in practice | When it is the right call |
|---|---|---|
| Fix it | remodel the subject area, migrate the downstream assets, retire the old shape | any of the three signals below is present |
| Live with it | document the compromise where consumers will see it, name an owner, revisit on a date | none of the signals is present and the blast radius is one team |
| Ringfence it | freeze the bad shape, forbid new dependencies, build new work against a corrected model beside it | a fix is right but cannot be sequenced this quarter |
LiveThe three signals that mean fix it now4 min▶
- 1. The number is in an external commitment. Board reporting, an investor deck, a regulator, a customer SLA, a partner rebate. Once a wrong number leaves the building, correcting it is no longer a data project - it is a communications event.
- 2. The error compounds with volume. If growth makes the gap bigger rather than proportionally the same, waiting is actively expensive. Double-counting in a payments join gets worse as retry rates rise.
- 3. A second team is about to build on it. This is the signal leaders can see and the data team often cannot, because it lives in the roadmap rather than the schema. Every new consumer multiplies the eventual rework, so the moment before onboarding one is the last cheap moment.
If none of the three is present, "live with it, documented, with an owner and a review date" is a legitimate engineering decision - and it is a far better answer than a vague promise to clean things up, because it can be audited.
Self-studyWhat "documented" has to mean to count3 min read▶
"We know about it and it is documented" is only a real answer if a consumer would actually hit the documentation. A wiki page nobody opens is not a control. This course's own model carries its compromises in a known_limitations block inside the data contract that ships beside the tables, which means anyone connecting to the model meets the caveats on the way in.
- Name the compromise, its blast radius, and who owns it. "Carts are attributed to the merchant on the highest-value line, so mixed carts credit all abandonment to one merchant."
- Put it where the query happens, not where the meeting happened - in the catalog entry, the contract, the metric definition's caveat field.
- Give it a review date, because an undated compromise becomes a permanent one on its first anniversary.
Session a5 turns this into a standard, and the builder track ships the artifact. What matters for a leader is the test: could someone build on this model and never learn what it does not do? If yes, it is not documented.
Price one problem you already know about ◆ run this in your own meeting
Every organisation has a known modeling problem that has been discussed and not fixed. This exercise turns the discussion into a number and a decision, in about fifteen minutes.
Name it in one sentence, as a grain or definition problem: "our sales table has one row per shipment, but we report it as orders".
Count the dependants. How many dashboards, alerts, pipelines and external commitments read that table? The count is the rework multiplier, and it is usually higher than anyone in the room guesses.
Test the three signals. External commitment? Compounds with volume? A team about to build on it? One yes means fix.
Decide out loud, and write the decision down with an owner and a date - including if the decision is to live with it.
LiveWhat a good answer sounds like, and what to listen for4 min▶
Good: "One row is one shipment. Eleven dashboards and the partner rebate calculation read it. The rebate is contractual, so signal one is present. We fix it this quarter, Priya owns it, and we freeze new consumers from today."
Listen for these three failure modes:
- "It is only a reporting issue." Reporting is where the money is decided. This phrase almost always means nobody has counted the dependants.
- "We will handle it in the migration." A migration is the worst time to change a definition, because you lose the ability to tell a migration bug from a modeling change. Fix before or after, never during.
- Nobody can state the grain. If the room cannot finish "one row is one ___" for the table under discussion, stop the exercise. That is the finding, and it is a bigger one than the original problem.
The quarter nobody bills for
The shape repeats across industries. A team reports strong order growth; finance reports flat revenue; both are correct, because one counts lines and the other counts settled payments. The reconciliation takes weeks, the answer is a one-sentence grain difference, and the real damage is what happened in between: the leadership team stopped using either number and started asking people directly. The model was fixed in a fortnight. The habit of asking people instead of the dashboard took a year to unwind - which is cost line 3 doing exactly what the curve on this page predicts.
Questions to ask your data team 5 min live
- "What is one row in our most-used table?" If the answer takes more than one sentence, or two people give different sentences, you have found the cost line before it bills you.
- "How many assets depend on that table?" The number is the rework multiplier. A team that cannot answer it cannot price a fix, which is why fixes keep losing to features.
- "Which of our reported numbers appear in an external commitment?" That list is your fix-first queue, in order.
- "Where would a new consumer learn what this model does not do?" If the answer is a person rather than an artifact, the compromise is undocumented no matter what the wiki says.
- "What are we currently spending on reconciliation?" Ask for it in analyst-days per month. It is almost never zero, and it is almost never in a budget line with an honest name.
- "If we handed our data to an AI analyst tomorrow, what would it get wrong?" A specific answer means the team understands their model. A confident "nothing" means session a6 is going to be uncomfortable.
Before your next data review ◐ 30 min total, all conversations
- Pick your three most-quoted metrics and ask three different people to define each one precisely. Compare the answers. The gaps are your reconciliation cost, made visible.
- Ask for the dependant count on your busiest table. Then ask when the last new consumer was onboarded, and whether anybody checked the grain first.
- List every number your organisation reports externally. Beside each, write who can state its grain from memory. Blanks are risk, not admin.
- Find the person who maintains the spreadsheet that bridges two systems. Ask what it reconciles and how long it takes each month. That is a cost line with a name attached.
- Bring one known modeling problem to session a3, along with its dependant count - a3 is the workshop that makes grain arguments resolvable in minutes.
Sources covered
Full source map in materials/official-course-map.md. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Two modeling errors are found. Error A is 40% wrong and nothing has been built on it. Error B is 3% wrong and forty assets depend on it. Which is more expensive to fix?
The modeling change may be identical. The bill is everything downstream: forty assets to rebuild, plus every number already quoted from it. This is why the cheapest moment to fix is now, and the second cheapest is before the next team builds on it.
2 · Which cost line does not stay linear over time?
Rework and reconciliation are projects: scope them, staff them, finish them. Lost trust is a culture change, it compounds, and it is why the argument for fixing a model is rarely strongest when made purely in engineering hours.
3 · A team says a known grain problem is "documented, so we can live with it". What makes that answer legitimate rather than a deferral?
Documentation is only a control if a consumer would actually hit it on the way in. The test: could someone build on this model and never learn what it does not do? If yes, it is not documented - it is remembered, and memory leaves when people do.