Why this session exists
Here is the uncomfortable truth the first four sessions saved for now: everything you built a mental model of - the pickup, the shelf, the cleanup, the counting - fails regularly, at every company, forever. A 2022 Monte Carlo/Wakefield survey of 300 data professionals found teams spend roughly 40% of their day on data quality issues and suffer about 61 incidents a month. The mark of a good data team is not zero failures. It is how fast a failure is noticed. This session makes you the person in the meeting who asks the noticing questions.
The two ways a dashboard lies 7 min live
Every pipeline failure, whatever caused it, reaches the dashboard wearing one of two faces. Learn to name them and you can triage any data incident in one sentence - which is genuinely more than many managers can do.
LiveWhy "wrong" is the one that hurts3 min▶
A stale dashboard eventually confesses - someone notices Tuesday's number never changed, and the damage is usually a day of mild confusion. A wrong dashboard testifies. It carries a fresh timestamp, it sits in the Monday deck, it gets quoted in the board meeting, and every downstream decision inherits the error.
- Stale Daybreak: the 2am sync credential expired; the screen shows yesterday's $17,980 looking perfectly normal. Awkward, recoverable.
- Wrong Daybreak: a promo-code update starts writing prices as text; half the orders silently drop out of the total and the screen shows $9,120 - fresh stamp, wrong by half. Marketing kills the promo that was actually working.
One sentence to keep: freshness tells you the data arrived; it says nothing about whether the data is right. Two different questions, two different safeguards - that is part 3.
Self-studyTriage in one sentence2 min▶
When a number looks off, ask in this order: "Is it stale or is it wrong?" - then: stale means look at the truck (ingestion, credentials, the source system's night); wrong means look at the cleanup room and the definitions (a change shipped recently, upstream or in the transform). You will be right about where the problem lives most of the time, and your data team will quietly upgrade their opinion of you.
The four failures behind most incidents 6 min live
Post-mortems across the industry keep converging on the same short list. None of them is exotic, none needs code to understand, and you have probably lived through at least two.
LiveThe taxonomy, with Daybreak's version of each4 min▶
| Failure | What happens | Face on the dashboard |
|---|---|---|
| Upstream schema change (the most common) | The source app changes shape without warning - a column renamed, a price field switched to text. The truck arrives; the cargo no longer fits the shelf. | Usually WRONG |
| Late or missing data | The source had a slow night; the 2am pickup ran on time and found nothing. Data arrives at 10am into a pipeline that stopped waiting at 2. | STALE (or partial) |
| Expired credentials | The password or key the pipeline uses to log into a source rotates or expires. The least glamorous failure in data, and a regular one. | STALE |
| Bad deploy | A human ships a change to the cleanup logic that runs green and computes nonsense - divides by the wrong column, drops a filter. | WRONG |
Notice the pattern: the STALE causes are mostly operational (credentials, timing) and the WRONG causes are mostly change - someone, somewhere, changed something. That is why data teams get nervous on release days and why "what changed yesterday?" is the first question in every incident.
Self-studyWhy prevention alone cannot win3 min▶
Two of the four causes - schema changes and late data - originate in systems the data team does not control. The checkout app belongs to the product team; the ad data belongs to Google. You cannot code-review another company's release schedule. Since the causes cannot all be prevented, mature teams shift the goal: minimize time-to-notice, not incident count. That reframing is the entire logic of part 3, and it mirrors how site-reliability teams think about servers - uptime through fast detection and recovery, not through the fantasy of nothing ever breaking.
Two safeguards, and what each one misses 7 min live
The industry's two standard safeguards map one-to-one onto the two faces. Neither replaces the other - and knowing which catches what is the most practical thing this course teaches.
LiveFreshness checks - the timestamp with an alarm3 min▶
A freshness check is a promise with a deadline: "orders data should never be more than 12 hours old - warn at 12, alarm at 24." The transform tool dbt ships this as a built-in feature (source freshness), and every monitoring product has an equivalent. It is cheap, it is one line of configuration, and it turns the silent STALE face into a loud one.
- Catches: expired credentials, dead syncs, late data, the success-with-zero-rows trap - anything that stops new data arriving.
- Misses: everything that arrives on time and damaged. The $9,120 morning sails straight past a freshness check, timestamp gleaming.
The dashboard-reader's version of this safeguard costs nothing: look at the "last updated" stamp before you trust a number. If your dashboard does not show one, that is a feature request worth making this week.
LiveData tests - tasting the batch before it ships4 min▶
A data test is an automatic check on the data's content, run inside the pipeline before results are published. The standard tool (dbt) ships exactly four built-in ones, and they are beautifully mundane:
| Test | Plain English | Daybreak example |
|---|---|---|
| not_null | this field must never be empty | every order has a price |
| unique | no duplicates allowed here | order #8412 appears exactly once |
| accepted_values | only these values are legal | order status is placed, shipped, or refunded - nothing else |
| relationships | this must point at something real | every order belongs to a customer that exists |
When a test fails, the pipeline can block publishing - the dashboard keeps yesterday's honest number instead of today's damaged one, and the on-call engineer gets paged. Trading "wrong" for "stale, loudly" is a wonderful trade.
- Catches: the text-price schema change (price fails not_null when parsing dies), duplicate-counting bugs, refund statuses nobody expected.
- Misses: whatever nobody thought to test - and definition drift. If two teams count "revenue" differently, every test passes and the numbers still disagree. Tests check data against rules; they cannot invent the rules.
Self-studyThe third safeguard: someone actually on call2 min▶
Checks that fire alerts nobody owns are decoration. The unglamorous third safeguard is a routing rule: a named person sees pipeline alerts the same morning. On the floor below, the on-call lever changes nothing about detection - it changes recovery, roughly "fixed in 3 hours" versus "found by the CFO on Thursday". In the 2022 survey numbers, that is the difference between the 4-hour detection average and the multi-day tail. When you ask your data team about safeguards, the third question after freshness and tests is simply: "who gets woken up?"
The pipeline floor 🔧 ★ 20 min · everyone breaks a pipeline
This is Daybreak's pipeline, and you are allowed - encouraged - to break it. Every failure from part 2 is wired into a stage. Break one, run the day, and read the dashboard the way an executive would. Then score a full scripted week and watch what each safeguard is worth. The numbers are scripted from real incident patterns; the lesson is not.
Run a healthy day first. No breaks, no safeguards. The dashboard shows $18,240, fresh at 6:05am. This is what "normal" looks like - remember its face.
Break Ingest (the expired credential), run the day. The dashboard shows $17,980 and looks utterly normal. That is the STALE face: yesterday's number wearing today's suit. Now switch the freshness badge ON and rerun - same number, but now it confesses.
Repair Ingest, break Checkout (prices-as-text), run the day. $9,120 - about half of reality, timestamp fresh. Turn the freshness badge on. Rerun. It sails straight through: freshness cannot see wrong. Feel free to be slightly horrified.
Turn data tests ON, rerun the same break. Publish blocked - the dashboard refuses to show the damaged number, and the story panel tells you what fired. Try the other WRONG breaks too (Transform's bad SQL: $3,648; Metrics' definition drift: $21,890 - note that one counts refunds as revenue, which no test catches).
Switch to "Score a full week." Three scripted incidents. With no safeguards: 0 of 3 caught, 6 days trusting a bad number. Freshness on: 2 of 3, 2 blind days. Both on: 3 of 3 caught same-day, 0 blind days. That ladder - 0, 2, 3 - is the business case for data quality tooling in one widget.
Last: toggle the on-call alert on any broken day. Watch the recovery line change from "the CFO, on Thursday" to "fixed before the 9am meeting". Detection plus ownership - that is the whole playbook.
Homework 📝 ◐ 20 minutes
- Find your blind spot: for the most important dashboard you use, answer the three safeguard questions - is there a freshness stamp? do the core tables have tests? who gets alerted? "I do not know" is a fine answer; it becomes a question for session 6.
- Recall one incident you have lived through (a number that was off, a report that stopped updating). Classify it: stale or wrong? Which of the four causes fits best? Which safeguard would have caught it?
- On the floor, find the one break that both safeguards miss (part 3 named it). Write one sentence on why no automatic check can catch it.
Three questions before you go 🎯 ◐ 90 seconds
1 · Daybreak's dashboard shows $9,120 with a fresh 6:05am timestamp on a day that really earned $18,240. Which safeguard can catch this?
The data ARRIVED on time - freshness sees a gleaming timestamp and waves it through. Only a test on the content (not_null on price, a row-count check) sees the damage and blocks publish.
2 · The most common cause of pipeline failures in industry post-mortems is...
Schema changes top the lists - the source app renames a field or changes a type, no malice involved, and the cargo no longer fits the shelf. It is also the cause the data team can least prevent, since the source belongs to another team.
3 · Mature data teams aim primarily to...
Two of the four causes live in systems the data team does not control. Since prevention cannot be total, the winning metric is detection speed - the week-mode ladder (0 caught, then 2, then 3 of 3) is exactly this idea made visible.
What this session covered from the official universe
Certificates, videos, and graded assessments stay with the official providers - this course teaches the working core in plain English.