Why this session exists
Nobody ships coffee beans straight from the truck to a customer's cup. And nobody should ship raw data straight to a dashboard - it arrives full of duplicates, test orders, typos, and refunds living in another system. Yet the dashboard number looks clean, confident, and singular. The gap between those two states is the cleanup room, and it is where trust is manufactured. Understand this room and you will never again hear "the data needs cleaning first" as an excuse - you will hear it as the actual work.
The shelf, briefly 5 min live
Before we clean anything, thirty seconds on the room itself. A data warehouse is one big shelf everything lands on - a database built for reading and analyzing lots of history at once, not for running the app. The checkout database answers "process this one order, right now"; the warehouse answers "sum every order this year, by month". Same data, opposite jobs.
LiveWarehouse, lake, and where the depth lives5 min▶
One honest paragraph on a distinction you will overhear: a warehouse is a structured shelf - data arrives in tidy rows and columns, ready to be queried. A data lake is an everything-bucket - files of any shape (logs, images, exports) dumped cheaply now, organized later, if ever. Modern platforms increasingly blur the two, but the instinct holds: shelf means structure up front, bucket means structure deferred. For this course, Daybreak has a warehouse, and that is all we need.
And an honest pointer: this course walks past the warehouse building; it does not go inside. If you want the interior tour - how the shelf is organized, real SQL, why some questions run fast and others crawl - that is the sister course learn-data-warehouse-with-phoebe, built on the same Daybreak case. Same company, same order #8412, one floor deeper. We refuse to pretend five minutes covers what that course takes six sessions to teach.
Keep the raw copy, clean in layers 8 min live
The best habit in the building sounds wasteful the first time you hear it: keep an untouched raw copy of everything, forever. Then clean in stages - raw to clean to ready - so every number on the dashboard can be traced backward through the layers to the evidence. Here is order #8412 wearing all three outfits.
LiveWhy the untouched copy is the best habit in the building5 min▶
Three reasons teams keep raw data exactly as it arrived, typos and all:
- Replay after bugs. Suppose a cleaning rule turns out wrong - it was deleting real orders it mistook for tests. With the raw layer intact, you fix the rule and re-run the cleanup over history; every past number heals. Without it, the mistake is permanent, because the evidence was edited.
- An audit trail. "Prove where this figure came from" - an auditor, a payment partner, a skeptical board member. The raw layer is the receipt: here is what the source said, here are the written rules, here is the result.
- It costs almost nothing. Cloud storage is cheap enough that keeping everything raw is a rounding error next to the cost of ever being unable to answer the first two points.
The layer names, decoded: you may hear bronze / silver / gold - that is Databricks' terminology for exactly this layering. Bronze is data as-is from the source, silver is cleansed and merged, gold is consumption-ready for dashboards and reports. Other teams say raw / staging / marts. The outfit names vary; the wardrobe is the same.
| Layer | Also heard as | Order #8412 looks like |
|---|---|---|
| Raw | bronze, landing | two rows (the duplicate click intact), the QA test order sitting nearby |
| Clean | silver, staging | one trustworthy row, refund joined in from Stripe |
| Ready | gold, marts | $42 inside daily_revenue: Sunday, $18,240, one row per day |
Self-studyWhy not just clean it before storing it?3 min▶
A fair question sits under this whole design: if raw data is unusable, why shelve it at all? Why not clean it on the truck, before it ever lands? That is not a naive idea - it is how the entire industry worked for decades. Cleaning before storing saves shelf space, and when shelf space is the expensive thing, you save it.
So what changed? The price tag. That story - and how it quietly flipped the order of operations everywhere - is part 3, and it explains more overheard jargon per minute than anything else in this course.
The ETL vs ELT story 7 min live
Two acronyms, one letter swapped, and behind the swap a genuine turning point in how companies handle data. ETL: extract, transform, load - clean the data before shelving it. ELT: extract, load, transform - shelve it raw first, clean it inside the warehouse. The letters moved because a price collapsed.
LiveThe price collapse that flipped an industry4 min▶
In the old world, storage was the expensive thing - so you cleaned data on the way in and shelved only the polished result. Sensible economics, with a hidden cost: no raw copy meant no replay, no audit trail, and cleaning logic buried inside specialized tools few people could read.
Then cloud storage collapsed in price - from roughly a million dollars per gigabyte in the early days to cents per gigabyte today. When the shelf becomes nearly free, the smart move inverts: shelve everything raw first, then clean inside the warehouse, where there is room to keep every layer. The consequences are exactly the habits from part 2:
- The raw layer becomes affordable, so it becomes standard.
- Mistakes become replayable - fix the rule, re-run history.
- Cleaning rules move into the warehouse as SQL - written down where anyone on the team can read and review them, instead of locked inside a tool.
| ETL - the old road | ELT - the new road | |
|---|---|---|
| Cleaning happens | before the shelf, in a specialized tool | inside the warehouse, written as SQL |
| Raw copy | discarded - the shelf was too expensive | kept forever - the shelf costs cents |
| A wrong rule is | permanent - the evidence was edited | replayable - fix it, re-run history |
| Who can review rules | tool specialists | anyone on the team who reads SQL |
Self-studydbt, decoded: the cleanup room's rulebook3 min▶
ELT created a new question: if cleaning now happens inside the warehouse, who keeps the cleaning rules organized? The answer you will overhear constantly is dbt. Decoded: dbt is the cleanup room's rulebook - a way to write every cleaning rule down, keep every version, and let anyone review a change before it takes effect. "Collapse duplicate clicks", "orders from QA accounts do not count", "join refunds from Stripe" - each rule written out, versioned, reviewable.
Hear the echo from session 1: the Friday spreadsheet died because the rules lived in one person's head. dbt is the industry's answer at pipeline scale - rules living in a shared, reviewable rulebook instead of in Sarah. It is not a database and it does not store data; it is the discipline layer for the transform box. Writing actual dbt models is code and stays with dbt's own free Fundamentals course - but you now know exactly what the name means in a meeting.
Spot the cleaning rules 🧼 ★ 18 min · everyone works
Notebook exercise. In session 1 you traced one of your own numbers back through the six stages. Today you interrogate its cleanup: for that number to be true, a set of cleaning rules MUST have run - and you can deduce most of them from your chair, without asking anyone.
Take your traced number from session 1's napkin - weekly sales, sign-ups, whatever you picked. Write it at the top of the page.
List five things that must have been cleaned for it to be true. Steal from Daybreak's list: duplicates collapsed, test orders removed, refunds subtracted, currencies converted, late-arriving orders assigned to the right day. Which five apply to yours?
Guess which layer each fix lives in. Raw is untouched by definition, so every fix is clean-layer or ready-layer. Rule of thumb: fixing individual rows (dedupe, remove tests) is clean; shaping and summing into the final metric (one row per day) is ready.
Write ONE rule as a plain sentence. Daybreak's example: "orders from @daybreak.com emails are test orders and do not count." No code - if you can say the sentence, someone can write the rule.
Find the sentence's soft spot. Every rule has an edge case: what about an employee who genuinely buys a gift bundle with their work email? Note one way your rule could quietly miscount - you have just done a data-quality review.
Mark what would break silently. If one of your five fixes stopped running tomorrow, would anyone notice, or would the dashboard just smile with a slightly wrong number? Circle the silent ones - session 5 is about exactly those.
Homework 📝 ◐ 20 minutes
- Write out all five of your cleaning rules as plain sentences, like the demo's one. Keep the page - session 6 turns it into questions for your data team, who can tell you which rules actually exist.
- Ask one person (or check one dashboard tooltip) whether your company's numbers keep a raw layer - "if a cleaning rule turned out wrong, could we recompute last quarter?" The answer is revealing either way.
- Listen for the layer words this week: bronze, silver, gold, staging, marts, raw. You now know they all describe the same wardrobe - note which dialect your company speaks.
Three questions before you go 🎯 ◐ 90 seconds
1 · The main reason teams keep an untouched raw copy of all incoming data is...
Raw is not more accurate - it is full of duplicates and test orders. Its value is being the evidence: fix a wrong rule and re-run history over the raw layer, and show auditors exactly what the source said. Cheap storage made this habit affordable.
2 · What flipped the industry from ETL (clean before shelving) to ELT (shelve raw, clean inside)?
ETL made sense when the shelf was the expensive thing. Once storage cost cents per gigabyte, shelving everything raw first became affordable - and bought replayability, audit trails, and cleaning rules written as reviewable SQL. Economics moved the letter.
3 · dbt is best described as...
dbt stores no data and displays nothing. It organizes the transform box: every cleaning rule written out, every version kept, every change reviewable - the industry's answer to rules living in one person's head, which is exactly how session 1's spreadsheet era died.
What this session covered from the official universe
Certificates, videos, and graded assessments stay with the official providers - this course teaches the working core in plain English.