Why this session exists
Everyone trusts the dashboard. Almost nobody can say how a number gets onto it - and that gap is where bad decisions are born. A data pipeline is not a technology; it is a supply chain for facts, and like any supply chain it has trucks, warehouses, quality checks, and days when a shipment silently goes missing. Once you can picture the road, every conversation with your data team gets easier, and every suspicious number gets easier to question.
What a data pipeline actually is 7 min live
A data pipeline is the automated road a piece of data travels from where it is born to where a human uses it. Six stages, always the same shape: something creates data, something copies it out, something stores it, something cleans it, something turns it into a metric, and something displays it. The industry has a formal name for this spine - the data engineering lifecycle - but the napkin version below is the one you will actually use.
LiveThe supply-chain analogy that makes it stick3 min▶
Think of a physical supply chain: a farm grows beans, a truck collects them, a warehouse shelves them, a roastery processes them, a barista turns them into a drink, and a customer tastes the result. A data pipeline is exactly this, for facts:
- The farm is a source system - the checkout app, the ad platform, the support tool. Data is a by-product of business happening.
- The truck is ingestion - a scheduled pickup that copies new data out, typically overnight.
- The warehouse is, literally, called a data warehouse. The industry gave up on metaphor here.
- The roastery is transformation - raw data is full of duplicates, test orders, and typos, and someone has to clean it before anyone drinks it.
- The tasting is serving - the dashboard, the weekly email, the alert, increasingly the AI model.
The analogy earns its keep on bad days: when a truck breaks down, shelves go empty and everyone notices. When a pipeline breaks, the dashboard keeps smiling and shows old numbers. Session 5 is entirely about that difference.
LiveWhy "pipeline" and not just "a report someone makes"3 min▶
The word pipeline signals two promises that a hand-made report never makes:
- It runs by itself. Nobody clicks anything at 2am. The pickup, the cleaning, and the refresh happen on a schedule - which is why the Monday dashboard exists before anyone is awake.
- It runs the same way every time. The same cleaning rules, the same definition of revenue, every single day. A human rebuilding a spreadsheet applies slightly different judgment every Friday, and nobody can say which Friday was right.
Automation is not about saving effort first - it is about making the number reproducible. "Where did this figure come from?" has an answer when a pipeline made it. It has a shrug when a spreadsheet did.
Self-studyThe two-word version of every stage4 min▶
For your napkin: Source = business happens. Ingest = copy it out. Store = one big shelf. Transform = raw to ready. Metrics = agree on definitions. Serve = put it in front of a human (or a model).
| Stage | Plain-English job | Tool names you may have overheard |
|---|---|---|
| Source | Where data is born | your app's database, Stripe, Shopify, Google Ads |
| Ingest | The scheduled pickup | Fivetran, Airbyte |
| Store | The shelf everything lands on | Snowflake, BigQuery, Databricks |
| Transform | The cleanup room | dbt |
| Metrics + serve | Definitions, then the screen | Looker, Tableau, Power BI |
Do not memorize the right column - session 6 hands you a full decoder table. The left two columns are the actual lesson.
Meet order #8412 ☕ 8 min live
Our running case all course: Daybreak, a coffee-subscription brand. On Sunday at 11:41pm, a customer buys a $42 gift bundle - order #8412. Here is that single order's overnight journey to the Monday 9am revenue dashboard, with clock times. Every session from now on zooms into one or two of these stops.
LiveWhat each hand-off quietly does to the order4 min▶
Order #8412 does not just move - it changes shape at every stop, and each change is a decision someone made:
- At pickup (2am): only new and changed rows are copied - the truck does not re-haul the whole history every night. Deciding what counts as "changed" is a real design choice.
- On the raw shelf (2:15am): the order is stored exactly as the app wrote it, typos and all. Keeping an untouched copy sounds wasteful; session 3 explains why it is the single best habit in the building.
- In the cleanup room (4:30am): the customer's duplicate click is collapsed, the QA team's test orders vanish, and the refund from a different system is matched to it. The order is now opinionated.
- At counting (5:45am): someone, at some point, decided whether gift bundles count as "subscription revenue" or "one-time revenue". Order #8412 inherits that decision forever.
Self-studyWhy Daybreak, and where you have seen it before2 min▶
Daybreak is the running company across this shelf's data-engineering courses - the sister course learn-data-warehouse-with-phoebe opens with Daybreak's app database meeting its first real dashboard, badly. This course walks the whole conveyor belt in plain English; that one moves into the Store box with real SQL. Same company, deeper floor. If this course leaves you wanting the technical layer, that is your next door.
The spreadsheet era, and why it always ends 7 min live
No company starts with a pipeline. Every company starts with a smart person, an export button, and a spreadsheet - and it genuinely works, for a while. The pipeline gets built the week the spreadsheet fails publicly. The failure is so predictable it deserves a diagram.
LiveThe four-step life of the Friday spreadsheet4 min▶
Daybreak's early days, and probably your company's too:
The hero era. An analyst exports orders every Friday, cleans them by hand, and emails "the numbers". Fast, free, and correct - because one careful person holds every rule in their head.
The copy era. The file gets forwarded, edited, and re-saved as revenue_FINAL_v3_realfinal.xlsx. Marketing's copy and finance's copy quietly diverge. Meetings start with ten minutes of "whose number is right?"
The absence. The analyst takes two weeks off. Either the numbers simply stop, or a stand-in rebuilds them slightly differently and nobody can say which version was right. Both outcomes shake trust.
The public failure. A number quoted upward - to a board, an investor, an audit - turns out wrong, and nobody can reconstruct how it was made. THIS is the week the pipeline project gets funded. Daybreak's version: a payment partner asked how refund figures were produced, and the honest answer was "Sarah makes them".
Notice what actually failed: not effort, not intelligence - reproducibility. The pipeline is the company's way of writing the rules down somewhere a vacation cannot erase.
LiveWhat a pipeline buys you (and what it does not)3 min▶
Honest ledger, both columns:
| A pipeline buys | A pipeline does NOT buy |
|---|---|
| The same rules applied every night, written down and reviewable | Correct rules - a wrong definition runs punctually forever |
| Numbers ready before anyone wakes up | Numbers anyone actually reads - adoption is a people problem |
| One shared source instead of forked copies | Agreement on what "revenue" means - humans still have to decide that |
| A place to attach quality checks (session 5) | Freedom from breakage - things WILL break; the win is noticing fast |
Self-studyWhen a spreadsheet is honestly fine2 min▶
Fairness demands the other direction too: a one-off analysis, a tiny team, a question you will ask exactly once - a spreadsheet is the right tool, and building a pipeline for it is procrastination wearing a hard hat. The signal to upgrade is repetition: the third time someone rebuilds the same numbers by hand, the rules deserve to live in a pipeline instead of a person.
Trace one of your own numbers 🧭 ★ 18 min · everyone works
No laptop needed - a notebook works. Pick one number you saw at work this week and walk it backward through the six stages. You will not know every answer, and the blanks ARE the exercise: each blank is a question your data team can answer in one sentence.
Pick your number. Something you actually saw: weekly sales, sign-ups, ticket count, campaign clicks. Write it down, with its dashboard's name.
Name the source. Where was this data born? Which app or system recorded the underlying events? (If the answer is "a spreadsheet someone fills in", write that honestly - it is a source too, just a fragile one.)
Guess the truck schedule. Does the dashboard change during the day (near-real-time pickup) or once each morning (an overnight batch)? Look for a "last updated" note anywhere on it - just looking for one teaches you session 5's lesson early.
Spot one cleaning rule. Name one thing that MUST have been filtered or fixed for your number to make sense - test accounts removed, refunds subtracted, currencies converted, duplicates collapsed. There is always at least one.
Question the definition. Write your number's definition as a sentence: "revenue = ___ minus ___, counted when ___". If you cannot finish the sentence, circle it - that circle is the most valuable thing you produce today.
Draw the napkin. Six boxes, arrows, your number's actual journey with everything you know filled in. Blanks stay blank. Keep it - session 6 turns it into a set of questions for your data team.
Homework 📝 ◐ 20 minutes
- Finish your napkin if the session ran short, and photograph it - you will re-draw it in session 6 and enjoy the difference.
- Find the "last updated" stamp on two dashboards you use. If one has none, note that - you have found a freshness blind spot two sessions before we name it.
- Collect one tool name you hear this week (in a meeting, a job post, a Slack channel) and park it. Session 6's decoder table will probably have it.
Three questions before you go 🎯 ◐ 90 seconds
1 · The six stages of the pipeline journey, in order, are...
Born, picked up, shelved, cleaned, counted, shown. The supply chain runs one direction - and every company's pipeline has this same spine, whatever tools sit on the boxes.
2 · The deepest thing the Friday-spreadsheet era lacks is...
The hero analyst works plenty hard and Excel is plenty fast. What dies is the ability to answer "where did this figure come from?" - the rules live in one person's head, and heads take vacations.
3 · Two departments report different numbers for "yesterday's revenue". The most common culprit is...
Usually both pipelines work fine - one team counts refunds and gift bundles one way, the other team another. The disagreement lives at stage five, in a human decision, not in the machinery.
What this session covered from the official universe
Certificates, videos, and graded assessments stay with the official providers - this course teaches the working core in plain English.