learn-data-warehouse-with-phoebe / Builder session 1 of 10
Learn Data Warehouse with Phoebe · Builder track · Session 1 of 10

Row store meets column store

Daybreak - the coffee-subscription brand from the SQL course - grew up. Its app database now groans every time someone opens a dashboard. Over ten sessions you will build Daybreak's first real data warehouse, and every page runs DuckDB: an actual columnar OLAP engine, live in your browser. Tonight you learn WHY warehouses exist - by feeling the difference, not reading about it.

🟢 Builder track Practitioners: analysts · DE · DS · PMs Runs in your browser · DuckDB Start here
0-3 · Welcome 3-18 · OLTP vs OLAP & columnar 18-42 · Build-along: feel the difference 42-45 · Q&A
Part 0

How this track works

Ten sessions, one running project: Daybreak's six OLTP tables (customers, products, orders, order items, subscriptions, events) play the source system, raw and messy - dates stored as text and all. Session by session you will stage it (b2), reshape it into a star schema (b3-b4), track history with slowly changing dimensions (b5), load it properly (b6), serve it to the business (b7), make it fast and cheap (b8), meet the lakehouse (b9), and assemble the whole warehouse end to end (b10). If you took learn-sql-with-phoebe, this is the same database - now you build what b10 only hinted at.

Live - presented in session Self-study - read after class ▶ Live warehouse - editable & runnable Official sources covered
★ What you walk out with today The one distinction the whole field rests on (OLTP vs OLAP), a gut feel for why column storage crushes analytics workloads, and a browser tab running a real warehouse engine against 300,000 rows - no install, no cloud account, no bill.
Part 1 · covers IBM DW Fundamentals M1, 365DS "Foundations"

OLTP vs OLAP: two jobs, two shapes 7 min live

Daybreak's app database answers thousands of tiny questions a day: "insert this order", "update this address", "show Ava her subscription". Each touches one row. The founder's questions are the opposite shape: "revenue by month by channel, all time" touches every row but only three columns. One database serving both is the design mistake warehouses exist to fix.

OLTP · the app database many tiny transactions, all day insert order update address fetch 1 row touches: ONE row at a time · needs: speed + safety per write shape: narrow and deep - the whole row, right now OLAP · the warehouse few huge questions, whole history revenue by month churn by cohort touches: EVERY row, few columns · needs: scan speed shape: wide and shallow - three columns, all of history copy Run analytics on the app database and both sides lose: · the founder's big scan locks tables the checkout flow needs - carts hang · the app's row-by-row layout makes the big scan painfully slow anyway A warehouse is a COPY of your data, restructured for the second shape of question.
🔍 Click to zoom - two workloads that want opposite database designs
LiveThe day Daybreak's dashboard froze the checkout3 min

Every data team has this origin story. Daybreak's version: marketing opened a "lifetime revenue by cohort" dashboard at 9am on launch day. The query scanned every order ever placed - on the same database processing live checkouts. Carts spun. Support lit up. Engineering's fix was not "optimize the query"; it was "get analytics off the app database". That sentence is the birth certificate of every data warehouse.

  • OLTP (online transaction processing): the app's workload. Many small reads/writes, one row at a time, correctness per transaction is sacred.
  • OLAP (online analytical processing): the analyst's workload. Few queries, each scanning millions of rows but touching few columns.
  • The fix is a copy: extract data out of the app database, restructure it for scans, and point every dashboard at the copy. The copy is the warehouse.
Real world

Why "just add an index" fails. Indexes accelerate finding few rows. Analytics reads all rows. There is no index for "scan everything and add it up" - the table's physical layout itself has to change. That layout change is Part 2.

Self-studyWhere the copy lives: warehouse, mart, lake3 min read

Three cousins you will hear named together (IBM's course opens with them; a4 in the leader track compares them for executives):

  • Data warehouse: the structured, modeled copy - cleaned, typed, shaped for analytics. This course.
  • Data mart: a slice of the warehouse for one team (finance mart, marketing mart). Session b7 builds them.
  • Data lake: cheap storage for raw files of any shape - and the lakehouse puts warehouse-style tables directly on lake files. Session b9 is that story, and you will query real Parquet files when you get there.
Part 2 · covers 365DS "Modern architectures", Snowflake micro-partition ideas land in b8

Why column storage wins analytics 8 min live

The single deepest idea in warehousing: change how bytes sit on disk. A row store keeps each record's fields together - perfect for "fetch order 1017, all of it". A column store keeps each column's values together - perfect for "sum line_amount across every order ever". Same data, opposite physical layouts, and the layout decides which questions are fast.

Row store: records live together id 1001 Jan 06 41.00 id 1002 Jan 11 18.00 id 1003 Jan 14 50.00 "SUM(amount)" must read EVERY block - ids and dates come along for the ride, wasted. Column store: columns live together id 1001 id 1002 id 1003 Jan 06 Jan 11 Jan 14 41.00 18.00 ... "SUM(amount)" reads ONLY the amber blocks. Two-thirds of the disk never gets touched. Bonus 1 · compression: a column of similar values (all dates, all prices) squeezes far smaller than mixed rows. Bonus 2 · vectorized execution: the engine crunches one column in CPU-friendly batches, not row by row. Snowflake, BigQuery, Redshift, DuckDB - every modern warehouse engine is columnar. This is the reason.
🔍 Click to zoom - same three orders, two physical layouts, opposite strengths
LiveMeet DuckDB - the warehouse in your browser3 min

DuckDB is a real columnar OLAP engine - the same breed as Snowflake or BigQuery, minus the cloud. It compiles to WebAssembly, which is why every editor on these pages runs an honest-to-goodness warehouse engine inside your browser tab. First Run downloads the engine once (~8 MB, then cached); after that everything is local and instant.

  • The SQL you know transfers: SELECT, JOIN, GROUP BY, window functions - all identical to what you learned in the SQL course.
  • What is new here: columnar layout, CREATE TABLE AS pipelines, MERGE, Parquet - warehouse muscles SQLite never had.
  • Honest note: the first Run on each page needs network for the one-time engine download. Everything after runs offline.
SELECT count(*) AS orders,
       min(order_date) AS first_order,
       max(order_date) AS last_order
FROM orders;
LiveSpot the mess: why this source needs a warehouse3 min

Look closely at what the app database actually hands us. Run this and check the typeof column:

SELECT order_id, order_date,
       typeof(order_date) AS stored_as
FROM orders
LIMIT 4;

Dates stored as text. Revenue split across two tables. Statuses like refunded mixed into every count. This is normal, healthy OLTP design - and exactly what analytics cannot live with. Session b2 builds the staging layer that fixes it; tonight we just feel the pain.

Part 3 · your ten-session build

The warehouse you are about to build 3 min live

Here is the whole track on one strip. Every session adds one layer to Daybreak's warehouse, and each layer exists to answer a business question the raw source fumbles.

SessionYou buildDaybreak gets
b2staging layertyped, trusted copies of the source
b3-b4star schema: facts + dimensionsquestions answered in one obvious join
b5slowly changing dimensionshistory that does not rewrite itself
b6loading patterns: incremental + MERGEtonight's orders in tomorrow's warehouse, safely
b7marts, views, rollupseach team its own clean slice
b8performance + cost tuningfast dashboards, small bills
b9lakehouse: Parquet + external tablesone foot in the modern open stack
b10capstone: the whole pipelinesource to staging to star to served answer
Demo 1 of 2

Feel columnar speed on 300,000 rows ★ 12 min · everyone builds

Thirty-three orders prove nothing. So this playground generates 300,000 synthetic order lines (table big_orders) before your query runs - then lets a columnar engine loose on them. Watch the millisecond counter under each result.

Run the aggregate below as-is. Note the time: a full scan of 300k rows, grouped and summed, in your browser tab.

Add status to the GROUP BY (both lines). Still instant - one more column read, the rest untouched.

Now try SELECT * FROM big_orders LIMIT 5; - fetching whole rows is the one thing this layout does NOT love. On 300k rows it survives; on 300 billion you would pay for it.

Say the lesson out loud: warehouses scan columns, not rows. Ask for the columns you need, never SELECT *.

SELECT channel,
       count(*) AS orders,
       ROUND(SUM(quantity * unit_price), 0) AS revenue
FROM big_orders
GROUP BY channel
ORDER BY revenue DESC;
Real world

The bill version of this lesson. Cloud warehouses charge by data scanned. SELECT * on a wide table reads every column and bills you for all of them; naming three columns reads three. Analysts who learned on row stores drag the * habit into BigQuery and the invoice notices before they do. Session b8 turns this into a cost playbook.

Demo 2 of 2

Your turn: three founder questions ★ 10 min · build your own

Same rules as the SQL course: write it, run it, read the error, fix it, run again. All three run against the fresh 300k-row big_orders table.

LiveQ1 · Monthly revenue: which month was biggest?4 min
SELECT strftime(order_date, '%Y-%m') AS month,
       ROUND(SUM(quantity * unit_price), 0) AS revenue
FROM big_orders
GROUP BY month
ORDER BY month;
LiveQ2 · What share of orders get refunded, by channel?3 min
SELECT channel,
       count(*) AS orders,
       ROUND(AVG(CASE WHEN status = 'refunded' THEN 1.0 ELSE 0 END) * 100, 2) AS refund_pct
FROM big_orders
GROUP BY channel;
Self-studyQ3 · Weekend vs weekday: when do people order?3 min

A taste of what dim_date will make trivial in b4 - for now, derive it by hand:

SELECT CASE WHEN dayofweek(order_date) IN (0, 6)
            THEN 'weekend' ELSE 'weekday' END AS day_type,
       count(*) AS orders
FROM big_orders
GROUP BY day_type;
Homework

Try it yourself - this week ◐ 20-30 min total

Source material

Official sources covered

This track teaches the working core of the major data-warehousing curricula, run on a live engine instead of slides. Certificates, graded labs, and videos stay on the official platforms. This page covers:

IBM Data Warehouse Fundamentals (Coursera) - Module 1: warehouses, marts, lakesPart 1 + self-study card · systems tour continues in a4/b9
365DS Intro to Data Warehousing - Foundations & core componentsParts 1-3 · OLTP/OLAP split and why the copy exists
DeepLearning.AI Data Engineering C4 (Joe Reis) - serving framingPart 1 · "analytics is a serving problem"; modeling depth lands in b3-b5
365DS - Performance optimization in queryingDemo 1 · columnar scan intuition; the full tuning kit is b8
Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · Marketing's dashboard keeps timing out and checkout slows to a crawl whenever it loads. The textbook fix is...

Indexes speed up finding few rows; analytics reads all rows. The workloads want opposite designs, so you separate them - the warehouse is that copy.

2 · Why does a column store make SUM(line_amount) over 300k rows so fast?

Columnar layout stores each column's values together, so an aggregate touches only the columns it names - plus similar values compress better and crunch in batches.

3 · Which question is OLAP-shaped?

OLAP questions scan many rows but few columns (all history, two fields). One-row lookups and writes are OLTP - the app database's home turf.

Builder session 1 cheat sheet · pin this

OLTPApp workload: many tiny transactions, one row each, correctness per write. Keep it away from dashboards.
OLAPAnalytics workload: few queries, all rows, few columns. What warehouses are built for.
A warehouse isa restructured COPY of source data, optimized for scans - not a faster app database.
Columnar storageEach column's values stored together: aggregates read only named columns, compress better, vectorize.
Never SELECT *In a warehouse you pay (time and money) per column scanned. Name what you need.
DuckDBReal columnar OLAP engine running in your browser here. Same SQL you already know.
The source is messyText dates, split revenue, mixed statuses - healthy OLTP, useless analytics. b2 fixes it.
Running projectDaybreak's warehouse, built layer by layer through b10. Next: the staging layer.