Learn Data Engineering with Phoebe

Two tracks from raw source to a warehouse-ready pipeline

A leader track that teaches you what data engineering actually is, where data is born, why real-time costs what it costs, and how to judge build-vs-buy without writing a line of code - and a builder track that constructs the pipeline feeding Daybreak's warehouse: connect and extract, ingestion patterns, file and table formats, batch and streaming transforms, storage engineering, and the reliability seam that hands off to operations. Every builder page runs DuckDB - a real analytical engine - live in your browser: it reads CSV, JSON, and Parquet, converts formats, partitions, and moves data over 300,000 rows. No install, no cluster, no bill.

16sessions
2tracks
10live-playground sessions
300,000rows in your browser

The leader track 🤝 · for C-level, managers & curious minds · 6 x 45 min · no code to write

Thinking mode: what data engineering is and is not, why you never own the source, why real-time is a cost not a feature, how to read the modern stack as decisions rather than logos, and how to fund a DE team crawl-walk-run - each session ends with questions to ask your data team.

🔧 Leader 1 · easy

What data engineering is

The plumbing under every dashboard and model: the five-stage lifecycle, and where DE ends and warehousing, DataOps, and SQL begin.

▶ Leaders start here 🔌 Leader 2 · easy

Source systems

Where data is born - app databases, APIs, files, event streams - and the hard truth that you rarely own the systems you depend on.

Meet the source zoo ⏱️ Leader 3 · medium

Batch vs streaming

Two clocks for moving data, when each is worth it, and why "real time" is a bill you sign up for - not a free upgrade.

Two-clocks exercise 🛒 Leader 4 · medium

Build vs buy the stack

The modern data stack as decisions, not logos - Fivetran, dbt, Airflow, Spark, Kafka - and when a managed tool beats writing your own.

Stack-map exercise 🗄️ Leader 5 · medium

Storage engineering

File and table formats, hot vs cold tiers, and why the shape you store data in quietly decides what it costs and how fast it reads.

Format & tier session 🧭 Leader 6 · medium

The DE org and roadmap

The roles on a data engineering team, how they hand off to warehouse and DataOps, and a crawl-walk-run plan you can actually fund.

Roadmap exercise

The builder track 🛠️ · for practitioners · 10 sessions x 45 min (b10: 60) · live DuckDB in your browser

One running project: Daybreak - the coffee-subscription brand from learn-sql-with-phoebe and learn-data-warehouse-with-phoebe. It has a warehouse now, but nothing reliably feeds it - so its OLTP database plays the source system, and session by session you build the pipeline that ingests, moves, reshapes, and serves its data. Every page runs real DuckDB via WebAssembly - editable code over 300,000 rows, reading CSV, JSON, and Parquet, converting formats and partitioning like a production pipeline. Spark, Kafka, and Airflow appear as honest read-only snippets - they cannot run in a browser. The engine loads once from a CDN (about 8 MB, cached); everything after that is offline.

🚰 Builder 1 · easy

The lifecycle in one pipeline

Run the whole thing once: pull Daybreak's raw data, land it, reshape it, serve it - the five-stage lifecycle in a single live script.

▶ Builders start here 📥 Builder 2 · medium

Connect and extract

Meet the source zoo and pull data out of it: read Daybreak's OLTP tables and extract them cleanly to files you own.

Extract to file 🔁 Builder 3 · medium

Ingestion patterns

Full vs incremental loads and the watermark - copy only what changed since last time, and prove a re-run does not double-count.

Full vs incremental 🧱 Builder 4 · medium

File and table formats

CSV, JSON, and Parquet round trips live in the browser, plus what table formats add on top - and why the format is a real decision.

Convert the formats 🔄 Builder 5 · medium

Batch transformation

ELT in practice: clean, cast, and aggregate Daybreak's landed data, and build a transform that is idempotent - safe to run twice.

Build the ELT 🌊 Builder 6 · medium

Streaming and CDC

Read an event log instead of a table, and capture change data - the streaming mindset, with Kafka shown as an honest snippet.

Read the event log 🗜️ Builder 7 · hands-on

Storage engineering

Partition Daybreak's data by date, measure the scan you save, and map hot vs cold tiers to what they cost in the cloud.

Partition live 🕸️ Builder 8 · hands-on

Distributed processing

What the shuffle really costs, why single-node often wins, and when data is big enough to justify Spark - shown as a snippet.

Shuffle vs single-node 🛡️ Builder 9 · hardest

Reliability seams

Catch schema drift before it breaks the warehouse, write a data contract, and build the seam where DE hands off to DataOps.

Catch the drift 🏁 Builder 10 · hardest · 60 min

Capstone: the whole pipeline

Assemble the full pipeline end to end - source to extract to transform to serve - and hand clean data to Daybreak's warehouse door.

End-to-end build
easy - everyone medium hands-on hardest

This is the deng-bucket capstone: it builds the pipeline that feeds the warehouse. Best after learn-sql-with-phoebe (the query language) and alongside learn-data-warehouse-with-phoebe (models what you serve). Once your pipeline runs, learn-dataops-with-phoebe schedules and monitors what you build.

Built from the DeepLearning.AI Data Engineering Professional Certificate (Joe Reis) and Fundamentals of Data Engineering (Reis & Housley) - certificates stay official.

Choose your path 🗺️

Two doors, one pipeline. Leaders never write code; builders never sit through theory they have not just run themselves.

🤝 The leader path (judge & fund) a1→ a2→ a3→ a4→ a5→ a6
🛠️ The builder path (build Daybreak's) b1→ b2→ b3→ b4→ b5→ b6→ b7→ b8→ b9→ b10
🚀 The full journey (lead AND build) a1-a6→ b1-b10

The knowledge map 🧠

Both tracks at a glance - hover a session to spotlight its concepts, click any node to jump in.