The lifecycle in one pipeline

Ten sessions, one running pipeline. Daybreak - the coffee-subscription brand from learn-sql and learn-data-warehouse - has a warehouse now, but nothing reliably feeds it. That is your job this track: build the pipeline that ingests Daybreak's source data, moves it, reshapes it, and lands it ready for the warehouse. You will meet source systems (b2), ingestion patterns (b3), file formats (b4), transformation compute (b5), streaming and CDC (b6), storage engineering (b7-b8), and the reliability seam that hands off to operations (b9), then assemble it all in a capstone (b10).

Joe Reis and Matt Housley framed data engineering as a lifecycle, not a tool list - and it stuck because it survives every tool that comes and goes. Five stages move data from where it is born to where it is used; a set of undercurrents runs beneath all of them. Learn this shape once and every tool you meet slots into it.

Data engineering, data warehousing, and DataOps blur together in job ads, but they are three distinct crafts. Knowing the lines keeps you from re-learning what a sibling course already teaches - and tells you exactly where to hand work off.

Real pipelines run on clusters. Learning them should not need one. DuckDB is a full analytical engine that runs in your browser tab - it reads files, converts formats, moves and transforms data exactly like a production pipeline, minus the ops. Where a real stage needs Spark or Kafka (which cannot run in a browser), you will see the real code as a read-only snippet, clearly marked.

SELECT order_id, customer_id, order_date,
       typeof(order_date) AS stored_as, status
FROM orders
LIMIT 5;

The whole lifecycle, end to end, in one runnable pipeline. You will pull from the source (generation), copy orders out to a Parquet file (ingestion + storage), reshape into a clean daily table (transform), and produce the served result a warehouse would load. Every stage is one step.

-- GENERATION + INGESTION: extract the source to a Parquet file
COPY (SELECT * FROM orders) TO 'raw_orders.parquet' (FORMAT PARQUET);

-- STORAGE: the data now lives in a file, not the source system
CREATE TABLE landed AS SELECT * FROM 'raw_orders.parquet';

-- TRANSFORM: clean + shape for analytics
CREATE TABLE daily_revenue AS
SELECT CAST(order_date AS DATE) AS day,
       count(*)                 AS orders,
       count(*) FILTER (WHERE status = 'completed') AS completed
FROM landed
GROUP BY day;

-- SERVING: the table the warehouse would load
SELECT * FROM daily_revenue ORDER BY day LIMIT 8;

Each editor starts from the raw source. Write the step, run it, read the error, fix it - that loop is the whole skill. All three run against Daybreak's source tables.

COPY (SELECT * FROM customers) TO 'raw_customers.parquet' (FORMAT PARQUET);
SELECT count(*) AS customers_landed FROM 'raw_customers.parquet';

SELECT o.channel,
       ROUND(SUM(oi.quantity * oi.unit_price), 2) AS revenue
FROM orders o
JOIN order_items oi ON o.order_id = oi.order_id
WHERE o.status = 'completed'
GROUP BY o.channel;

• Bookmark this page - the editor is your pipeline scratchpad, a real engine any time.
• Name the five lifecycle stages from memory, then map one pipeline you touch at work onto them. Which stage is missing or fragile?
• Take Demo 1's script and add a second transform table (revenue by customer plan). Serve both.
• For one data task on your plate, say the one-sentence test out loud: is it engineering, warehousing, or DataOps? Route it accordingly.
• Bring one source system you struggle to get data out of to session b2 - source systems is exactly what b2 is about.

This track teaches the working core of the DeepLearning.AI Data Engineering Professional Certificate (Joe Reis) and Reis & Housley's Fundamentals of Data Engineering, run on a live engine instead of slides. Certificates, AWS labs, and videos stay on the official platforms. This page covers: