Distributed processing

Seven sessions in, Daybreak's pipeline extracts from source (b2-b3), captures change (b6), lands data in Parquet, and stores it partitioned for fast scans (b4, b7). The data is shaped and the files are laid out well. Now the question every engineer eventually asks: what if it does not fit on one machine? That is when people reach for a cluster - and, too often, reach for it far too early. This session gives you the model and the judgement.

One machine has a fixed ceiling: so much memory, so much disk, so many cores. When a dataset exceeds that ceiling, you split it across many machines and let each chew on a slice. That is distributed processing, and almost every big-data engine follows the same three-move shape: map (each worker processes its slice), shuffle (workers exchange data so related rows meet), reduce (combine into the answer).

A cluster carries huge fixed overhead: spinning up machines, coordinating them, moving data on every shuffle, paying for idle nodes. A single-node engine has none of that. Modern machines are large - a laptop has 16-64 GB of memory, a cloud box hundreds - and engines like DuckDB are built to stream data that exceeds memory straight off disk. The result surprises people: single-node engines crush datasets up to ~100 GB and beyond, often faster than a cluster, because they skip the coordination tax entirely.

-- one machine, 300k rows, full aggregate - watch the milliseconds
SELECT count(*)                                   AS rows,
       round(sum(quantity * unit_price), 2)       AS total_revenue,
       round(avg(quantity * unit_price), 2)       AS avg_line_value
FROM big_orders;

Apache Spark is the default distributed engine. Its core ideas are worth knowing even if you rarely run it: an immutable dataset (RDD, or the friendlier DataFrame) split across the cluster, and lazy evaluation - transformations do not run when you write them; they build a plan, and only an action (write, collect, count) triggers the whole plan to execute. That lets Spark optimise the plan before moving a single byte.

One machine, no cluster, 300,000 rows. You will run three aggregates - by channel, by month, and a window - and read the millisecond timer under each result. The lesson lands in the numbers: at this scale, spinning up a cluster would be slower than the laptop you are on.

-- 1) revenue by channel
SELECT channel, count(*) AS orders,
       round(sum(quantity * unit_price), 2) AS revenue
FROM big_orders
GROUP BY channel;

-- 2) orders by month
SELECT strftime(order_date, '%Y-%m') AS month, count(*) AS orders
FROM big_orders
GROUP BY month
ORDER BY month;

-- 3) a window: running monthly revenue (this result renders)
WITH monthly AS (
  SELECT strftime(order_date, '%Y-%m') AS month,
         sum(quantity * unit_price)    AS rev
  FROM big_orders
  GROUP BY month
)
SELECT month,
       round(rev, 2)                                      AS revenue,
       round(sum(rev) OVER (ORDER BY month), 2)           AS running_revenue
FROM monthly
ORDER BY month;

Two hands-on queries and one thinking exercise. Each editor re-seeds the 300k-row table fresh. Write, run, read the timer - then decide whether any of this would justify a cluster.

SELECT channel,
       strftime(order_date, '%Y-%m')        AS month,
       count(*)                             AS orders,
       round(sum(quantity * unit_price), 2) AS revenue
FROM big_orders
GROUP BY channel, month
ORDER BY channel, month;

SELECT customer_id,
       channel,
       round(sum(quantity * unit_price), 2) AS spend,
       rank() OVER (PARTITION BY channel
                    ORDER BY sum(quantity * unit_price) DESC) AS rank_in_channel
FROM big_orders
GROUP BY customer_id, channel
ORDER BY channel, rank_in_channel
LIMIT 20;

• Take one pipeline at work you believe needs "big data" tooling. Find its actual row count and byte size. Would it fit on a 64 GB machine?
• In Demo 1, add a fourth aggregate (revenue by product) and note how little the timer moves. Parallelism across cores is doing the work.
• Explain map-shuffle-reduce to a teammate in three sentences, and point to which step costs the most.
• Download DuckDB (duckdb.org) and run Demo 2's window query against a real CSV of your own. Feel the single-node speed on real data.
• List one workload in your world that genuinely justifies a cluster, and one that does not. Be honest about which is more common.

This session teaches the distributed-processing core of the DeepLearning.AI Data Engineering Professional Certificate (Joe Reis) and Reis & Housley's Fundamentals of Data Engineering, run on a live single-node engine. Certificates, cloud labs, and videos stay on the official platforms. This page covers: