Learn with Phoebe · Data Engineering & Infra

Learn Streaming Data with Phoebe

At 2pm the revenue dashboard still shows midnight, because the pipeline runs nightly. This course closes that 14-hour gap the honest way: flip the stream on and watch the number go WRONG, then earn correctness back one lever at a time - idempotency, event-time windows, watermarks, replay. Every number in the simulator is computed live from a deterministic day of coffee-shop orders. And the first lesson is the unfashionable one: most dashboards should stay batch.

6sessions
14hours of lag to erase
78.9% where naive streaming lands
100% where replay ends
45min per session

The six sessions ⚡ · one dashboard, upgraded across all of them

The running artifact is Daybreak, the coffee shop from Learn Data Pipelines, now demanding its numbers in real time. Same orders, same world - this is that course's real-time sibling.

🕑 Session 1 · easy

The stale dashboard

The nightly batch is not wrong, it is old. What an event is, why facts beat rows, and the three-zone test for whether you need streaming at all.

▶ Start here 📜 Session 2 · medium

The log

Topics, partitions, offsets and consumer groups - the four ideas under every streaming system. Flip the stream on and watch freshness arrive and honesty leave.

Kafka core 🔁 Session 3 · medium

In and out

At-least-once is the default, retries arrive twice, and exactly-once is narrower than the brochure says. Idempotency fixes the double-count for real.

halfway scorecard 🪟 Session 4 · hands-on

Windowing

Tumbling, hopping and session windows, and the two clocks. Doing it right makes the number visibly WORSE - the most honest dip in the course.

the dip ⏳ Session 5 · hands-on

Late data

Watermarks decide how long an hour stays open. Stragglers get caught, the truly late get counted instead of vanishing, and replay converges the day to 100%.

watermarks + replay 🚢 Session 6 · hardest

Ship it

Consumer lag as the one metric that matters, what always-on infra really bills, lambda vs kappa from the people who coined them - and the lever that never helps.

🔒 final scorecard
easy - everyone medium hands-on hardest

Your passport 🎫

Clear all three quiz questions in a session and it gets stamped. Four stamps unlock the replay lever. Stamps live in this browser only.

Choose your path 🗺️

Three doors into the same conviction: freshness and correctness are different axes, and you pay for each separately.

🔥 Someone wants our dashboards real-time 1→ 6 the three-zone test, then what it really costs to run
🛠 I have a stream and the numbers look off 3→ 4→ 5 duplicates, then buckets, then stragglers - in that order
⚡ The whole journey 1→ 2→ 3→ 4→ 5→ 6 from a stale dashboard to a stream you can defend

What is real here 🧾

A course built around a simulator owes you a precise account of its own numbers.

ClaimStatus
The accuracy ladder: 78.9 → 89.1 → 88.0 → 93.6 → 100.0Computed live in your browser from a fixed-seed day of order events. The generator is a simulation; the aggregation math is real and re-runs on every click
The dip at the windowing rungReal arithmetic, kept on purpose: strict windows drop stragglers that sloppy bucketing was at least counting somewhere
Infra cost figures ($140 / $340 / $1,980 per month)Modelled, and badged as modelled on the widget. The always-on billing SHAPE is real: AWS bills a provisioned Kinesis shard per hour whether or not data flows
Kafka semantics: at-least-once default, idempotent producer, per-partition orderingQuoted from the Apache Kafka documentation, verified September 2026
Window and watermark definitionsQuoted from the Apache Flink documentation
"Most workloads do not need streaming"Tigani (MotherDuck, 2023): 90% of measured queries under 100MB. Taught alongside the steelman - fraud and alerting genuinely do

The knowledge map 🧠

The whole course at a glance - hover a session to spotlight its concepts, click any node to jump in.