The six sessions 🌙 · one nightly build, upgraded across all of them
The running artifact is Daybreak's 2am warehouse build - the third course set in the world of Learn Data Pipelines (what a pipeline is) and Learn Streaming Data (the real-time layer). This one is about running the batch reliably.
The 2am job
Cron promises punctuality and nothing else. Thirty mornings on time, twelve silently wrong - and the three failures a clock cannot see.
🕸️ Session 2 · easyFrom cron to DAG
Replace "run at 2:30" with "run after clean succeeds". Wrong mornings collapse - and failed nights become honestly, visibly missing.
🔁 Session 3 · mediumRetries
Transient blips heal on attempt two. And the recovered nights start double-writing - availability bought with correctness, measured.
♻️ Session 4 · mediumIdempotency
Overwrite the night's partition, never append. Reruns converge, the double-writes vanish, and the pipeline becomes auditable.
📅 Session 5 · hands-onBackfills
A bug shipped on day 10 and was found on day 14. The fix repairs tomorrow; the backfill repairs the record - if your tasks earned it.
📟 Session 6 · hands-onRun it like production
Page on the missing dashboard, never the restarting process. And the retry dial's true price: ten attempts, three fewer hours of night.
Your passport 🎫
Clear all three quiz questions in a session and it gets stamped. Four stamps unlock the backfill lever. Stamps live in this browser only.
Choose your path 🗺️
Three doors into the same conviction: on time and correct are different promises, and each lever pays for exactly one of them.
What is real here 🧾
A course built around a simulator owes you a precise account of its own numbers.
| Claim | Status |
|---|---|
| The 30-night ladder: 12 wrong on cron, the retry dip 4 → 6, the 12-minute page | Computed live in your browser. Which nights blip or run slow is a fixed-seed script so runs are comparable; what the scheduler does about each - waiting, skipping, retrying, rerunning, restating - is real arithmetic, recomputed on every click |
| The retry rung making correctness WORSE before idempotency repairs it | Real arithmetic, kept on purpose: a crashed writer has already appended half its rows, and the retry appends them again |
| Cron's limits: no dependencies, no retries, no overlap protection | Verified as feature ABSENCE from the crontab(5) and cron(8) man pages - stated as absence, never as a quotation |
| DAG, catchup, backfill, data interval definitions; SLAs removed in Airflow 3 | Quoted from the Apache Airflow 3.3.1 documentation, verified September 2026 |
| Overwrite-the-partition idempotency and safe backfills | Quoted from Maxime Beauchemin's Functional Data Engineering (2018) - he created Airflow - and Airflow's own best-practices page |
| Retry amplification, retry budgets, alert-on-symptoms | Quoted from the Google SRE book (cascading failures + monitoring chapters) and the AWS Builders' Library |
The knowledge map 🧠
The whole course at a glance - hover a session to spotlight its concepts, click any node to jump in.