Where the artifact stands
Daybreak's build now has real dependencies and two retries with backoff: 28 mornings on time out of 30. But silently wrong sits at 6, up from the DAG rung's 4, and the trace knows why - two nights the crashed writer had already appended rows before the retry ran the whole write again. The lever that saved the schedule poisoned the data. This session ends with the same 28 on-time mornings, silently wrong back at 4, and double-writes at zero - without touching the retry dial.
★ What you walk out with: the deterministic-plus-idempotent pair kept properly separate, the overwrite-the-partition pattern that makes any crash-and-retry converge, and a three-item checklist of determinism sins - now(), "latest data", and append - to hunt through your own tasks tomorrow.
Tasks as pure functions 8 min live
The clearest statement of tonight's idea comes from Maxime Beauchemin - who identifies himself in the article as the creator of Airflow - writing on functional data engineering: batch tasks should behave like pure functions over partitions of data.
Live Two properties, kept properly separate 5 min ▶
Beauchemin's definition packs both into one sentence: "A pure task should be deterministic and idempotent, meaning that it will produce the same result every time it runs or re-runs". Those are two different promises, and a nightly task wants both.
- Deterministic - same inputs, same output. Run the task against night 12's data next Tuesday and you get night 12's answer, not next Tuesday's.
- Idempotent - running twice equals running once. The second execution lands the system in the same state as the first, so a retry is free.
The write side is where idempotency is won or lost, and the prescription is blunt: "A pure task should always fully overwrite a partition as its output". More generally: "Thinking of partitions as immutable blocks of data and systematically overwriting partitions is the way to make your tasks functional". Why care this much? Because "Having idempotent tasks is vital for the operability of pipelines" - session 3's retry dial, and session 5's backfill button, are both only safe when this property holds.
Daybreak's two dip nights, replayed: the aggregate crashes at 3:40am after appending half its rows. The retry - working exactly as configured - runs the task again and appends all of them. The morning total is inflated, the dashboard renders, nothing anywhere errored. Rewrite the task to overwrite the night's partition and the identical crash-plus-retry converges to the correct table, every time.
Self-study The deeper prize: reproducibility 3 min ▶
Rerun-safety is the operational win, but Beauchemin's argument goes one floor further down: "immutable data along with versioned logic are key to reproducibility". If partitions are immutable blocks and each one records which version of the logic wrote it, then any historical number can be regenerated and checked - the pipeline becomes auditable, not just restartable.
That matters beyond incidents. When finance asks why March 12 revenue changed between two pulls of the same report, an append-style pipeline can only shrug. A partition-overwriting one can answer: this partition, that logic version, rewritten on this date, for this reason. The same discipline that makes retries safe makes the numbers defensible.
Write like a transaction, not a diary 7 min live
The Airflow best-practices page turns the paradigm into rules you can grep for. Its framing device is the one every backend engineer already trusts: "You should treat tasks in Airflow equivalent to transactions in a database" - they should "never produce incomplete results".
Live Partition by the data interval 4 min ▶
Session 2 established that a run is ABOUT a night - the logical date names the data interval, not the moment of execution. Idempotency is where that idea starts paying rent. The best-practices page spells out each rule: on writes, "Do not use INSERT during a task re-run, an INSERT statement might lead to duplicate rows in your database. Replace it with UPSERT". On time, now() "should never be used inside a task". On reads, "Never read the latest available data in a task" - instead, "Read and write in a specific partition. You can use data_interval_start as a partition."
Put together, Daybreak's aggregate becomes a pure function of one night: read the night's cleaned partition, compute, overwrite the night's aggregate partition. Same code, any night, any number of times. The retry stops being a gamble, and - quietly, for next session - so does rerunning a night from three weeks ago.
Self-study Where idempotency is genuinely hard 3 min ▶
Overwriting a partition works because the warehouse is yours to rewrite. Some task effects leave the building: sending an email, charging a card, calling a third-party API that creates something on the other side. Run those twice and the customer is emailed twice - no partition to overwrite. The working pattern is to make the WRITE idempotent even when the ACTION cannot be: record every intended side effect with a dedupe key (say, night plus customer plus message type) in a table you do own, and only perform the action for keys not already marked done. A retry then re-reads the ledger, skips what happened, and finishes what did not. It narrows the double-fire window rather than abolishing it - the honest claim is "at most once per key, verified against our ledger", and for a nightly batch that is usually exactly enough.
The dip, repaired at the source ★ 22 min · everyone builds
Same thirty nights, same crashes, same retry dial. One new lever: tasks that overwrite their night's partition instead of appending to it.
Rebuild session 3's end state: DAG dependencies on, retries on. Confirm the board reads 28 mornings on time, 6 silently wrong, and the counter nobody likes: 2 nights double-wrote. Those two nights are the honest price of retrying tasks that append.
Flip "Idempotent tasks" and watch two numbers move together: double-wrote drops to 0, and silently wrong falls back to 4 of 30. Notice what you did NOT touch - the retry dial. The dip was never a retry problem; it was a write-strategy problem that retries exposed.
Kill a writer task - clean or aggregate - and read tonight's trace. The crash happens, the retry fires, and the line that used to warn about rows appended twice is gone: the rerun overwrote the night's partition and converged. Crash plus retry now equals a clean run.
Name what remains: 4 wrong mornings, days 10 through 13. That is the aggregate logic bug - correct scheduling, correct reruns, wrong code. No scheduler fixes it. Code bugs need a repair path for HISTORY, and that is exactly where session 5 picks up.
Why this lever is invisible in the wrong metric: idempotency did not add a single on-time morning - 28 before, 28 after. It made the mornings you already had TRUE. That is why it never shows up in uptime dashboards and always shows up in audits, and why teams that only track availability skip it until the first restated quarter.
Before session 5 20 min
- Open the pipeline you brought in session 1 and grep its tasks for the three sins: now() or CURRENT_TIMESTAMP inside task logic, any "latest" read that is not pinned to an interval, and any plain INSERT or append on a path a retry can reach. Write down what you find - counts, not feelings.
- For its most important output table, answer one question in writing: if tonight's write crashed halfway and reran, would the table converge or inflate? If you cannot answer from the code, that is the answer.
- Pick the single worst offender and sketch its overwrite version: which partition key names the night, what the full rewrite looks like, and what (if anything) breaks downstream. Do not ship it yet - session 5 gives the second reason to.
Three questions
1. A task always produces the same output for the same inputs, but rerunning it appends a second copy of that output. What is it?
The two properties are separate promises. Deterministic is about the VALUE (same inputs, same output); idempotent is about the STATE (running twice equals running once). Daybreak's aggregate was perfectly deterministic on both dip nights - and the retry still doubled the rows.
2. Why does overwrite-the-partition beat append for a nightly task?
Performance and storage are side issues. The point is convergence: a full overwrite makes every rerun land the identical partition, so the second, third or tenth execution is harmless. Beauchemin's rule - a pure task should always fully overwrite a partition as its output - is a rerun-safety guarantee.
3. What is actually wrong with calling now() or reading "the latest data" inside a task?
Both sins tie the output to the moment of execution instead of the data interval. The Airflow guidance is to read and write a specific partition, using data_interval_start as the key - then night 12's run gives night 12's answer whether you execute it tonight or replay it in three weeks.
Answer all three to see your score.