Where the artifact stands
Daybreak's ladder so far: dependencies, bounded retries, rerun-safe tasks. The board reads 28 mornings on time, 4 silently wrong, double-writes zero - and one counter that has not moved since session 1: days of history correct, stuck at 26 of 30. Every lever so far protected FUTURE mornings. This session's lever is the first one that points backwards.
★ What you walk out with: the catchup-versus-backfill distinction (filling gaps versus deliberately redoing what ran wrong), the reason idempotency is a hard precondition rather than a nice-to-have, and the one-sentence stakeholder framing for every restatement you will ever run.
Two ways to run the past 8 min live
Because session 2 made runs be ABOUT intervals, the past is addressable: every night has a name, and a named night can be run again. Orchestrators expose that in two forms that look alike and mean very different things.
Live Daybreak's case: fixed pipeline, unfixed tables 5 min ▶
The Airflow definitions first, because the words get swapped constantly. With catchup=True, "the scheduler will kick off a Dag Run for any data interval that has not been run since the last data interval (or has been cleared)" - the scheduler filling slots it knows are empty. A backfill is the deliberate version: "re-run all the instances of the dag_id for all the intervals within the start date and end date". In Airflow 3 it is a first-class command: airflow backfill create --dag-id --from-date --to-date --reprocess-behavior. The distinction that matters operationally: catchup handles intervals that NEVER ran; a backfill reprocesses intervals that DID run - because the code or the data was wrong when they did.
Daybreak is squarely in the second case. The bug shipped day 10, was found day 14, and the hotfix means every night from 15 onward computes correctly. But nothing about fixing the code touches what the bug already wrote: days 10 through 13 are wrong in the aggregate table, wrong in everything joined onto it, and wrong in every export taken since.
Day 20, the month-end report gets pulled. It aggregates over the whole month, so it still contains the bug-era numbers for days 10-13 - a week after the "fix" went out. A fixed pipeline with unfixed tables keeps re-serving the old mistake every time someone queries the window it lives in. Until history is restated, the fix is a promise about the future only.
Self-study The same verb in partition-native vocabulary 3 min ▶
Dagster, which thinks in assets and partitions rather than dag runs, defines it in one line: "Backfilling is the process of running partitions for assets that either don't exist or updating existing records". Same two cases, same verb - fill what is missing, redo what is wrong - which is the tell that this is a concept of scheduled data work, not a feature of one tool. Whatever orchestrator you land on, look for the operation that says "run this window of the past again" and check what it assumes about your tasks before you trust it. The next part is exactly that assumption.
A backfill is reruns at scale 7 min live
Session 4 made one rerun safe. A backfill is nothing but reruns in a row - so whatever a rerun does to one night, a backfill does to the whole window, at once, on purpose.
Live The downstream blast radius 4 min ▶
Beauchemin states the scope plainly: when a source table's history changes, "all of the data structures derived from that table, for that time range, need to be reprocessed, or backfilled". Restating the aggregate is not enough if three tables are built from it - each of those needs days 10-13 redone too, in derivation order. And the redo must be honest: "apply the exact same compute scheme as we did on the original pass, on top of the new, corrected data" - the fixed logic, the same partitioning, no quiet extra adjustments smuggled in while nobody is looking.
Derivation order is exactly what one artifact in this course already knows: the DAG. That is why the backfill is an orchestrator feature and not a shell script - the same dependency graph that ordered tonight's run can order the reprocessing of any past window, task by task, downstream table by downstream table.
Self-study What a backfill cannot repair 3 min ▶
A backfill repairs the RECORD. It does not repair the mornings. On days 10 through 13, store managers opened a dashboard that was wrong, and whatever they decided on it - orders placed, staffing shifted, a memo written - already happened. No reprocessing reaches back into those decisions. This is why restatements deserve honest framing rather than quiet fixing: the trustworthy sentence for stakeholders is "history is now correct; here is the window where it was not." It tells people the numbers changed, why, and exactly which past reports to distrust - and it costs far less credibility than someone discovering the numbers moved on their own.
Restating days 10-13 ★ 22 min · everyone builds
The first backwards-pointing lever. It is gated twice - once by your passport, once by physics.
Rebuild the ladder: DAG dependencies, retries, idempotent tasks. Confirm the board reads 28 mornings on time, 4 silently wrong - and days of history correct still stuck at 26 of 30. Those four are days 10-13, the bug window.
Find the backfill lever. It is passport-gated - four stamps open it, and the honest bypass link is there if your quiz record is short. Now test the second gate: flip idempotency OFF and try the backfill checkbox. It refuses, and its tooltip says why: backfilling non-idempotent tasks rewrites history unpredictably. The simulator will not let you commit last diagram's right-hand panel. Flip idempotency back on.
With idempotency on, flip "Backfill the bug window" and watch the counter that never moved before: days of history correct goes 26 of 30 to 30 of 30. The fixed aggregate logic reruns days 10-13 in dependency order, each night overwriting its own partitions.
Now read what did NOT change, because it is the honest half of the lesson: the calendar still shows the 4 red mornings, and silently wrong still reads 4 of 30. Those mornings happened - people saw wrong numbers and acted on them. The backfill restated the record, not the past. Say the stakeholder sentence out loud: history is now correct; here is the window where it was not.
Schedule the backfill conversation, not just the backfill: put it in the same meeting where the bug is announced. The repair window is a decision - how far back to restate, at what compute cost, and who needs to re-pull which reports - not a technicality to be discovered later. A bug announcement without a restatement plan is half an announcement.
Before session 6 20 min
- For the pipeline you have carried since session 1, answer in writing: has its history ever been restated? If yes, find how it was done (rerun window, manual SQL, or hope). If never, decide whether that means the code has never been wrong - or that nobody has ever repaired it.
- Trace one important table's blast radius: list every table derived from it. That list is the true size of any future backfill, and most people have never written it down.
- Draft your version of the stakeholder sentence for a hypothetical four-day bug in that pipeline: what would be restated, which reports would need re-pulling, and who would need to hear it. Keep it - you will not want to be writing it during the incident.
Three questions
1. The scheduler was paused for three days and, on resume, starts runs for the missed intervals. Which mechanism is that?
Catchup is the scheduler's own gap-filling: with catchup=True it kicks off a run for any data interval that has not been run (or has been cleared). A backfill is the deliberate human act of reprocessing intervals that DID run - because what they ran was wrong.
2. Why does the backfill lever refuse to work without idempotent tasks?
Whatever one rerun does to one night, a backfill does to the whole window. If a rerun appends, the "repair" adds duplicates to every night it touches - repair becomes vandalism. Overwrite-the-partition tasks make the same command converge every night to the corrected truth.
3. After a clean backfill of the bug window, what is still true?
A backfill repairs the record, not the past. The warehouse now holds correct history for days 10-13, but the dashboards people read on those mornings were wrong, and what they did about it already happened. Hence the stakeholder sentence: history is now correct; here is the window where it was not.
Answer all three to see your score.