Where the artifact stands
Thirty mornings on time, twelve silently wrong, and a crontab that cannot see the difference. Session 1 ended with the diagnosis: "run at 2:30" is a guess about duration standing in for the thing we actually meant, which was "run after the extracts succeed". Tonight we say the second sentence to a scheduler that can enforce it, watch what it costs, and meet the vocabulary twice - once in Airflow's words, once in Dagster's - to prove the ideas belong to no product.
★ What you walk out with: the run-after-not-run-at mental shift, the three downstream run states and why "skipped" is the honest one, a two-vocabulary phrasebook (Airflow and Dagster) for one set of concepts, and a simulator calendar where failures finally show up as missing mornings instead of wrong ones.
Run after, not run at 7 min live
Airflow's docs define the object this whole course orbits: "A Dag is a model that encapsulates everything needed to execute a workflow" - and crucially, it "dictates the order in which they have to run, and which tasks depend on what others". The arrows from session 1's diagram stop being wishful thinking and become configuration.
Live The contract, and the three downstream truths 4 min ▶
In Airflow you declare the arrows in code, with the >> and << operators: extracts >> clean >> join. That single line replaces every padded minute from session 1. "Run at 2:30" was a guess about how long the extracts take; "run after clean succeeds" is a fact about whether they finished. Facts do not need safety margins.
Once the arrows are enforced, every task downstream of a failure lands in one of three states, and each one is telling you something true:
- Success - it ran, on inputs that actually existed. The only green worth trusting.
- Failed - it ran and died. Loud, visible, attributable to a task and a night.
- Skipped - it never ran, because its upstream did not succeed. Nothing executed on missing input.
Skipped is the state people instinctively read as a defect, and it is the opposite: it is the DAG refusing to do what cron did every night - run anyway on whatever data was lying there. No output beats wrong output.
Daybreak's clean task now waits for all three extracts. When extract orders dies at 2:04, clean never fires - and neither does anything after it. The 7am dashboard is visibly missing instead of quietly wrong: a store manager sees a gap and picks up the phone, instead of reading yesterday's numbers with today's date and drafting a memo about a trend that does not exist.
Self-study A run is about a night, not just at a time 3 min ▶
There is a second idea hiding in DAG runs, and it will carry session 5 on its back. Airflow's docs: "Each Dag run in Airflow has an assigned 'data interval' that represents the time range it operates in." And the run's logical date "denotes the start of the data interval, not when the Dag is actually executed."
Read that twice, because it quietly changes what a run IS. Under cron, a run was an event: something that happened at 2am. Under a DAG, a run is a claim about a time range: "this run processes Tuesday night's data" - whenever it actually executes. Tonight, or Thursday, or three weeks from now when you discover Tuesday was wrong. The run being ABOUT a night rather than just AT a time is exactly what makes history repairable, and session 5 will spend its whole build-along cashing that in.
The same ideas everywhere 7 min live
Everything in part 1 sounded like Airflow features. It is not - it is what orchestration IS, and the fastest proof is hearing a second tool say the same things in different words. Dagster calls a unit of data an asset: "a logical unit of data such as a table, dataset, or machine learning model". Different noun, same arrows.
Live The phrasebook, and picking the starting gun 5 min ▶
Dagster's own one-liners complete the mapping. A schedule is "a way to automate jobs or assets to occur on a specified interval". A sensor is "a way to trigger jobs or assets when an event occurs". Hold those two next to each other, because they are the last decision cron never let you make: WHEN should the night start at all?
- Schedule - a clock: kick off the run at 2:00am. Right whenever the cadence itself is the contract - "the dashboard covers yesterday, built overnight".
- Sensor - an event: kick off the run when the source file actually lands. Right whenever the trigger you care about is a THING arriving, not a time passing.
Notice what the sensor version quietly finishes off. Session 1's crontab started extracts at 2:00 hoping the upstream systems had exported by then - one last guess, one last piece of padding. "Run when the file lands" replaces it with the same move part 1 made downstream: an event fact where a clock guess used to be. Inside the DAG, completion drives the order; at the front door, arrival can drive the start.
Self-study What an orchestrator does not fix 3 min ▶
Honest scoping, before the build-along makes it visceral: an orchestrator manages WHETHER and WHEN tasks run. It has no opinion about whether the code inside them is correct. Daybreak's aggregate task carries a logic bug on days 10 through 13, and it will produce wrong numbers under cron, under a DAG, under every lever this course pulls - because on those nights every upstream genuinely succeeded and every dependency was genuinely satisfied. The DAG did its whole job. The code did its job wrong.
What orchestration buys you is that failure becomes visible and recovery becomes cheap: a dead extract shows as failed, its downstream shows as skipped, and nothing publishes on garbage. What it never buys you is correctness of logic. Keep the two ledgers separate - it is the difference between blaming your scheduler and testing your code.
Turning the dependencies on ★ 22 min · everyone builds
Same thirty nights, same seeded failures as session 1. One new lever. Watch which numbers it moves, and - just as important - which it does not.
Baseline first: run the month on bare cron and confirm session 1's pair - 30 of 30 mornings on time, 12 of 30 silently wrong. The punctual liar, exactly as you left it.
Flip "DAG dependencies" and read the whole board before reacting. On time falls to 23 of 30. Silently wrong collapses from 12 to 4. Six mornings are now never published, and one is late but right. Look at the calendar: the reds did not get fixed, they got HONEST - failed nights turned into grey missing squares instead of normal-looking wrong ones.
Kill a task - pick an extract - and run tonight's trace. Downstream, clean and everything after it now reads "skipped - upstream did not succeed" instead of running on stale data. Compare that sentence with session 1's trace, where the same night ended in a dashboard that looked exactly like every other morning.
Name the trade out loud, because it is the session: honesty first. Failed nights became loud missing mornings. Nothing retries yet, and nobody is paged yet - a transient 2:04am blip still costs the whole night, and the gap is discovered by a human at 7am. Those are sessions 3 and 6, and now you know exactly why they exist.
About the 4 that will not budge: the four remaining silently-wrong mornings are the days 10-13 aggregate logic bug. Every dependency was satisfied; the code was simply wrong. No scheduler - not this one, not any lever left in the course - can fix code. Watch that number stay put all the way to session 5, where the backfill at least repairs the record.
Before session 3 15 min
- Take the chain you drew after session 1 and mark every arrow that is currently enforced only by clock padding. Each mark is a place where "run at" is standing in for "run after" - and a candidate for the first DAG you write.
- For that chain's very first task, decide the starting gun: clock or event? If the honest answer is "when the upstream export lands", you have found a sensor wearing a cron schedule as a disguise.
- Find one job at work whose green checkmark you now trust less: it succeeds every night, but would it succeed on EMPTY input too? Write down what "skipped" would have saved you the last time its upstream failed.
Three questions
1. What does a DAG add that cron structurally cannot?
The DAG "dictates the order in which they have to run, and which tasks depend on what others" - run-after replaces run-at. It makes nothing faster and, as the days 10-13 bug proves all course long, it cannot make code correct.
2. An extract dies at 2:04 and the morning shows clean as "skipped". Why is that a feature?
Skipped is the DAG's honesty: downstream never executed on stale or missing data. Under cron the same night produced a normal-looking dashboard built on yesterday's partition - the expensive kind of failure, because nobody knows to distrust it.
3. A run's logical date is Tuesday, but it executed Thursday. What does that mean?
The logical date "denotes the start of the data interval, not when the Dag is actually executed". A run is a claim about a time range, not just an event at a time - which is exactly what will make history repairable when session 5 reruns old nights.
Answer all three to see your score.