Where the artifact stands
Twenty-three mornings on time, four silently wrong, six never published, one late but right. The DAG made failure visible; it did nothing to make failure cheap. A 2:04am network blip still costs Daybreak the entire night, because the run dies once and stays dead. Tonight the scheduler learns to try again - carefully, with backoff, a bounded number of times - and we read the fine print on what "again" does to a task that was halfway through writing.
★ What you walk out with: the transient-versus-real sorting test, the backoff-jitter-budget discipline straight from the SRE and AWS literature, first-hand sight of the course's honest dip - the night a retry turned a crash into a double-write - and a halfway scorecard for the build as it stands.
Failures that pass on the second try 7 min live
A network blip, a warehouse hiccup, a container eviction: try again and they pass, because the world moved on. A schema break: try again forever and it fails forever, because the code is wrong. Retries exist for the first kind only, and every retry setting you will ever choose is really an answer to "how sure am I which kind this is?"
Live The dials, and why you never hammer 4 min ▶
The mechanics are refreshingly small. In Airflow, retries is the maximum retry count for a task; retry_delay spaces the attempts; retry_exponential_backoff grows that spacing, capped by max_retry_delay; a task waiting for its next attempt sits in the up_for_retry state; and execution_timeout caps a runaway attempt so a hang cannot eat the night. Dagster says the same things through a RetryPolicy - a delay, Backoff.EXPONENTIAL, and Jitter. Two vocabularies again, one idea: try again, wait longer each time, do not all wait the same amount, and stop.
Why the ceremony - why not retry immediately, and forever? Because the systems literature is unusually blunt about what naive retries do at scale. Google's SRE book: "retries can destabilize a system", and when layers retry independently the amplification can be "as large as the product of the number of attempts at each layer". Their prescriptions: "Always use randomized exponential backoff", and cap the damage with a retry budget - their worked example would "only allow 60 retries per minute", after which requests simply fail.
AWS's Builders' Library tells the layered story from production scars: with three layers each retrying independently, one struggling dependency at the bottom saw its load multiply to 243x - every layer's attempts compounding on the ones below. The fix they eventually shipped INTO the AWS SDK itself, in 2016, was a token bucket: retries flow while tokens last, then stop. From the article "Timeouts, retries, and backoff with jitter", by Marc, an AWS employee.
Self-study Selfish, and similar to a powerful medicine 3 min ▶
The same AWS article carries the two framings worth memorizing verbatim. First: retries are "selfish" - a retrying client spends more of the server's time to raise its own odds of success, which is exactly why a whole fleet of clients retrying at once can finish off a struggling service. Second: retries are "similar to a powerful medicine" - the right dose fixes the transient blip, and an overdose does damage of its own.
Tonight Daybreak takes the right dose: two retries, with backoff, then fail loudly. Hold on to the medicine framing, because session 6 puts the overdose on a meter - the same simulator, retries cranked to ten, and a number that only moves in the wrong direction.
The dip: what a rerun does to a half-finished write 6 min live
Here is the fine print nobody reads until it bills them. When a task that APPENDS rows crashes halfway through, its first attempt's rows are already sitting in the table. The retry does not know that. It runs the task again, from the top, and appends everything again.
Live The honest accounting 3 min ▶
Write the ledger the way the numbers will show it in the build-along. Retries buy availability: mornings that used to die on a blip now recover and publish. And when a task is not rerun-safe, retries quietly spend correctness to do it: the recovered morning publishes numbers with a double-counted first half. Availability up, correctness down, and the second effect invisible unless you are counting.
Resist the instinct the dip will trigger. The fix is NOT fewer retries - dialing back to zero just re-buys session 2's six dead mornings. The fix is making reruns safe, so that running a task twice lands the same rows once. That property has a name, idempotency, and it is the whole of session 4. Tonight we take the dip with our eyes open and leave the dial alone.
Self-study The fingerprint trail a recovery leaves 3 min ▶
A night that recovered is not the same as a night that ran clean, and your orchestrator records the difference even when the dashboard does not. A task that failed and is waiting to go again sits in up_for_retry - that state in last night's history means attempt one died, whatever the final color says. An attempt that hit execution_timeout tells you the task did not fail fast, it hung until the cap killed it - a different disease than a crash.
So the habit: after a rocky night, read the task states before trusting the green morning. A run whose writer shows two attempts is exactly the run whose output deserves a row-count sanity check - because as of tonight you know precisely what attempt two may have done to the table. Green says the run finished. The trail says how, and how is where the double-write hides.
The recovery, and the dip ★ 22 min · everyone builds
Same thirty nights. One new lever, two numbers moving in opposite directions - watch both, because only one of them is on the dashboard.
Rebuild session 2's state: DAG dependencies on, retries off. Confirm the board - 23 of 30 on time, 4 of 30 silently wrong, six never published. This is the honest-but-fragile baseline the dip gets measured against.
Flip "Retries with backoff" - two attempts, spaced out. On time jumps to 28 of 30: five missing mornings came back, every one of them a transient blip that passed on a later attempt. Only day 14 stays dead, and the trace says why - extract_payments hard-breaks, and a retry cannot fix a schema break. The sorting gate, live.
NOW read silently wrong, the counter you stopped watching: it went UP, 4 to 6. Click into it and the counter names the mechanism - 2 nights double-wrote. On each, a crashed writer had already appended half its rows when it died; the retry ran the task from the top and appended them all again. The recovered morning published an inflated total, on time, in green.
Say the trade out loud: availability was bought with correctness. And do NOT reach for the retry dial to fix it - fewer retries just trades those 2 wrong mornings for 5 dead ones. The real fix is making reruns safe, and that is the entire next session. Then fill in the halfway scorecard below, honestly.
Where the build honestly stands
| Check | Daybreak tonight | Where it lands |
|---|---|---|
| Dependencies enforced | ✓ clean waits for all three extracts | Session 2's wire, holding firm |
| Transient failures self-heal | ✓ 28/30 on time, five mornings recovered | Tonight's lever, correctly bounded |
| Reruns are safe | ✗ two nights double-wrote this month | Session 4: idempotency |
| History is restatable | ✗ days 10-13 still sit wrong in the record | Session 5: backfill |
| Anyone paged when it dies | ✗ day 14 waited for a human at a dashboard | Session 6: alerting |
Reading the verdict: self-healing, and honest about half of it. Halfway is a real place - plenty of production pipelines live here for years - but do not ship it as done. The two ✗ rows that touch data (reruns, history) are the ones quietly costing money right now; the third costs you the worst morning of the month.
Before session 4 15 min
- List the last three failures of the pipeline you have been carrying since session 1, and sort each through the gate: transient (would a retry have passed?) or real (code, schema, logic). If you cannot tell from the logs, that inability is itself a finding - write it down.
- Find its retry settings, or confirm there are none. If retries exist, check for backoff and a cap; an unbounded or zero-delay retry is tonight's literature ignored. If none exist, note which of the three transient failures above each retry would have absorbed.
- Pick its most important writer task and answer one question in writing: if this task ran twice tonight, what would the table look like tomorrow? If the answer involves the word "twice", you have found your session 4 homework early.
Three questions
1. Day 14: extract_payments hits a schema break. What do retries do for that night?
Retries exist for transient failures - blips the world causes and the world clears. A schema break fails the same way on every attempt; no attempt count fixes code. The right outcome for day 14 is failing loudly with the night's remaining hours intact.
2. Retries went on and silently wrong rose from 4 to 6. What happened on those two nights?
The crashed attempt's rows survived in the table; attempt two ran the task from the top and appended everything again, so the early rows counted twice. The retry behaved exactly as designed - the task was not safe to rerun. That is idempotency's cue.
3. Why backoff with jitter and a cap, instead of retrying immediately and forever?
The SRE book's warning is that retries can destabilize a system, with amplification as large as the product of attempts at each layer - hence randomized exponential backoff and a retry budget like their 60-per-minute example. AWS's 243x story is the same math in production.
Answer all three to see your score.