Where the artifact stands
Twenty-eight mornings on time, history restated to 30 of 30, reruns that overwrite instead of double-writing. The remaining exposure is time-to-detection: the day-14 hard break sat undiscovered for nine hours because the only detector was a human opening a dashboard. Tonight that number becomes twelve minutes, and the final scorecard decides whether this thing is actually done.
★ What you walk out with: the symptom-versus-cause alerting frame from the SRE book applied to data deadlines, the evidence for why aggressive retries delay your own alarms, and a strict ship-or-not scorecard for any scheduled pipeline you operate.
Page on the missing dashboard, not the restarting process 8 min live
Google's SRE book draws the line that data teams keep tripping over: the "what's broken" is the symptom, the "why" is the cause - and it is better to spend much more effort catching symptoms than causes.
Live The two rules that keep a pager trustworthy 5 min ▶
The SRE book's paging discipline compresses to two sentences. Every page should be actionable. And every page response should require intelligence - if a page merely merits a robotic response, it should not be a page. A restarting worker fails both tests: the robotic response is "wait, it restarts", so it should never have paged. The missing 7am dashboard passes both: someone must decide whether to rerun, restate, or warn the stores.
For a nightly build, the outcome contract writes itself as a deadline: the dashboard exists, is fresh, and arrived by 7am. Orchestrators grew first-class support for exactly this - and the ground moved recently, so check your version: the SLA feature from Airflow 2 was removed in Airflow 3.0 and replaced in 3.1 with Deadline Alerts. Same idea, new name: declare when the outcome is due, page when it is not.
The freshness half of the contract is so common it is a stock feature downstream too: dbt lets a source declare warn_after and error_after against a loaded_at field, and literally runs a max(loaded_at) query to decide whether your data is stale against the SLA you defined. The check is a SELECT statement, not a heroic monitoring project.
Self-study The four golden signals, translated for pipelines 3 min ▶
The SRE book's four golden signals are latency, traffic, errors and saturation. They were written for serving systems, and they translate cleanly to a nightly build: latency is minutes-past-deadline, traffic is rows processed per night (a sudden halving is a silent extract failure wearing a green checkmark), errors are failed and skipped tasks, saturation is how close the slowest night runs to the deadline.
The translation earns its keep on trend, not on any single night: a build that finishes 6:40, 6:47, 6:52 across three weeks has not failed yet, and is telling you exactly when it will.
Retries are a loan against your detection time 7 min live
Session 3 set retries to two, with backoff, and stopped. Tonight the simulator shows what "just crank it to ten" actually purchases. The systems literature is unusually unanimous here.
Live What the SRE book and AWS both learned the hard way 5 min ▶
Google's cascading-failures chapter is blunt: retries can destabilize a system, and a request retried at several layers multiplies - the total attempts can be as large as the PRODUCT of the attempts at each layer. Their prescriptions are the ones session 3 baked in: always use randomized exponential backoff, never retry indefinitely, and cap the damage with a retry budget - their worked example allows a process 60 retries per minute, after which requests simply fail.
AWS's Builders' Library reaches the same place from production scars, with the best one-line framing in the literature: retries are selfish - a retrying client spends more of the server's time to raise its own odds - and they are similar to a powerful medicine, useful in the right dose and damaging beyond it. Their worked example has three layers of independent retries turning one database brownout into 243x load. Their fix shipped into the AWS SDK itself in 2016: a token bucket that lets everything retry while tokens last, then throttles.
- And the line that closes the loop on this course's pager: the same AWS article notes, in so many words, that retries can MASK the failures your timeouts caught. The SRE monitoring chapter lists "failures masked by retries" as exactly the thing white-box monitoring exists to surface.
The dial has a correct setting, and it is boring: enough retries with backoff to absorb genuinely transient blips - the simulator's transients all recover by attempt two - and a hard stop after, so real failures reach the pager while there is still night left to act in.
Self-study Why "just to be safe" feels right and is not 3 min ▶
Cranking retries to ten feels like adding a safety margin because on the transient failures it is harmless: attempt two succeeds, attempts three through ten never run, nothing visible changes. The cost only exists on the failures retries cannot fix - and those are precisely the ones where you needed the time. The lever's damage is invisible on good nights and maximal on the worst night of the month, which is the exact profile of a trap.
The general test, worth applying to any "more of a good thing" dial: name the failure mode the extra amount handles that the current amount does not. For retries beyond two-with-backoff, on a nightly batch, there is no honest answer.
The page, the storm, and the verdict ★ 22 min · everyone builds
Every lever from five sessions, plus the two that decide whether anyone finds out when it all fails anyway.
Rebuild the full ladder: DAG, retries, idempotent tasks, backfill (your passport should open it by now - the bypass is there if not). Confirm the month reads 28 on time, 4 silently wrong, history 30 of 30. Those four red mornings are the days the logic bug lived; the backfill restated the record, and nothing can un-ring the mornings themselves.
Flip "Alert on outcomes" and watch exactly one number move: time to detect the day-14 break, 9 hours 4 minutes to 12 minutes. The mornings did not change - alerting never fixes anything. It converts a failure a person discovers at 11am into one an engineer starts fixing at 2:12am, with five hours of night still in hand.
Now the anti-lever: "Retry everything 10x". Watch the board first - on time 28, wrong 4, not a single square changes color, because everything transient was already recovered by attempt two. Then read the detection counter: 12 minutes has become 3 hours 28 minutes. Ten attempts at a schema break ran their full backoff schedule before the run was allowed to die and page anyone.
Kill a task with the anti-lever on and read the trace: attempt after attempt against code that cannot succeed. Then say the trade out loud: the extra eight retries fixed zero additional mornings this month and moved the worst night's alarm three and a half hours closer to the deadline. Safety theater, measured.
Run the final scorecard below on Daybreak's build as it now stands - then keep it. It is the ship-or-not test for every scheduled pipeline you own.
Ship it, or not yet
| Check | Daybreak tonight | The standard |
|---|---|---|
| An outcome deadline exists and pages | ✓ no-dashboard-by-7am fires at 2:12am | The page names the broken promise, not a process |
| Every alert is actionable | ✓ one rule, zero restart noise | A page that merits a robotic response is deleted |
| Retries are bounded with backoff | ✓ two attempts, then fail loudly | A retry cap you can defend failure-mode by failure-mode |
| The backfill path is tested, not believed | ✓ days 10-13 restated this month | A restatement that has actually run, recently |
| The four morning outcomes are reviewed | ✗ nobody counts silent-wrong mornings yet | A monthly look at the calendar, not just the incidents |
Reading the verdict: every ✗ is a reason the pipeline is not done, whatever tonight's demo looked like. Daybreak ships with one honest gap - and now it is written down with an owner, which is the difference between a gap and a surprise.
After the course 30 min
- Write the outcome deadline for the pipeline you brought in session 1, as one sentence: WHAT must exist, HOW fresh, by WHEN, or WHO gets paged. If your orchestrator supports deadline alerts, wire it; if not, a scheduled freshness query and one alert rule is a fine start.
- Audit its retry settings against the failure modes you have actually seen. For every retry beyond two-with-backoff, name the real failure it handles. Delete the ones with no answer.
- Run the final scorecard on it, honestly. Share the ✗ rows with whoever owns the pipeline with you - a written gap with an owner beats a green demo every time.
Three questions
1. Which alert belongs on a pager, by the SRE book's own test?
Every page should be actionable, and every response should require intelligence. The restart merits a robotic response (wait), so it fails the test - and worse, it trains the team to ignore the pager that will one day say something true.
2. Cranking retries from 2 to 10 changed exactly one number in the simulator. Which, and why?
Every transient blip already recovered on attempt two, so attempts three through ten fixed nothing. On the one failure retries cannot fix, they burned three and a half hours of night before the run was allowed to die and page someone.
3. What is a retry budget, per the cascading-failures chapter?
The SRE book's example allows a process 60 retries per minute and then fails requests outright, because unbounded retries multiply across layers and can destabilize the whole system. Bounded, backed-off, then loud - that is the discipline.
Answer all three to see your score.