Where we are
Four sessions, four decisions, each one made in isolation. Tonight they are all on the same board with the same budget, which is the first time you find out whether they add up.
★ What you walk out with tonight. A verdict on Daybreak's architecture against two different requirements, a written answer to "what happens when this fails" that does not bluff, and the single most useful sentence in this course: retries do not fix a contract break.
Blast radius is a number, and most people carry it as a feeling
6 min live"If this breaks, quite a lot of stuff breaks" is not a design input. Count the assets. Daybreak has 42 downstream tables, dashboards and models, and on day one a single upstream change reaches 82 percent of them.
LiveTransient or permanent: the distinction that decides everything5 min▶
Every failure is one of two kinds, and every mechanism in your toolkit handles exactly one of them. Getting this backwards is why so many systems have elaborate resilience and still go down.
- Transient. A timeout, a throttle, a node that went away, a lock that was held too long. The same request, sent again in ten seconds, succeeds. Retries, backoff and dead-letter queues exist entirely for this class.
- Permanent. A renamed column, a bad credential, a schema that no longer parses, a definition that changed. The same request, sent again in ten seconds, fails identically. Retrying is not recovery, it is repetition with extra logging.
Retries do not fix a contract break. They just fail five times. If someone proposes resilience for a class of failure that is permanent, the correct answer is not more retries - it is contracts, validation, or a circuit that stops trying and tells a human.
You can test this on the board below in about fifteen seconds: switch on retries, leave contracts off, and ship the schema change. Seventeen assets still break.
The tradeoff board
★ 22 min · everyone buildsNine levers, four meters, two requirement profiles. Every meter has a budget and the budget comes from the requirement at the top. Start with everything off and work up the ladder. Then, at the end, change one thing that is not a lever.
9,400 ms · $1,180 · 5 min old · 82% · 2 of 4 budgets. Daybreak on day one. Note that it already passes the freshness budget by a mile, because it is production. It fails latency and blast radius badly.
5,170 ms · $1,440 · 7 min · 76% · 2 of 4. A read replica for $260 removes 4,230 ms of lock contention. The verdict has not moved yet, but 45 percent of the latency has.
1,551 ms · $1,860 · 1,440 min · 62% · 2 of 4. The column store takes latency inside budget and simultaneously blows the freshness budget entirely. You just traded one green meter for another. This is the honest shape of the decision.
1,551 ms · $2,040 · 18 min · 62% · 3 of 4. Incremental loading buys the freshness straight back for $180. Three of four. Only blast radius is still red.
543 ms · $2,180 · 24 min · 62% · 3 of 4. Marts for $140 take another 1,008 ms off. Remember this number when you meet the trap lever.
543 ms · $2,240 · 24 min · 28% · 3 of 4. Contracts for $60 remove 34 points of blast radius in one move. Still not inside the 20 percent budget, but nothing else on the board trades like this.
543 ms · $2,330 · 27 min · 16% · 4 of 4. Retries and idempotency take blast radius inside budget. All four meters green. On this requirement, this architecture is correct.
543 ms · $2,480 · 23 min · 4% · 4 of 4. The orchestrator is the only lever that improves two meters at once - blast radius down 16, freshness better by 4 minutes, because work stops waiting for a fixed clock gap.
Climb the ladder in order. Switch the levers on one at a time, top to bottom, and read the verdict line after each. You are looking for the two moments where a meter goes red - they are the two real decisions in the whole architecture.
Stop at the warehouse rung and sit with it. Latency went green and freshness went red in the same click. Every "modernise the platform" project contains this moment, and most design documents do not mention it.
Now the trap. Turn everything off and switch on "just buy a bigger box" alone. Then compare it against marts alone. The box costs $980 and takes 1,692 ms off. Marts cost $140 and take 6,110 ms off. Seven times the money, a quarter of the result, and no other meter moves at all.
Get to 4 of 4 on the morning dashboard. Every design lever on, streaming off, box off. $2,480 a month and all four green. Write that configuration down.
Now change nothing and click "Minute alerting". The same architecture drops to 3 of 4. Nothing about the system got worse. Somebody in a meeting said the word "alerting" and one budget moved from 12 hours to 5 minutes.
Fix it for the new requirement. Switch on streaming: 4 of 4 again, at $3,380. Now click back to the morning dashboard with streaming still on - 3 of 4, because you are $580 over a budget that exists for a dashboard four people read. The same lever is correct and incorrect, and only the requirement decided which.
Run the failure drill three ways. With contracts on. With everything off. Then with orchestrator, retries and incremental on but contracts off. Read that last result carefully - 17 assets break, the retries fire five times, and the retries were never going to help.
Write the honest paragraph. "When the upstream schema changes, N assets break, we find out in M minutes, and the mechanism that catches it is X." If you cannot fill in all three, that is the gap in your design, not a gap in the exercise.
If you take one thing from tonight. "Is this a good architecture" is not a question. "Is this correct for this requirement, and what does it cost when the requirement changes" is two questions, both answerable, and both of them are what a design review is actually for.
How you know whether the failure design is working
5 min liveA design that is resilient on paper and unmeasured in production is a belief. Four numbers turn it into a fact, and they are the same four numbers your engineering organisation is probably already arguing about.
LiveThe DORA four keys, read as design feedback4 min▶
| Metric | What it tells a designer |
|---|---|
| Deployment frequency | How small your changes are. Big infrequent changes are a blast-radius decision made by process rather than by architecture |
| Lead time for changes | How much friction sits between deciding and shipping. Often it is a boundary problem, not a pipeline problem |
| Change failure rate | The direct read on your isolation. Elite performers sit near 5 percent |
| Failed deployment recovery time | Whether your rollback is real. If nobody has used it this quarter, it is a diagram |
The 2025 DORA report replaced the old four performance tiers with seven team profiles, on the argument that a single ranking hid too much about culture and constraints. The metrics themselves survived, and change failure rate remains the most direct feedback you get on whether your boundaries are real.
Chaos engineering belongs here and is out of scope for this course: it is the practice of testing the failure design deliberately rather than waiting. If you have never done it, start with the smallest honest version - turn off one non-critical dependency in a staging environment and see whether the alerts you believe in actually fire.
Self-studySilent failure is the expensive failure4 min▶
Rank your failure modes by how long they stay invisible, not by how dramatic they are:
- Loud and instant. The pipeline crashes, the page 500s. Found in minutes, fixed the same day. The cheapest failure you can have.
- Loud and delayed. The nightly load fails at 02:00 and pages someone at 04:12. Found before the business sees it, if the alert is real.
- Silent and bounded. One partition is wrong. Found when someone queries that date range. Weeks, sometimes.
- Silent and unbounded. A definition changed. Every number since the change is subtly wrong, decisions have been made on them, and the discovery is somebody's spreadsheet not tying out. This is the one that ends up in a board meeting.
The design instinct that follows: prefer failures that are loud. A job that refuses to write anything when the row count is 40 percent below the trailing median is choosing a loud failure over a silent one on purpose, and that is a good trade almost every time.
Before session 6
30 min- Count your assets. How many tables, dashboards and models are downstream of your single most-depended-on input? That count is your blast-radius denominator and most people have never had it.
- Classify your last five incidents. Transient or permanent. Then check what mechanism you have for each class. The mismatch is usually the finding.
- Write the honest paragraph. "When X changes upstream, N assets break, we find out in M minutes, and the mechanism is Y." One paragraph, real numbers.
- Find one silent-and-unbounded risk. A definition, a unit, a currency, a timezone. Something where a change would pass every check you have. Write down how you would catch it.
- Run your own profile flip. Take your design and ask what would have to change if the freshness requirement went from a day to five minutes. If the answer is "everything", you have found out how much your architecture depends on a sentence somebody said once.
Three questions
Clear all three and session 5 gets stamped in your passport.
1 · With retries, a dead-letter queue and an orchestrator all switched on, an upstream team renames a column. What happens?
Retries, DLQs, circuit breakers and bulkheads all handle transient failures. A renamed column fails identically on every attempt. The only mechanism that catches this class is a contract in the producer's CI, which is why it sits on the board as its own lever.
2 · The full design-lever set scores 4 of 4 on the morning dashboard and 3 of 4 on minute alerting. What does that tell you?
Not a single lever moved between those two verdicts. One budget went from 12 hours to 5 minutes because somebody described a different use case. This is why every design review starts with the requirement, and why "what is the best architecture" has no answer.
3 · Buying a bigger box costs $980 a month and takes 1,692 ms off p95. Marts cost $140 and take 6,110 ms off. Why does the bigger box keep getting proposed?
It moves exactly one meter and moves it least per dollar. It keeps winning because it is the only option that requires nobody to change anything. Naming that honestly in a review is more persuasive than the arithmetic alone, because it is the actual reason.
What this session teaches, and where it came from
Every simulator constant is listed in the course's source map. The board is a teaching model calibrated to make the shape of each trade visible - it is not a benchmark of your platform, and the page says so on the board itself.
Session 5 cheat sheet · pin this
The board has four meters on it. Session 6 is about the fifth thing, the one that has no meter and shows up as a letter from a regulator instead: who can see what, where the personal data went, how you would delete it, and who is actually paying for all this. Every copy of the data is a new obligation, and deletion is the one property you cannot retrofit.