How this course works
Eight sessions, 45 minutes each, and one platform you architect the whole way through. The platform belongs to Daybreak, the coffee-subscription brand from the SQL and warehouse courses. Three years old, one Postgres, dashboards querying production, and a founder who has started saying the word "scale" in meetings.
★ What you walk out with tonight. Daybreak's four design meters with real numbers against them, a capacity and cost estimate you did on paper in under ten minutes, and the first page of a design document that sessions 2 through 8 will keep filling in.
LiveThis course is the decision, not the build2 min▶
Four courses on this shelf touch the same platform and none of them do this job. It is worth thirty seconds to make sure you are in the right room:
- learn-ai-infra-with-phoebe is what the layers are and what they cost. An inventory and an investment case.
- learn-data-engineering-with-phoebe is how to build the ingestion-to-serving pipeline, hands on real DuckDB.
- learn-dataops-with-phoebe is how to test, deploy and operate the thing once it exists.
- This course is how to choose between them, under a budget, and defend it to a room. It stops at the design review, and the design review is session 8.
If you have never built a pipeline, this course still works. You will be reasoning about trades, not writing DAGs. If you have built ten, this course is probably the one you have been missing, because nobody teaches the argument.
Self-studyThe artifact: one design document, in eight passes3 min▶
| Session | What your design doc gains |
|---|---|
| 1 · tonight | The problem statement, four measured constraints, and a capacity estimate |
| 2 | Service and domain boundaries, plus the contract each one publishes |
| 3 | The storage and serving decision, with the freshness you are giving up named out loud |
| 4 | The orchestration topology and the data SLAs it has to hold |
| 5 | Failure design: blast radius, what retries do and do not cover |
| 6 | The governed shell - access, lineage, PII boundaries, retention, cost attribution |
| 7 | Where the model, the index and the agent loop sit, with budgets on each |
| 8 | The ADR set, and a review you survive |
If you only do three, do 1, 5 and 8. Session 1 is the requirement, session 5 is the trade, session 8 is the defence. Everything else is the middle of the argument.
A complaint is not a requirement, and the gap is where designs die
6 min live"The dashboard is slow" is four different problems wearing one sentence, and each one has a different correct answer. Until you know which, every architecture you draw is a guess with confidence.
LiveThe four questions that turn a complaint into a design brief4 min▶
Ask these four, in this order, and write the answers down with units. If you cannot get a number, write down the guess and the name of the person who has to confirm it.
- How fast, at the ninety-fifth percentile? Not "fast". Not the average. The average latency of a dashboard is the experience of nobody, because the slow loads are the ones people remember and complain about.
- How old is the data allowed to be? This is the question people find hardest, because the honest answer is usually much older than the answer they give first. "Real time" almost always means "before I get in tomorrow."
- What is it worth per month? A design with no budget is a wish. If nobody will tell you the number, propose one and watch how fast a number appears.
- What breaks if this is wrong, and who finds out first? This is blast radius, and it is the meter that separates people who have been on call from people who have not.
The trick with the freshness question: do not ask "how fresh does it need to be". Ask "what would you do differently if this number arrived an hour earlier". If the answer is nothing, you have just saved yourself a streaming pipeline.
"The revenue dashboard is slow and finance does not trust it. Can we move to real-time? I think we need Kafka."
Three problems and a solution, arriving in one sentence, in the wrong order. If you build what was asked for you will ship a streaming platform that is still slow and still disagrees with finance.
"Revenue by SKU, opened at 08:40 by four people. p95 under 2 seconds. Data may be up to 12 hours old - they compare to yesterday, never to this hour. Budget is under $2,800 a month. If it is wrong, finance restates the board pack, and they find out three days later."
Four meters, one owner, one consequence. Kafka never comes up, because nothing in that paragraph asked for it.
Back of envelope, before any technology has a name
6 min liveMost architecture arguments are settled by one multiplication that neither side did. Ten minutes with a pen removes entire options from the table, and it removes them in a way nobody can argue with.
LiveLittle's Law, the one formula worth memorising3 min▶
L = λW. The number of items in a system equals the arrival rate times the time each one spends inside. That is it, and it catches more bad designs than any diagram.
Worked, on Daybreak's proposed queue: 5 messages a second arriving, each taking 3 seconds to process. L = 5 x 3 = 15 messages in flight. Fine. Now the consumer slows to 30 seconds because someone added an enrichment call to a model: L = 5 x 30 = 150. Still fine. Now the model provider throttles you and W becomes 300 seconds: L = 1,500 in flight, and your queue depth alarm - if you have one - is about to be the only thing that tells you.
Whenever anyone proposes putting a queue, a buffer or a retry loop in a design, ask what W becomes on the worst day. If λW is bigger than the buffer, the design has a cliff in it, and cliffs do not appear in diagrams.
Self-studySLI, SLO and error budget - the vocabulary you will be graded on4 min▶
- SLI - the indicator. A number you actually measure. "p95 dashboard load time."
- SLO - the objective. The line you drew on it. "p95 under 2 seconds, 99 percent of business days."
- Error budget - the one percent you are allowed to miss, expressed as permission. This is the part people skip, and it is the useful part: it turns "never fail" into a spendable amount, which is what makes it possible to ship anything at all.
- SLA - the same objective with money or a contract attached. Most internal platforms should have SLOs and no SLAs, because an SLA with no penalty is a wish with paperwork.
Google's SRE book is the source for all four, and its most useful claim is the unpopular one: an SLO of 100 percent is a design error, because it removes every mechanism you had for deciding what to work on next.
The seeded sentence above is 31 words and contains zero numbers, zero owners and zero consequences. It could be about any company on earth. That is the tell: a design document that would survive a find-and-replace of the company name is not a design document, it is a mood.
Write Daybreak's first page, then price it
★ 22 min · everyone buildsYou are going to write five short blocks. Not prose - five blocks, each of which is mostly numbers. This page is the thing sessions 2 to 8 keep adding to, and in session 8 it is the thing you defend.
Problem. Two sentences, in the language of the person who complained. Who is hurting, when, and how you would know it stopped. No technology words are allowed in this block at all.
Constraints. The four meters with units: p95 latency, maximum data age, monthly budget, and blast radius. Plus the constraints you did not choose - the team is three people, the company is on one cloud, there is no on-call rota.
Options. At least three, and one of them must be "do nothing, and here is what that costs". If you cannot write a fair version of an option you dislike, you do not yet understand it well enough to reject it.
Decision. One paragraph. Which option, and the single sentence of reasoning that would survive being quoted back at you in a year.
Consequence. What gets worse. Every real decision makes something worse, and the doc that admits which thing is the doc people trust.
Write the problem block for Daybreak. Here is the raw complaint: "Revenue by SKU takes forever, and last Tuesday it was blank all morning. Also finance says our number is wrong." That is three problems. Write two sentences that name only the one you are solving first, and say why that one.
Fill in the four meters. p95 latency, maximum data age, monthly budget, blast radius. If you do not know a number, write your guess and the name of the person who confirms it. A named guess is a design input. An unnamed one is a liability.
Do the arithmetic. 18,000 orders a day, 6 rows each, 400 bytes a row. Rows per year, bytes per year, peak writes per second at a 4x lunch peak. Write the three numbers down. They should take you under three minutes.
Now price the options below. The meter starts at Daybreak's current bill. Every architecture decision on the list is a real line item, and the numbers are the same ones the session 5 simulator uses. Switch on the ones you would propose for a dashboard four people open at 08:40.
Switch on "just buy a bigger box" last. Look at what it costs against what marts cost. Then look at the fact that it moves no other meter at all. This is the single most commonly proposed option in real rooms, and this is why it keeps losing.
Write the consequence block. Whatever you chose, one thing got worse. For most of these it is data freshness - you are about to make Daybreak's numbers a day old in exchange for a dashboard that loads. Write that sentence honestly. Session 3 is entirely about that trade.
Read it out loud in under 90 seconds. If you cannot, it is not one page yet. Cut the option you were never going to choose and the paragraph that was there to sound thorough.
If you take one thing from tonight. Write the four meters down with units before you draw a single box. Every argument you will have for the next seven sessions is an argument about which of those four to spend, and you cannot have that argument without the numbers.
Before session 2
25 min- Pick a real system you are responsible for or annoyed by. It does not have to be a data platform. A report, a service, a spreadsheet everyone depends on. That is your parallel project for the whole course.
- Write its four meters. p95, maximum acceptable data age, monthly cost, blast radius. Where you do not know, write the guess and the name of the person who could tell you. Then actually go and ask one of them.
- Do the arithmetic once. Volume per day, size per item, peak rate. Find the multiplication nobody has done and do it.
- Ask one person the freshness question properly. Not "how fresh does this need to be" but "what would you do differently if it arrived an hour earlier". Write down the answer verbatim. It is usually the most useful sentence in the whole design.
- Draft the problem block only. Two sentences, no technology words. Bring it to session 2, because session 2 is about drawing the line around it.
Three questions
Clear all three and session 1 gets stamped in your passport.
1 · A stakeholder says "the dashboard is slow and we need real-time". What is the first thing you do?
"Slow" and "real-time" are two different meters and they were said in one breath. Latency is fixed with marts and column stores. Freshness is fixed by loading more often. Doing the second when the problem is the first ships an expensive system that is still slow.
2 · Why design to p95 rather than to the average?
Dean and Barroso's "The Tail at Scale" made the argument at Google scale: once a request fans out across components, the slowest one sets what the user feels. A mean that looks healthy can hide a tail that is the entire reason someone opened a ticket.
3 · Your design doc has four options and every one of them is described favourably except the one you dislike, which gets two dismissive lines. What is the problem?
A design review is not persuasion, it is evidence. The fastest way to lose a room is to be caught strawmanning the option someone in it proposed. Writing the fair version costs you five minutes and is also, occasionally, how you change your own mind.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map. Certificates, videos and assessments stay with the official providers.
Session 1 cheat sheet · pin this
Tonight you wrote down what the system has to do. Session 2 draws the lines: which team owns which piece, what each one promises the others in writing, and why the correct place for a schema change to fail is the producer's build, not your dashboard at 08:40 on a Tuesday.