learn-system-design-with-phoebe / Session 4 of 8
Learn System Design with Phoebe · Session 4 of 8

Orchestration topology

Cron knows what time it is. An orchestrator knows what finished. That one sentence is worth more than any tool comparison you will read this year, because almost every silent data failure you have ever had traces to a job that ran on schedule against inputs that were not there. Tonight: dependencies as a design decision, idempotency as a property rather than a trick, backfills that are boring on purpose, and SLAs written on data instead of on services.

🟡 Medium Anyone whose data arrives on a schedule Halfway scorecard tonight 45 min
0-3 · Welcome 3-18 · Time versus dependency, and idempotency 18-40 · Build-along: Daybreak's DAG and its data SLAs 40-45 · Q&A
Part 0

Where we are

Session 3 picked a load rhythm - 15-minute incremental micro-batches. Tonight is about the machine that runs that rhythm, and specifically about what it knows. This is the halfway point, and there is a scorecard at the end of the build-along that grades your design doc as a half-finished thing, which is what it should be.

🎫 Clear all three quiz questions to stamp your passport Live = presented in the 45 minutes Self-study = the full depth, after class

★ What you walk out with tonight. Daybreak's dependency graph on paper, one data SLA written with a clock time in it, an idempotency rule for every write in the graph, and an honest halfway grade on the document you have been building since session 1.

Part 1 · covers time versus dependency

The job ran. That is not the same as the job worked

7 min live

A schedule is a guess about how long the previous thing takes. It is a guess that is correct until the day it matters, and on that day nothing errors.

Cron · everything runs at a fixed time extract · normally 02:00-02:40 extract, on the bad day · 02:00-03:20 transform · starts 03:00 reads yesterday. Exits 0. Says success Orchestrator · things run when their inputs are ready extract, on the bad day · 02:00-03:20 transform · starts 03:20 correct, and 40 min late Late and correct is an incident you can see. On time and wrong is one you cannot.
🔍 Click to zoom - the cron version has no error anywhere in it, which is exactly the problem
LiveWhat "dependency-aware" actually buys you4 min▶
  • Correct ordering without guessing. Downstream starts when upstream finished, not when the clock says it probably has.
  • Failure that stops at a branch. If the payments extract fails, everything that depends on payments is skipped and marked, and everything that does not still runs. Under cron, the failure either takes out the whole chain or, worse, does not stop it at all.
  • A recorded history you can reason about. Which run produced this table, with what inputs, and did it succeed. This is the raw material for lineage in session 6.
  • SLAs that can be missed out loud. "This asset is expected by 05:10" is a statement the orchestrator can check and page on. Cron has no concept of expectation, only of instruction.

On the session 5 board this lever is worth 16 points of blast radius and $150 a month, and it also takes 4 minutes off freshness because work no longer waits for a fixed clock gap. It is the only lever on the board that improves two meters at once.

checkout extract 02:00 · 40 min payments extract 02:00 · 12 min staging typed, deduped transform star schema revenue mart SLA 05:10 refunds mart SLA 06:00 Two sources, one shared staging layer, two marts with different promises. Four numbered failure points below. None of them produce an error message.
Part 2 · covers idempotency

Idempotency is what makes "just run it again" a boring sentence

6 min live

Every retry, every backfill and every manual re-run in your future depends on one property: running it twice produces the same result as running it once. Get this wrong and your recovery mechanism is a corruption mechanism.

✗ INSERT INTO fact_orders Run 1 · 18,000 rows · $412,000 Timeout on the write. Retry fires Run 2 · 18,000 rows appended again Total: 36,000 rows · $824,000 Nothing errored. The board pack is wrong ✓ DELETE partition, then INSERT Run 1 · partition 2026-08-25 written Timeout on the write. Retry fires Run 2 · same partition replaced Total: 18,000 rows · $412,000 Run it five times. Same answer The unit of idempotency is the partition. Pick it in the design, not when the retry fails. Everything downstream of a non-idempotent write inherits the problem.
🔍 Click to zoom - the retry did what it was told, which is why the design has to make that safe
LiveThree ways to get idempotency, in order of preference4 min▶
  • Replace a partition. The run owns a time slice and rewrites it whole. Simplest to reason about, easiest to backfill, and it is what you should reach for unless something prevents it.
  • Merge on a key. Upsert by a natural or surrogate key. Correct, slightly more expensive, and it needs a key you genuinely trust to be unique - which is a contract question from session 2.
  • Deduplicate on read. Write freely, resolve duplicates in the query. Sometimes the only option with an append-only source, but it moves the cost to every reader forever, and readers forget.
The design rule

Write down the unit of work for every node in the graph: "one calendar day of orders", "one hour of events". If two nodes disagree about the unit, backfilling one of them will silently half-fill the other. This is the most common cause of a backfill that looks like it worked.

Self-studyTask-centric versus asset-centric orchestration4 min▶

Two shapes, and the difference is what the graph's nodes are.

  • Task-centric (the classic Airflow shape). Nodes are things to do. "Run this script, then that one." Flexible, familiar to anyone who has written a shell pipeline, and the graph knows nothing about what the tasks produce. Freshness questions have to be answered outside it.
  • Asset-centric (the Dagster shape). Nodes are things that exist - tables, files, models. The graph knows that fact_orders exists, when it was last materialized, and what it depends on. "Which assets are stale" becomes a question the system can answer rather than one you ask a person.

Neither is correct in general. The useful test: how often do you need to know whether a specific table is currently trustworthy? If that question comes up weekly, the asset shape is doing work for you. If your pipeline is five steps and one output, the task shape is less machinery for the same result.

The product docs own the click-through. What this session owns is the choice, and the choice is about which question you ask most.

Self-studyBackfills, late-arriving data and the reprocessing window3 min▶

Three design decisions people make by accident and then live with for years:

  • How far back can you reprocess? If raw data is retained for 30 days, your maximum backfill is 30 days, and you have just capped your ability to fix any error older than a month. That is a design decision whether or not anyone made it deliberately.
  • What happens to a record that arrives three days late? Does it land in the partition of the day it happened, or the day it arrived? Both are defensible. Only one of them is written down, and the other one produces a permanent quiet discrepancy with finance.
  • Can a backfill run alongside the normal schedule? If not, every correction is an outage, and people will avoid correcting things.
Build-along

Draw Daybreak's graph, then write one promise on it

★ 22 min · everyone builds

Five nodes, two sources, two marts. You are going to write the unit of work and the promise for each one, and then grade the whole document you have been building since session 1.

Nodes. Name the things that exist, not the scripts that run. "fact_orders", "revenue mart", "staging orders". If a node's name is a verb, you are drawing a task graph by accident.

Edges. Draw them from what a node reads, not from what order things happened to be built in. Every edge you draw is a blast-radius path, and every one you can delete is worth deleting.

Units. One calendar day, one hour, one full snapshot. Write it on every node. Two adjacent nodes with different units is a bug waiting for a backfill.

Promises. A clock time and a business-day qualifier on the nodes people actually depend on. "Revenue mart, correct and complete by 05:10 on business days." Two or three promises, not twenty.

Recovery. For each node: what does a re-run do, how far back can you go, and who is allowed to press it. If the answer to the last one is "one person, and they are on holiday", that is your real availability number.

Draw your own graph as nouns. Five to ten nodes. Do it on paper. If you cannot fit it on one page, that is a finding about the system, not about the page.

Circle the shared node. The one with the most inbound and outbound edges. That is your highest-blast-radius asset, and it should be the best tested thing you own. Usually it is the least tested.

Write the unit of work on every node. Then check two adjacent nodes for disagreement. Roughly a third of real graphs have one, and it is invisible until a backfill.

Write exactly one promise, with a clock time. Then go and check whether it is currently true over the last 14 runs. This is the highest-value 20 minutes in the homework.

Test one write for idempotency, on paper. If this ran twice, what is in the table? If the answer is "twice as much", you have found the corruption mechanism before it found you.

Now grade your design doc below. It is meant to be half-finished - you are four sessions in. The scorecard is looking for whether the bones are honest, not whether it is complete.

Your four meters have units and numbers, not adjectives "p95 under 2 seconds" rather than "fast". A guess with a named owner counts.
The problem block contains no technology words Two sentences about who is hurting and when. If a product name appears, it is a solution block wearing a problem block's clothes.
At least one option you are rejecting is written fairly Good enough that its advocate would recognise it. This is the part reviewers test.
The consequence sentence names what got worse Freshness, cost, complexity, something. A decision with no downside is a decision you have not finished analysing.
Every published boundary has an owner who can be woken up A team and a rota, not an individual. From session 2, and the thing session 5 is about to lean on hard.

If you take one thing from tonight. Add one data SLA with a clock time to something people depend on, and make something watch it. Nearly every quiet data failure is an expectation that existed in people's heads and nowhere in the system.

Homework

Before session 5

30 min
  1. Draw your real graph as nouns, on one page. Bring it to session 5 - the simulator's blast-radius meter will mean much more if you have a picture of your own graph in your head.
  2. Check one promise against 14 real runs. What time does your most depended-on table actually land? Compare against what everybody believes. Write down both numbers.
  3. Find one non-idempotent write. There is at least one. Work out what a double run would do, and how you would notice.
  4. Answer the late-data question in writing. A record arrives three days late. Which partition does it land in, and is that written down anywhere? If it is not, you have just found a permanent quiet discrepancy.
  5. Note your maximum backfill window. How far back can you reprocess, given what you retain? That number is a cap on every future correction.
Check yourself

Three questions

Clear all three and session 4 gets stamped in your passport.

1 · The upstream extract takes 80 minutes instead of its usual 40. Under a fixed cron schedule, what happens?

Cron knows what time it is, not what finished. The job did what it was told. Nothing in the run produced an error, which is precisely why this failure is discovered by a human reading a wrong number rather than by an alert.

2 · A load times out after writing, and the retry fires. Which design makes that safe?

Partition replace is the simplest idempotency to reason about and the easiest to backfill. An alert catches the damage after the fact. Disabling retries removes your recovery mechanism because a write was designed badly - fix the write.

3 · What makes an asset-centric orchestrator worth its extra machinery?

Nodes are things that exist rather than things to do. If you frequently need to know whether a specific table is trustworthy right now, the asset shape answers it. If your pipeline is five steps and one output, it is machinery you do not need yet.

Sources covered

What this session teaches, and where it came from

✓Cron versus dependency-aware orchestrationWorked on a real late-extract scenario where nothing errors. Worth 16 points of blast radius and $150 a month on the session 5 board
✓Idempotency as a design propertyPartition replace, merge on key, deduplicate on read - in order of preference, with the doubled-revenue failure shown explicitly
✓Units of work and the backfill trapAdjacent nodes with different units silently half-fill during a backfill. The most common cause of a backfill that appears to have worked
✓SLAs written on dataA clock time plus a business-day qualifier, checked by the orchestrator rather than believed by people
✓Task-centric versus asset-centric shapesAirflow's DAG-of-tasks against Dagster's DAG-of-assets, decided by how often you need to know if a table is trustworthy
◐Airflow and Dagster product specificsOperators, sensors, resources and the UI stay with the vendor docs. This session owns the choice between the two shapes

Every fact on this page, and its verification tier, is recorded in the course's source map. Certificates, videos and assessments stay with the official providers.

Session 4 cheat sheet · pin this

The sentenceCron knows what time it is. An orchestrator knows what finished.
Late beats wrongLate and correct is a visible incident. On time and wrong is an invisible one.
Nodes are nounsIf a node's name is a verb, you drew a task graph by accident. Edges are blast-radius paths.
Unit of workWrite it on every node. Two adjacent nodes disagreeing is a backfill that half-works.
Partition replace firstThen merge-on-key, then dedupe-on-read. The last one taxes every reader forever.
A promise needs a clock"Correct by 05:10 on business days", watched by the system. On a wiki it is just a sentence.