learn-data-reliability-with-phoebe / Builder session 1 of 8
Learn Data Reliability with Phoebe · Builder session 1 of 8

SLIs for one pipeline

Every reliability conversation that goes badly goes badly for the same reason: somebody made a promise before anybody made a measurement. Tonight is the measurement. You take one pipeline, decide what a good event and a bad event are, write the three indicators that the service-reliability literature says a data pipeline needs, and run them over thirty nights of real history. Harborlight's team believed their orders pipeline was "basically 99 percent". It was 95.14, and the difference is a month of unkept promises.

🟢 Foundational Data and analytics engineers Anyone who has been asked "is the data reliable" 45 min
0-3 · Welcome 3-22 · Good events, bad events, three indicators 22-40 · Build-along: measure one pipeline 40-45 · Q&A
Part 0

How the builder track works

Eight sessions, 45 minutes each, one artifact: a reliability contract for one pipeline that you have measured, promised, alerted on and reported. Tonight the indicators. Then the target in session 2, the error budget and burn-rate alerts in session 3, runbooks in session 4, backfills in session 5, the incident package in session 6, toil in session 7, and in session 8 the whole contract is replayed over thirty nights on call. The pipeline throughout is orders at Harborlight, a marketplace whose hourly partitions feed a settlement report that somebody has to sign.

Live = presented in the 45 minutes Self-study = the full depth, after class 🔍 Every diagram zooms

★ What you walk out with tonight. Three working indicator queries for a pipeline you own, using the freshness, correctness and coverage menu. The distinction between what you decided to measure and how you happened to measure it, which is where most indicators quietly lie. And a single honest number for last month that nobody in your team currently knows.

LiveWhy "good event or bad event" before anything else3 min▶

An indicator is a ratio: good events over total events. Everything downstream - the target, the budget, the alert, the report - is arithmetic on that ratio. So the only design decision that matters tonight is what counts as one event and what makes it bad. Get that right and the rest is a spreadsheet; get it wrong and the rest is a spreadsheet about the wrong thing.

For a request-driven service the event is obvious: one request. For a pipeline it is not, and the wrong answer is "one run per night", because a run that lands 23 of 24 partitions is not the same failure as one that lands none. Harborlight's event is one hourly partition, which gives 720 events a month and enough resolution to see a partial load as partial.

Real world

Harborlight reported "27 of 30 nightly runs succeeded" and called it 90 percent. Counting partitions, the same month was 685 of 720 good, or 95.14 percent. Same pipeline, two numbers, and neither team had noticed they were measuring different things until the settlement report was late on a night the run had "succeeded".

Self-studyThe vocabulary for the whole track4 min▶
  • SLI - service level indicator, a carefully defined quantitative measure of some aspect of the service. Tonight.
  • SLO - service level objective, a target value or range for an SLI. Session 2.
  • SLA - service level agreement, a contract with consequences for missing the SLO. If there is no explicit consequence, it is an SLO with a grander name.
  • Error budget - one hundred percent minus the SLO, expressed as the number of bad events you are allowed in the window. Session 3.
  • Burn rate - how fast the budget is being spent relative to the rate that would exactly exhaust it at the end of the window. Session 3 pages on it.
  • Specification versus implementation - what you decided to measure, against how you happened to measure it. The gap between them is where indicators lie. Part 2.
  • Window - the period over which the ratio is computed. Thirty nights here, rolling, because a calendar month resets the budget at midnight on the first and teaches everybody to ship on the second.
Part 1 · covers the SLI menu for data processing

Three indicators a pipeline needs, and one it borrows from storage

9 min live

The service-reliability literature gives data processing systems a short menu, and the shortness is the point. Freshness, correctness and coverage between them describe every way a pipeline can let a reader down. Durability is borrowed from storage for the copy you would restore from.

Freshness proportion of data updated more recently than a time threshold bad: partition landed after its hour counts: late load, outage, partial load Correctness proportion of records in that produced the correct value out bad: partition on time, values wrong counts: corrupt batch Coverage proportion of jobs that processed above a target amount of data bad: partition landed with a shard missing counts: partial load, flapping connector Durability (storage) proportion of records written that can be read back bad: a backup that does not restore counts: none this month, tested anyway Harborlight's five incidents are all counted by the first three. That is the menu doing its job.
🔍 Click to zoom - the data-processing menu, and which incident each indicator would have counted.
LiveFreshness: on time is a property of the partition, not the run3 min▶

The indicator is the proportion of partitions that landed before their deadline. Harborlight's deadline is the end of the hour the partition belongs to plus a grace period agreed with the settlement team, and the whole indicator is one comparison per partition.

★ Try it now select count(*) as partitions, count_if(landed_at <= partition_hour + interval '60 minute') as fresh, fresh / partitions::float as freshness_sli from ops.orders_partition_log where partition_hour >= dateadd('day', -30, current_timestamp);

Two things to notice. The window is a rolling thirty days, not a calendar month. And the grace period is part of the indicator's specification: change it and you have changed what you are measuring, which is fine as long as everybody knows you did.

LiveCorrectness: the indicator that catches on-time and wrong3 min▶

Harborlight's night 15 landed every partition on time with a currency column that had arrived as text. Freshness said perfect. The settlement report was wrong for six hours. Correctness is the indicator for that class of failure, and it needs a definition of "correct" that a query can check: a type, a range, a reconciliation against a control total, or a comparison with a trusted source.

Real world

The control total that caught night 15 was already being computed by finance in a spreadsheet every morning. Turning it into a correctness indicator was one query and a column. The pipeline did not get more reliable that week; it got measured, which is what let the next six sessions happen.

Self-studyCoverage and durability: the two that are easy to skip4 min▶
  • Coverage asks whether a partition processed roughly as much data as it should have. Night 27 landed four partitions on time and correct, each with one of three shards missing. Freshness and correctness both passed. Coverage is the indicator for a partial load, and it is usually the row count against the same hour last week.
  • Durability comes from the storage menu: the proportion of what was written that can be read back. For a pipeline it means the restore point. Nobody counts a bad durability event until the night they need one, so the indicator is a restore test on a schedule rather than a query over history.
  • Why not throughput? It is on the menu and it matters for capacity, but it does not describe a way the reader was let down. Fewer indicators with business meaning beat more indicators with dashboards.
Part 2 · covers specification against implementation

What you decided to measure, and how you happened to

8 min live

The indicator's specification is the outcome you care about, stated without reference to any system: partitions available to the settlement team within an hour. The implementation is where you read it from: the orchestrator's success flag, the warehouse's load timestamp, a probe that queries the table, or the settlement team's own experience. Same specification, four implementations, four different numbers.

Specification: partitions readable by settlement within the hour orchestrator success flag cheap, already there blind to partial and wrong "27 of 30" came from here warehouse load timestamp per partition, cheap blind to wrong values the freshness query above reader-side probe queries as the reader would sees fresh, correct, covered costs a query an hour the settlement team says the truth, eventually days late, no history what an SLA is measured on Closest to the reader wins. The success flag is the cheapest implementation and the one that lied.
🔍 Click to zoom - one specification, four implementations, four numbers.
LiveThe implementation that lied, and the one that cost a query an hour4 min▶

The orchestrator's success flag is where nearly every team starts, because it already exists. It measures whether the job exited zero, which is a fact about the job. The specification is a fact about the reader. Night 27's partial load exited zero four times; night 15's corrupt batch exited zero six times. Ten bad partitions the flag counted as good, which is the entire gap between "basically 99" and 95.14.

The reader-side probe is the honest implementation: a scheduled query that asks the table what the settlement team would ask it, an hour after each partition is due. It costs something, it needs maintaining, and it is the only implementation that sees all three indicators from where the damage happens.

Self-studyWrite the specification down before the query3 min▶

One sentence, no system names: "Each hourly orders partition is readable by the settlement report, with reconciled totals and all three shards, within 60 minutes of the hour." Everything a query needs is in that sentence and nothing that a query would smuggle in is.

  • It survives a platform change. Swap orchestrators and the sentence is still true or false. The success-flag implementation would have to be rewritten and would silently measure something different.
  • It is what the leader track reports against. Nobody signs off on a load timestamp. They sign off on the settlement report being right.
  • It exposes the grace period as a decision. Sixty minutes was negotiated with the people who read the report. Change it in the sentence, not in the query.
Build-along

Measure one pipeline for last month

★ 18 min · everyone builds

Bring a pipeline that somebody reads. If you have nothing to hand, use Harborlight's partition log: 30 nights, 24 partitions a night, 720 events, with five incidents in it that you do not yet know the dates of.

Write the specification sentence. Who reads it, what they need, by when. No system names. Read it to somebody who reads the output and change it until they agree.

Choose the event. A partition, a file, a batch - the smallest unit a reader would notice missing. For Harborlight it is the hourly partition, which is why the month has 720 events rather than 30.

Build the partition log if it does not exist. One row per event: partition hour, landed at, row count, reconciled flag, shard count. Session 3's alerts and session 8's replay read this table, so its shape matters more than tonight's numbers do.

Write the three queries. Freshness as landed-before-deadline, correctness as reconciled, coverage as row count within tolerance of the same hour last week. Each returns good, total and the ratio over a rolling 30 days.

Combine them into one good-event definition. A partition is good when all three pass. Run it. Harborlight's answer is 685 of 720, which is 95.14 percent, and the 35 bad partitions cluster into five incidents on nights 4, 9, 15, 21 and 27.

Compare with what the team believed. Write both numbers down next to each other, with the implementation each came from. That pair of numbers is the opening slide of leader session 1, and it is the reason the rest of the track exists.

★ The combined indicator with events as ( select partition_hour, landed_at <= partition_hour + interval '60 minute' as fresh, reconciled as correct, row_count >= 0.9 * lag(row_count, 168) over (order by partition_hour) as covered from ops.orders_partition_log where partition_hour >= dateadd('day', -30, current_timestamp) ) select count(*) as total, count_if(fresh and correct and covered) as good, good / total::float as sli from events; -- Harborlight, last 30 nights: total 720, good 685, sli 0.9514

Do not set a target tonight. The number you just computed is the only input session 2 needs, and setting a target before you have it is how teams end up promising 99 percent to a pipeline that delivers 95.

Homework

Before session 2

30 min
  1. Get the partition log recording for one pipeline and backfill it as far as your warehouse's history allows. Ninety days is enough to set a target from; thirty is the minimum.
  2. Run the combined indicator and write down the number. Then write down what the team believed it was, and where that belief came from.
  3. Read the specification sentence to the person who signs the report. Note every word they changed.
  4. Find the ten worst events in the window and group them into incidents by hand. You will need the incident list in session 3 and the grouping is harder than it looks.
Check yourself

Three questions

Harborlight's orchestrator reported 27 of 30 runs succeeded. Counting partitions gave 685 of 720. Why do the two numbers differ?

Night 27's partial load and night 15's corrupt batch both exited zero. Ten bad partitions the flag counted as good is the whole gap between "basically 99" and 95.14.

Night 15 landed every partition on time with a currency column arrived as text. Which indicator counts it?

On-time-and-wrong is a distinct failure class. Without a correctness indicator the pipeline scores perfectly on a night the settlement report was wrong for six hours.

Why write the indicator's specification as one sentence with no system names before writing the query?

The same specification has four implementations with four different numbers. Writing it first is what lets you notice that the cheapest implementation measured the job rather than the reader.

Sources covered

What tonight maps to

✓ SRE workbook, Implementing SLOs - the SLI menu for data processing Freshness, correctness and coverage as documented, durability borrowed from storage, and the specification-versus-implementation split.
✓ SRE book, Service Level Objectives - the SLI, SLO and SLA definitions Quoted in the vocabulary card. Target selection and error budgets are sessions 2 and 3.
◐ Choosing the target Deliberately not tonight. Session 2 sets it from the number you computed here.
◐ Detection thresholds and monitors The data observability course owns how a bad partition gets noticed. Here it only has to be counted.
· Traces, spans and alerting for AI services Not covered by design. That is the AI observability course; this track is tables, partitions and the people on call for them.

Session 1 cheat sheet

An SLI is a ratioGood events over total events. Decide the event first; everything else is arithmetic.
The event is the partitionNot the run. 720 events a month sees a partial load as partial.
Three indicatorsFreshness, correctness, coverage. Durability from storage for the restore point.
Specification firstOne sentence, no system names, agreed with the reader.
The success flag liesIt measures the job. Ten bad partitions exited zero at Harborlight.
Closest to the reader winsA probe that queries as the reader would sees all three indicators.
Rolling windowThirty days, rolling. A calendar month teaches everybody to ship on the second.
95.14, not 99Measure before you promise. The gap is a month of broken promises.