Where the framework earns its place
Sessions 2 to 5 built freshness, volume, uniqueness, schema and null-rate monitors as SQL on a schedule against analytics.orders_daily at Bellwether. That works, and it stops working the moment three people own the thresholds. Tonight the checks move into Great Expectations Core: the same arithmetic, expressed as objects that live in a repo, run from a single entry point and produce a readable result. Session 7 adds lineage, and session 8 scores the whole set against 30 nights of history.
★ What you walk out with tonight. A working Checkpoint that validates last night's row of the metrics table and notifies on failure. The twelve component names in the order you actually meet them, so the documentation stops feeling like a maze. And a written list of which monitors did not translate, which is the most useful artifact of the evening because it tells you what the framework is not for.
LiveWhat a framework buys over a scheduled query3 min▶
A scheduled query is the cheapest possible monitor and it has three failure modes that show up around month two. The threshold lives in a scheduler UI where nobody reviews it. The output is a row in a log, so a failure has to be explained by whoever wrote it. And the same check gets copied to a second table with one number edited, which is how two monitors on the same signal end up disagreeing.
A framework fixes exactly those three and nothing else. The assertion is a declared object, so a change to a threshold is a diff on a pull request. The run produces a structured result and a rendered page, so a failure is legible to the person who reads it at 07:00 rather than only to its author. And the assertion is separate from the data it points at, so one definition covers twenty tables.
Bellwether had a volume check in the warehouse scheduler and a nearly identical one in the orchestrator, set six months apart by two people. The scheduler one used a plus or minus 30 percent band, the orchestrator one used same-weekday medians. On the eight weekend nights of a 30-night window they disagreed, and the on-call learned to trust neither. Neither threshold had ever been reviewed by a second person, because neither lived anywhere a review happens.
Self-studyWhat a framework does not buy3 min▶
- It does not choose your thresholds. An expectation with
max_value=32is exactly as arbitrary as a scheduled query with the same number. The arithmetic from sessions 2 to 5 is still where the value is. - It does not give you a rota. A failing validation is a result object until an Action puts it in front of a named human. That was part three of the monitor anatomy in session 1 and it is still not code.
- It does not model the channel. A framework will happily emit 25 notifications in 30 nights, and the muted channel that produces is the thing session 8 measures.
- It does not replace build-time tests. Row-level assertions inside a transform project are a different discipline with their own tooling, and that is what the analytics engineering track is for.
The Great Expectations component model
9 min liveTwelve objects, and the reason there are twelve rather than three is that each one is deliberately reusable independently of the others. Met in workflow order they stop being a maze: you connect to data, you pick a slice of it, you say what should be true, and you run that and record what happened.
LiveThe twelve objects, one sentence each5 min▶
Read the table top to bottom once. That order is the order you write the code in, which is why it is worth learning as a sequence rather than as a glossary.
| Component | What it is for |
|---|---|
| Data Context | The entry point to the Python API, holding your configuration and metadata for everything else. |
| Data Source | Represents the platform or system your data lives in and carries the connection to it. |
| Data Asset | A collection of records within a Data Source: a table, a view or a query. |
| Batch Definition | Describes how records are retrieved from a Data Asset, whole table or partitioned by a column. |
| Batch | The records actually returned when a Batch Definition is resolved. |
| Expectation | A single verifiable assertion about data. |
| Expectation Suite | A collection of Expectations grouped together for a Data Asset. |
| Validation Definition | Pairs an Expectation Suite with a Batch Definition, so it names what to check against which data. |
| Validation Result | The output of running a Validation Definition, reporting success or failure for each Expectation. |
| Checkpoint | Runs one or more Validation Definitions and triggers Actions on the results. |
| Actions | What happens after validation, such as sending a notification or refreshing Data Docs. |
| Data Docs | Human-readable documentation rendered from Validation Results. |
LiveWhy the indirection exists3 min▶
The first reaction to twelve objects is that four would do. The reason there are twelve is that the seams are placed exactly where teams need to vary one thing without touching the others.
- Expectation Suite is separate from Batch Definition so the same suite runs against many slices. One nightly-metrics suite covers every tier 1 table you instrument, because the assertions are about the shape of a metrics row rather than about a specific table.
- Validation Definition is the join between the two. It is the only object that knows both, which means moving a suite from staging to production is a new Validation Definition rather than an edited suite.
- Checkpoint is separate from Validation Definition so scheduling and notification live in one place. Ten validations, one Checkpoint, one set of Actions, one thing your orchestrator calls.
- Data Source is separate from Data Asset so a credential rotation touches one object. This is the seam that saves you on the day the warehouse password changes.
The test for whether you have used the seams. Adding a twelfth table to your monitoring should be one new Batch Definition and one new Validation Definition, with the suite untouched. If it is a copied suite with the table name edited, you have twelve sources of truth and the next threshold change will miss some of them.
Self-studyThe version caveat, stated plainly3 min▶
Great Expectations moves fast, and component names have changed across major versions. The docs read for this material were version 1.22.0 of Great Expectations Core, and the twelve names above are the ones that version uses. Older material will show objects that no longer exist under those names, and code written against a 0.x release does not run unchanged.
So the working rule, which costs two minutes and saves an afternoon: open the current documentation for the exact version in your lockfile before you teach this to anybody or deploy it anywhere. Pin the version in your dependency file rather than tracking latest, and treat an upgrade as a change that needs its own pull request rather than something a resolver does for you overnight.
The single most common way this bites is a copied snippet. A blog post from a previous major version reads as perfectly sensible, the object names look plausible, and the failure is an import error at best or a silently different default at worst. Check the version stamp on any snippet before you paste it, and if there is no version stamp, treat the snippet as a sketch of the idea rather than as code.
Expectations versus monitors
8 min liveThis is the distinction that decides how much of your monitor set can live in a framework at all. An expectation is an assertion about a batch: it looks at the records in front of it and passes or fails. A monitor is a comparison against the table's own history: it needs the last thirty nights to say anything at all. Neither is better. They answer different questions, and confusing them is how people conclude that a validation framework "cannot do freshness".
LiveWhich of the five signals fit as expectations4 min▶
Sort your checks by what they need in order to answer. Anything answerable from the records in front of you fits an expectation natively. Anything that needs the table's own past does not, until you feed the past in as the batch.
| Check | Needs | Fits an expectation |
|---|---|---|
| Validity: a column is one of a known set | The batch only | Yes, natively |
| Range: a value sits between two bounds | The batch only | Yes, natively |
| Set membership: currency, status, region code | The batch only | Yes, natively |
| Uniqueness: the key does not repeat | The batch only | Yes, natively |
| Freshness against a trailing p95 | Thirty nights of lateness | Only via the metrics table |
| Volume against a same-weekday median | Four prior same weekdays | Only via the metrics table |
| Schema diff against last night's column set | Last night's column list | Only via the metrics table |
| Null rate against a trailing mean plus 3 sd | Thirty nights of null rates | Only via the metrics table |
The top four are the ones a validation framework was designed for, and they are genuinely valuable: a hard accepted_values assertion on a region code catches a class of upstream change that no metadata monitor sees. The bottom four are what sessions 2 to 5 were actually about, and they are comparisons.
LivePoint the suite at the metrics table4 min▶
The move is one line of aim and it resolves the whole tension. Instead of a Data Asset over analytics.orders_daily, build the Data Asset over analytics.obs_metrics, the one-row-per-table-per-night table from session 1. Last night's row is a Batch of exactly one record, and every trailing comparison you need is a column in it.
That works only if the row carries the comparison as well as the measurement. When you write tonight's row, write the reference values alongside it: the trailing p95 of lateness plus the pad, the same-weekday median of row count, last night's column-set signature. Then "lateness is within budget" is late_budget_minutes greater than late_minutes, which is an ordinary column-pair assertion about a single row.
- Store the reference, not just the reading. A metrics row of eight raw numbers cannot be validated without history. The same row plus three precomputed reference columns can.
- The arithmetic does not move, it relocates. The percentile and the median are still computed in SQL, once per night, in the job that writes the row. The framework validates rather than calculates.
- One suite covers every table. Because the assertions are about the shape of a metrics row, the same Expectation Suite validates orders_daily, orders_items and every other instrumented table. That is the reuse the component seams were built for.
- The failure message improves. A Validation Result on a metrics row says which reference value was breached and by how much, which is the sentence you wanted in the incident channel in session 1.
Self-studyActions and Data Docs: the parts that reach people4 min▶
A Validation Result is an object in memory. Nothing about a failing validation reaches a human until an Action does it, and this is the same gap that made a monitor without a destination into a chart that changes colour.
- Actions run after validation, on the result. A notification Action on failure is the minimum viable destination. Configure it to fire on failure only, because an Action that posts on every successful night is how a channel gets muted in week two.
- Data Docs is the legibility layer. It renders Validation Results as browsable pages: which Expectations ran, which failed, what the observed value was against the expected one. The audience is the analyst who was told the revenue number is wrong and wants to see the evidence without reading your Python.
- Host the docs somewhere durable. Docs rendered onto the laptop of whoever ran the Checkpoint are not an artifact. A static bucket behind the same auth as your BI tool is enough, and the link belongs in the notification.
- The notification and the docs do different jobs. The notification carries the number and the link. The docs carry the detail. Putting the detail in the notification is how people learn to scroll past it.
Write the notification text yourself. A default template says a suite failed. What the reader needs is the table, the signal, the observed value, the reference value and the blast radius: "orders_daily landed 230 min past the 06:00 promise, budget is 32 min, 3 dashboards and the finance export read it". Same Validation Result, different Monday.
A checkpoint over the metrics table
★ 20 min · everyone buildsNine steps, in the order the components appear. Bring the metrics table you started in session 1. If you have nothing recording yet, the reference columns are easy to fake for one night and the shape is what matters. Python throughout, with the reference values computed in SQL by the job that writes the row.
1. Write the reference columns into the metrics row. Before any framework code, make sure tonight's row carries its own comparisons. Three extra columns, computed in the same statement that writes the row.
2. Get a Data Context. File mode keeps the configuration in a directory you can commit, which is the point of moving the thresholds into a repo in the first place.
3. Define the Data Source and the Data Asset. The Data Source carries the connection. The Data Asset is the metrics table, not orders_daily, and that choice is the whole session.
The connection string is read from an environment variable through the Data Context, so no credential lands in the committed configuration. Rotating it touches this one object.
4. Add a Batch Definition for last night's row. Partition the asset daily on load_date. The Batch Definition is a reusable description; it does not hold a date.
5. Build the Expectation Suite: three Expectations, all of them ordinary. Lateness inside the stored budget, row count inside 25 percent of the stored same-weekday median, duplicate rate under 2 percent. The thresholds are the ones sessions 2 to 4 derived, and they now live in a file somebody can review.
6. Pair them with a Validation Definition. One object, two references. This is the only place that knows both which suite and which slice, which is why promoting from staging to production is a second Validation Definition rather than an edited suite.
7. Wrap it in a Checkpoint with an Action. The Checkpoint is what your orchestrator calls. The Action is what makes a failure reach a human, and notify_on="failure" is the setting that keeps the channel alive.
8. Run it and read the Validation Result. Then run it again against night 9, which landed 230 minutes past the 06:00 promise against a trailing p95 of 22 minutes, so a budget of 32. Read what the result says about the observed value, not just that it failed.
9. Write down what did not translate, and why. Two of your sessions 2 to 5 monitors do not become Expectations tonight even with the metrics table in play, and one of them never will. Being explicit about this is worth more than the working Checkpoint.
- Schema diff. An Expectation can assert that the column count equals a stored reference, or that a column-set signature matches last night's. What it cannot produce is the diff itself: which column left and which arrived. The alert needs "ship_region dropped, promo_code added", and that sentence comes from set arithmetic in SQL, not from a pass or fail. Keep the diff query and let the Expectation carry only the tripwire.
- Null rate against a trailing mean plus 3 standard deviations. Translates only if you write the band into the row as two more reference columns, and then the honest reading is that the framework is validating a threshold your SQL computed. Worth doing for the review trail; do not mistake it for the framework doing the statistics.
- The revert. Session 4's finding that a rollback pages too is a policy decision about whether a change back is a change. No Expectation encodes that, and the bench in session 8 counts night 24 as a false page precisely because the answer is arguable.
- The channel. Nothing in the component model knows whether anybody read the notification. An Action fires and returns success. Alert fatigue is a property of the humans and the rota, which is why session 8 models it rather than measuring it.
The result to aim for after 20 minutes. One Checkpoint, three Expectations, one notification Action, one hosted Data Docs page, and a four-line note in the repo README saying which two monitors stayed in SQL and why. The note is the part a future maintainer needs, because the obvious question in six months is why there are checks in two places.
Before session 7
30 min- Pin the Great Expectations version in your dependency file and open the documentation for that exact version. Note any component name on this page that differs from what you read, because the release cadence means it will eventually.
- Get the three reference columns writing into your metrics row every night. Without them the suite has nothing to assert against, and this is the step people skip.
- Point the same Expectation Suite at a second table's metrics rows using only a new Batch Definition and a new Validation Definition. If you had to edit the suite, find out why and fix the seam.
- Host Data Docs somewhere a colleague can open, then send the link to one person who is not on your team and ask them what they think it says. Whatever they answer is the real legibility of your monitoring.
- Write the notification text by hand for one failing Expectation. Table, signal, observed, reference, who reads it. Session 7 gives you the last part of that sentence properly.
Three questions
Which component pairs an Expectation Suite with a Batch Definition?
The Validation Definition is the join, and it is the only object that knows both. A Checkpoint runs one or more Validation Definitions and triggers Actions on the results, which is a separate job.
Why can a freshness check against a trailing p95 not be written as an ordinary Expectation over orders_daily?
An Expectation asserts something about the records in front of it. Aim the Data Asset at the metrics table instead, where the trailing p95 is a stored column in last night's row, and the same check becomes an ordinary column-pair assertion.
A schema monitor is moved into the framework as an Expectation that the column-set signature matches last night's. What is lost?
A pass or fail is a tripwire. The sentence the on-call needs names the columns, and that comes from set arithmetic in SQL. Keep the diff query and let the Expectation carry the tripwire only.