Where RetailPulse is tonight
Recap: RetailPulse v0.2 is a scheduled, backfillable Airflow DAG - extract, clean, load, running daily, idempotent by date. The code is tested and the pipeline is orchestrated. But the pipeline trusts its input blindly. If the upstream sales export ships quantity = -4 or drops the region column, RetailPulse cheerfully processes it and every dashboard downstream inherits the rot. Code tests cannot catch this, because the code did exactly what it was told. You need tests that look at the data.
Testing code vs testing data 6 min live
A unit test checks logic that is fixed: given this input, does the function return that output. It runs once and stays true. A data test checks something that changes every single day: the actual rows flowing through the pipeline. Both are essential, and they catch completely different failures.
LiveWhy data needs tests your code tests will never catch3 min▶
Your test_pipeline.py from b1 proves that given a null order id, clean_sales drops it. That is a fact about your code, and it stays true forever. It says nothing about whether today's file actually arrived, whether prices are suddenly negative, or whether the upstream team renamed a column overnight. Data fails in ways code cannot predict:
- Nulls where you need values: a required field arrives empty because an upstream form changed.
- Out-of-range values:
quantity = -4,price = -19.99- legal numbers, illegal for a sale. - Freshness: the file simply did not land today; the pipeline happily reprocesses yesterday and everything looks green.
- Volume: a partial export delivers 200 rows instead of the usual 50,000 - no error, just quietly wrong totals.
- Schema drift: a column is renamed, dropped, or retyped upstream. Your code did nothing wrong; the shape underneath it moved.
The negative-quantity return. A retailer's source system started encoding returns as negative quantities. The code had no bug - it summed exactly what it was given - so revenue quietly dropped 6% in every dashboard and nobody could find the cause for a week. A single data test, "quantity >= 0 or flagged as a return", would have failed the very first batch and named the problem in one line. Data tests turn week-long mysteries into 40-second build failures.
Self-studyThe five dimensions of data quality2 min read▶
A useful checklist when you write expectations for any dataset - not just RetailPulse:
| Dimension | Question it asks | RetailPulse example |
|---|---|---|
| Completeness | Are required values present? | order_id, region never null |
| Validity | Are values in the allowed set/range? | quantity >= 0, region in known list |
| Uniqueness | Are keys actually unique? | order_id has no duplicates |
| Freshness | Is the data recent enough? | max(order_date) is today or yesterday |
| Volume | Is the row count sane? | within, say, 20% of the trailing average |
You will not test all five on every column. Pick the ones whose failure would actually hurt a decision - that is data-centric AI thinking applied to a pipeline.
Great Expectations and dbt tests 6 min live
Two tools own this space. Great Expectations validates any DataFrame or table against a suite of expectations and gates the pipeline. dbt tests assert on models inside a SQL transformation layer. They overlap; knowing when each fits saves you from bolting on the wrong one.
LiveGreat Expectations vs dbt tests - when each fits3 min▶
Great Expectations (GE) is a Python-native validation library. Three words carry it:
- Expectation: one assertion about data, e.g.
expect_column_values_to_not_be_null("order_id"). Human-readable, documents itself. - Suite: a named bundle of expectations for a dataset - the full contract for RetailPulse's cleaned table.
- Checkpoint: the runnable gate that validates a batch against a suite and returns pass/fail. This is what your DAG and CI call.
dbt tests live inside a dbt SQL project and assert on models with four built-ins - not_null, unique, accepted_values, and relationships (foreign-key style). They are perfect when your transformations are already SQL models in a warehouse.
When each fits: use GE when you are validating files/DataFrames in a Python pipeline (RetailPulse, tonight). Use dbt tests when your logic lives in dbt models in a warehouse. Many teams run both - GE at ingestion, dbt tests on the modeled layer. They are complements, not rivals.
| Expectation for RetailPulse | Asserts |
|---|---|
| order_id not null & unique | every sale has one identity, no duplicates |
| quantity >= 0 | no negative quantities (returns handled separately) |
| price >= 0 | no negative prices |
| order_date not in the future | no time-travel rows from a bad clock |
| region in {North, South, East, West} | no typo'd or unknown regions |
| row count within expected band | partial export or duplicate load caught |
Self-studyExpectations are documentation that runs2 min read▶
The quiet superpower of an expectation suite is that it doubles as living documentation. expect_column_values_to_be_between("price", 0, 10000) tells the next engineer exactly what "valid" means for that column - and unlike a wiki page, it fails loudly the day reality drifts from the description. GE can even auto-generate a "Data Docs" HTML site from your suite, so the docs and the tests are literally the same artifact. When you write a data test, you are writing the spec everyone forgot to write.
Data contracts 4 min live
A data test protects you from bad data. A data contract prevents it - by making the agreement between whoever produces the data and whoever consumes it explicit, versioned, and enforced by the build. When the producer breaks the contract, their pipeline fails, not your dashboard three days later.
LiveWhat a data contract actually is3 min▶
A data contract is a written, versioned agreement about a dataset, checked into Git like any other code. It has four parts:
- Schema: the columns, their types, and which are required. "order_id: string, required; price: float, required."
- Quality expectations: the same assertions from your GE suite - not-null, ranges, allowed values - promised, not just hoped.
- Owner: a named team or person responsible. A dataset with no owner has no one to call when it breaks.
- SLA: the freshness and availability promise - "delivered by 5am daily, no more than 2 hours late."
The point is enforcement: the contract runs in CI. If the producer changes the schema or violates an expectation, their build fails and they find out immediately - instead of the consumer discovering it in a board meeting. This is Zhamak Dehghani's data-mesh idea in miniature: producers own their data quality, and the contract is how that ownership is made real.
Self-studyContracts, schemas, and the tools around them2 min read▶
Data contracts are a young, fast-moving space. You will see them expressed as plain YAML (what we use tonight), as JSON Schema, via the open Data Contract Specification, or through platform features in tools like dbt's model contracts and catalogs like DataHub or OpenMetadata. The format matters less than the discipline: a machine-readable spec, versioned in Git, enforced in CI, with a named owner. Once you have that, swapping YAML for a fancier standard later is a refactor, not a rethink. Do not wait for the perfect tool - a checked-in YAML that fails the build beats a Confluence page every time.
A Great Expectations suite on clean_sales ★ 12 min · everyone builds
Build the suite that encodes RetailPulse's real invariants and run it against the clean_sales.parquet your b2 DAG produces. You will validate not-null, uniqueness, ranges, allowed values, and freshness - and watch it pass on good data, then fail loudly on a poisoned row.
Load the cleaned parquet into a pandas DataFrame and wrap it as a GE validator (or use a GE pandas datasource pointed at the file).
Write the expectations from Prompt A: order_id not-null and unique, quantity and price non-negative, order_date not in the future, region in the allowed set, row count in a band.
Save the suite under a name like retailpulse_clean_sales and run it. On clean data every expectation should be green.
Poison one row on purpose - set a quantity to -3 - and re-run. Watch that single expectation flip red and report exactly which rows failed and why.
Commit the suite into RetailPulse under quality/. It is now a versioned, reviewable artifact - the spec for what "clean" means.
Quality gate in CI plus a contract file ★ 10 min · build your own
A suite that only runs when you remember it is useless. Now you wire the checkpoint into the Airflow DAG and CI so a breach fails the run, write a data contract YAML, and watch a pull request go red when an expectation breaks - exactly where you want to find out.
Add a validate task to the b2 DAG, between transform and load, that runs the GE checkpoint (Prompt B). If it fails, the task raises and load never runs - bad data cannot land.
Add the same checkpoint call to ci.yml as a step, so a PR that changes the pipeline is validated against sample data before it can merge.
Write contracts/clean_sales.yml from Prompt C: schema, expectations, owner, SLA. Check it into Git next to the code.
Open a PR that deliberately breaks an expectation - remove the region validation from the data, say - and watch CI go red with a message naming the failed expectation.
Fix it, see green, merge. RetailPulse is now v0.3: orchestrated, and quality-gated with an enforced contract.
The rename that failed at the source. An upstream team renamed price to list_price in their export. Before contracts, that broke a dozen dashboards silently and took three days to trace. After the contract went into CI, the producer's own build failed the moment they made the change - with a message saying "clean_sales contract: required column price missing." They fixed it in ten minutes, before it ever reached a consumer. The failure moved from the victim to the cause. That is the whole point.
Try it yourself - this week ◐ 40-55 min total
- Finish both demos if you did not live - especially the red PR. Everyone should watch a failed expectation block a merge at least once.
- Add a freshness expectation that fails if
max(order_date)is more than a day old - the classic "the file never arrived" catch. - Add a volume expectation tuned to your sample data's trailing average, then simulate a partial export (drop 80% of rows) and watch it fire.
- Generate GE Data Docs from your suite and open the HTML. Notice that your tests are now also your documentation.
- Optional reading: the Great Expectations "Get Started" tutorial and the dbt tests reference - now a review of what you already built.
What this session covers
This track teaches DataOps on an open-source stack from the tools' official docs, with AWS mapping notes. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Why is a green unit test not enough to trust a pipeline's output?
Code tests prove the logic is correct and stay true forever. Data tests check the changing data on every run - the two catch completely different failures, so you need both.
2 · In Great Expectations, what is a checkpoint?
An expectation is one assertion; a suite is a named bundle of them; a checkpoint is the runnable gate that executes the suite against a batch. The checkpoint is what fails your run.
3 · What makes a data contract different from just having data tests?
Tests protect the consumer after the fact; a contract is the explicit producer-consumer agreement, enforced in CI, so the producer's build fails the moment they break it - before a consumer ever sees bad data.