Where RetailPulse stands
Six sessions in, RetailPulse is a real data product: a clean_sales pipeline writing clean_sales.parquet, an Airflow DAG scheduling it, a Great Expectations suite plus a data contract gating quality, a versioned Postgres sales table, a demand-forecast model tracked and registered in MLflow, and a FastAPI /predict endpoint in Docker with Evidently drift monitoring and a retrain trigger. CI runs on every PR. It works. What it does not yet do is tell you when it stops working.
.env, and a docker-compose.yml that stands the entire RetailPulse stack - Postgres, Airflow, MLflow - up with one command. RetailPulse v0.7.
The three signals of data observability 6 min live
Observability is the ability to answer "what is happening and why" without shipping new code to find out. It rests on three kinds of signal: logs (what happened, event by event), metrics (how much and how fast, as numbers over time), and lineage (what depends on what, so a break upstream explains a break downstream). App observability watches these for services. Data observability watches them for the data itself.
LiveApp observability vs DATA observability3 min▶
Application observability asks: is the service up, fast, and error-free? Data observability asks a harder question: is the data flowing through it correct, complete, and current? A FastAPI endpoint can return HTTP 200 in 40ms all day while quietly serving predictions off a sales table that stopped updating on Tuesday. The service is healthy; the data is rotten. The classic frame is the five pillars of data observability:
- Freshness: how recently did the data update? A stale
salestable is the most common silent failure. - Volume: did the expected number of rows arrive? A load that drops from 50k rows to 3k is broken even if nothing errored.
- Schema: did the columns or types change out from under you? The b3 contract guards this; observability watches it in prod.
- Distribution: are the values in range? Negative prices, dates in 1970, a category that vanished - these pass row counts but poison models.
- Lineage: when something breaks, what upstream source caused it and what downstream depends on it? This turns a 2am mystery into a 2-minute trace.
The dashboard that was green and wrong. A retail team's exec dashboard showed flat sales for nine days. Every service was up, every check was HTTP 200, no page fired. The upstream export job had silently stopped writing; the warehouse kept serving the last good snapshot. A single freshness metric - "hours since sales last updated" - would have caught it on day one. App observability said healthy. Data observability was the missing lens.
Self-studyMetrics, logs, traces - the app-observability triad2 min read▶
In service land the triad is metrics, logs, and traces. Data land renames the third to lineage, but the discipline is the same: emit numbers you can chart, events you can search, and dependency links you can walk. For RetailPulse tonight you build the first two by hand - structured logs and a couple of metrics - because feeling them is how you learn what a paid observability platform is actually doing for you later.
| Signal | Service question | Data question for RetailPulse |
|---|---|---|
| Logs | What did the code do? | Did the cleaning step drop rows, and how many? |
| Metrics | Latency, error rate | Freshness hours, row count, null rate |
| Lineage / traces | Which call failed? | Which source table caused the bad forecast? |
Alerting that people trust 6 min live
Signals are only useful if someone acts on them. An alert is a promise: when this fires, a real problem exists and it is worth interrupting a human. Break that promise a few times - alert on noise - and the whole team learns to ignore the channel. Then the one alert that mattered gets muted with the rest. The craft is picking a small set of alerts that map to something a user would actually feel.
| Alert | Pillar | Condition for RetailPulse | Why it is worth a page |
|---|---|---|---|
| Pipeline failed | - | The Airflow DAG task errored or did not run | No fresh data at all downstream |
| Data late / stale | Freshness | sales last loaded > 26h ago (SLA 24h + buffer) | Forecasts and dashboards go quietly out of date |
| Row count out of band | Volume | Daily rows outside 30k-70k historical band | A partial or double load skews every metric |
| Schema changed | Schema | A column dropped or a type flipped vs the contract | Downstream code breaks or corrupts silently |
| Drift high | Distribution | Evidently drift score over threshold (from b6) | Model predicts on data it was not trained for |
LiveAlert on symptoms users feel, route to the owner3 min▶
Two rules save you from the alert-fatigue trap:
- Alert on symptoms, not causes. "Sales data is 30 hours stale" is a symptom a user feels. "CPU on the worker hit 80%" is a cause that may or may not matter. Page on the first; put the second on a dashboard you look at when investigating.
- Route to the owner. An alert with no clear owner is noise by design. Each RetailPulse alert names the person or team who can fix it and links straight to the runbook. If nobody owns it, do not alert on it - delete the check or assign the owner first.
- Make every alert actionable. If the honest response to an alert is "yeah, that happens sometimes," it is not an alert, it is a log line. Tune the threshold or drop it.
- Severity tiers. Page (wake someone) for "users are affected now." Ticket (business hours) for "will be a problem soon." Log (no notification) for everything else.
Secrets and Infrastructure as Code 4 min live
RetailPulse now needs a Postgres password, an MLflow URI, maybe a Slack webhook. None of those belong in Git - a committed secret is compromised forever, because history keeps it. And the stack now has enough moving parts (Postgres, Airflow, MLflow, the API) that "set it up by hand" is not reproducible. Two disciplines fix both: secrets kept out of code, and the environment itself described as code.
LiveSecrets management and Infrastructure as Code3 min▶
Secrets management has one iron rule and a ladder of maturity:
- Never in Git. Not in code, not in config, not in a notebook cell. History is forever; a leaked key is a rotate-everything incident.
- Environment variables + a gitignored
.env. The lightweight standard: code readsos.environ["DB_PASSWORD"], the value lives in.env, and.envis in.gitignore. Commit a.env.examplewith keys but no values so teammates know what to set. - A secret manager for anything shared or production: AWS Secrets Manager, SSM Parameter Store, HashiCorp Vault. Rotation, access control, and audit come built in.
Infrastructure as Code (IaC) means the environment is described in a file you commit, not clicked together by hand. Tonight docker-compose.yml is your lightweight IaC - one file that declares Postgres, Airflow, and MLflow and brings them all up together. Terraform is the same idea pointed at the cloud: it declares the real VPC, database, and cluster so prod is reproducible too. Same principle at two altitudes.
Self-studyWhen to graduate from .env to a secret manager2 min read▶
A gitignored .env is right for local dev and small teams. Move to a managed secret store the moment secrets are shared across people or environments, or the moment one lands in production. Signals it is time: you are pasting passwords in Slack, you cannot answer "who has access to the prod DB password," or you need to rotate a key and have no idea what would break. AWS Secrets Manager and SSM Parameter Store both inject values as environment variables at container start, so your code - os.environ[...] - does not change. Only the source of the value does.
Structured logging and a freshness alert ★ 12 min · everyone builds
RetailPulse v0.7, part one: make the pipeline observable. Add structured (JSON) logs so every run is searchable, compute a freshness and a volume check against the Postgres sales table, and emit an alert to a Slack webhook (or a log stub) when the data is stale or the row count falls out of band.
Add structured logging to pipeline.py: configure a JSON log formatter and log the row count in and out of clean_sales. Now every run leaves a searchable trail instead of a bare print.
Write monitor.py with two checks against the sales table: freshness (hours since max(loaded_at)) and volume (today's row count vs a historical band). Read the DB password from os.environ, never a literal.
Add an alert(msg, severity) function that posts to the Slack webhook from SLACK_WEBHOOK_URL if set, and otherwise logs at WARNING. This is the "route to a human" step - stubbed so it works with no webhook too.
Wire the check into the Airflow DAG as a task that runs after the load. Trip it on purpose: hold back today's load and watch the freshness alert fire.
Confirm the alert names the symptom ("sales stale: 30.2h > 26h SLA"), not a cause. That phrasing is what makes a page trustworthy.
low/high to roughly the 5th and 95th percentile. A band that is too tight cries wolf; too wide catches nothing. Store the band in config so it can be tuned without a code change.
Docker Compose stands the stack up ★ 10 min · build your own
RetailPulse v0.7, part two: one docker compose up brings the whole platform online - Postgres, Airflow, and MLflow, wired together, with every secret read from a gitignored .env that is never committed. New teammate, new laptop, one command, working stack.
Write docker-compose.yml (Prompt B) declaring three services: postgres, airflow, and mlflow, on a shared network so they can reach each other by name.
Pull every secret from the environment with ${VAR} syntax. Compose reads .env automatically. Confirm .env is in .gitignore and only .env.example is committed.
Add a named volume for Postgres data so the database survives a restart, and mount your dags/ folder into Airflow so DAG edits show up without a rebuild.
Run docker compose up -d. Watch all three come up. Open Airflow on :8080 and MLflow on :5000 - the same stack you built piecemeal across b2-b6, now reproducible from one file.
Tear it down with docker compose down, then bring it back. Same environment, every time. That reproducibility is the whole point of IaC.
The onboarding that used to take three days. A data team's setup doc was 40 steps of "install this, then edit that." Every new hire lost their first three days to it, and it was subtly wrong for half of them. They replaced it with a docker-compose.yml and a .env.example. New setup: clone, copy the example to .env, fill three values, docker compose up. Three days became twenty minutes - and the environment was finally identical for everyone.
os.environ[...] is unchanged. IaC: Terraform or CloudFormation declares the real VPC, RDS, and cluster the way docker-compose.yml declares your local stack. The disciplines are identical; only the scale changes.
Try it yourself - this week ◐ 40-55 min total
- Finish both demos if you did not complete them live. Everyone should trip the freshness alert on purpose at least once and feel it fire.
- Add a third check to
monitor.py: a schema check that alerts if thesalescolumns drift from the b3 contract. Reuse the contract you already wrote. - Add a
.env.exampleto the repo and audit your history for any secret that ever got committed. If you find one, rotate it - do not just delete the line. - Give one alert a real owner and a one-line runbook link. An alert nobody owns is noise; fixing that is most of what "trustworthy alerting" means.
- Optional reading: the Docker Compose spec and the CloudWatch alarms quickstart - now you can read them as a review of what you already built.
What this session covers
This track teaches DataOps on an open-source stack from the tools' official docs, with AWS mapping notes. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · What is the difference between app observability and data observability?
A service can return HTTP 200 in 40ms while serving off a sales table that stopped updating on Tuesday. Freshness, volume, schema, distribution, and lineage are the data lens the service lens misses.
2 · What causes alert fatigue, and what is the fix?
The failure is too many noisy alerts, not too few. A quiet channel where every message means "go look now" and routes to a clear owner is worth more than a firehose everyone mutes.
3 · Where should the Postgres password for RetailPulse live?
A committed secret is compromised forever because history keeps it. Config as code goes in Git; secret values are injected at runtime from .env or a secret manager, so os.environ[...] stays the same either way.