How this track works
Eight sessions, one growing artifact: RetailPulse, a retail sales data product that starts tonight as a repo with one small pipeline and graduates in session b8 with orchestration, data contracts, database migrations, a monitored forecast model, and a promote-to-prod release. The unit of work is not a tool - it is the diff: one reviewable change that can touch code, schema, data expectations, and a model at once, and either passes every gate or does not merge. Each session adds one more thing a single safe change can now move. Two rules run through the track: you build each capability by hand and feel it break first, so the tooling never gets to be magic - and everything you build is really a way to read a change and judge it, because by b8 the change you review will have been written by an AI agent, not you.
What DataOps actually industrializes 6 min live
Strip the buzzword and DataOps is a promise: any change to your data, models, or schema can go from a laptop to production quickly, safely, and without heroics - because automated gates catch mistakes instead of on-call humans. That promise rests on four moves borrowed straight from DevOps. Tonight is move one: put everything in version control and let a machine check it.
LiveThe four moves DataOps borrows from DevOps3 min▶
DataOps is not a product you buy. It is a set of practices - version control, continuous integration, continuous delivery, and observability - applied to the data and ML lifecycle instead of just application code. The four that matter:
- Version everything: pipeline code, SQL, config, and eventually the schema and the model - all in Git. If it is not in version control, it does not exist and cannot be reviewed, reverted, or trusted.
- Continuous integration (CI): every change is automatically linted and tested before it merges. The machine, not the reviewer, catches the broken import at 2am.
- Continuous delivery (CD): merged changes flow to environments through the same automated path every time - no manual copy-paste to a server.
- Containers: the code carries its own environment, so it runs identically on your laptop, in CI, and in prod. "Works on my machine" becomes irrelevant.
The Friday deploy that took the dashboards down. A retail analytics team shipped a one-line SQL change straight to prod on a Friday. It silently changed a column type; every downstream dashboard broke over the weekend and nobody knew until Monday's board meeting. The fix was not a smarter analyst - it was a CI gate that would have caught the schema change in 40 seconds. That gate is what you build tonight.
Self-studyDataOps, MLOps, DevOps - who owns what2 min read▶
The terms overlap and people fight about the borders. A working map:
| Discipline | Object it ships safely | In this course |
|---|---|---|
| DevOps | Application code | The spine - b1 (this session) |
| Data CI/CD | Pipelines and datasets | b2 orchestration, b3 testing |
| DBOps | Database schema | b4 migrations |
| MLOps | Models | b5 tracking, b6 deploy/monitor |
DataOps is the umbrella - the culture and automation that makes all four move at the same cadence. The leader track's session a2 maps each of these to a business risk; worth a skim if you ever have to justify this work to a budget holder.
The repo, and the flow that keeps it sane 6 min live
A data project that lives half in notebooks, half in someone's Downloads folder, cannot be industrialized. Step one is a real repo layout and a branching model simple enough that the whole team actually follows it. For most data teams that is trunk-based development, not the heavy GitFlow of a decade ago.
LiveWhy trunk-based beats GitFlow for data teams3 min▶
GitFlow - with its develop, release, and hotfix branches - was built for shrink-wrapped software with scheduled releases. Data teams ship continuously and are usually small. Trunk-based development fits better:
- One long-lived branch:
main, always releasable. Everything else is a short branch that lives hours to a day. - Small pull requests: a change that touches one transformation, reviewed and merged same-day. Small PRs get better review and cause smaller incidents.
- CI is the gatekeeper: the branch cannot merge until the automated checks pass. This is the rule that makes the whole thing safe - and it is a repo setting, not a person's willpower.
- Protect main: require the PR and require CI green. In GitHub: Settings -> Branches -> add a rule. Do it on day one, before anyone can push straight to main.
data/, .env, and model binaries in .gitignore. Repos are for code and config; real data lives in object storage or a database. A committed 2GB CSV is a mistake you cannot fully undo.
Self-studyA repo layout you will not outgrow2 min read▶
The src/ layout (package under src/, tests beside it) is the boring, correct default - it forces you to install your package the way users will and stops "it imports locally but breaks in CI" surprises.
Add a pyproject.toml declaring the package and its dependencies (pandas, pyarrow, pytest). This becomes the single place that says what RetailPulse needs - CI and Docker both read it.
CI: the safety net that never sleeps 3 min live
Continuous integration means: every push, a machine checks out your code, installs it clean, and runs your linters and tests. If anything is red, the pull request cannot merge. Two layers do this - pre-commit runs on your laptop before the commit, GitHub Actions runs in the cloud on the PR. Defense in depth.
LivePre-commit: catch it before it is even a commit2 min▶
Pre-commit hooks run on git commit and can auto-format and lint. They keep the diff clean and stop the whole team arguing about style in review. Your workspace default is black for formatting and ruff for linting.
Then pip install pre-commit && pre-commit install. The check-added-large-files hook is your insurance against the accidental committed dataset.
Scaffold RetailPulse and its first pipeline ★ 14 min · everyone builds
RetailPulse v0.1: a repo with one honest pipeline - read a raw retail sales CSV, clean it, write a parquet file - plus one test that proves it works. Nothing fancy. Everything in version control.
Scaffold with Prompt A above: the src/ layout, .gitignore, and git init -b main. Confirm data/ is gitignored before you put any CSV in it.
Write src/retailpulse/pipeline.py: a function clean_sales(in_path, out_path) that reads the CSV with pandas, drops null order ids, coerces the date column, and writes parquet. Keep it under 30 lines - it grows all track long.
Write tests/test_pipeline.py: build a tiny 3-row DataFrame in the test, run it through the cleaning logic, and assert the row count and that dates parsed. This is the test CI will run.
Wire pre-commit (Prompt B), run pre-commit install, then git add -A && git commit. Watch black and ruff tidy your files before the commit completes.
Run pytest -q locally and see it pass. You now have step 1 and the local half of step 2 from the Part 1 diagram.
Green CI on a pull request, and a Dockerfile ★ 8 min · build your own
Now the cloud half. A GitHub Actions workflow runs your tests on every PR; a Dockerfile makes the pipeline run identically anywhere. Push a branch, open a PR, and watch the green check appear.
Create .github/workflows/ci.yml from Prompt C. Commit it on main first so the workflow exists.
Push RetailPulse to a new GitHub repo. In Settings -> Branches, add a rule protecting main: require a pull request and require the CI check to pass.
Make a change on a branch - git switch -c feature/tidy-dates, tweak the pipeline, push. Open a pull request and watch Actions run install -> lint -> test in the cloud.
Break it on purpose: push a failing test. See the red X and the blocked merge button. Fix it, see green, merge. That loop is CI, felt.
Write the Dockerfile (Prompt D) and run docker build -t retailpulse . && docker run retailpulse. The pipeline runs in a clean container - the same one CI and prod will use later.
The green check that changed the culture. A data team added exactly this workflow to a two-year-old pile of scripts. In the first week it caught three broken imports and a hardcoded path that only worked on one laptop. Nobody had to be the bad cop in review anymore - the machine was. That is the quiet, real payoff of DataOps: fewer arguments, fewer 2am pages.
buildspec.yml instead of a workflow file), or you keep Actions and just deploy to AWS. The container you built runs unchanged on ECS, Fargate, or Batch. The concept - automated lint/test gate before merge - is identical whichever you pick.
Try it yourself - this week ◐ 30-45 min total
- Finish both demos if you did not complete them live - especially the deliberately-broken PR. Everyone in this track should feel a red check block a merge at least once.
- Add a second test to RetailPulse: assert that rows with a null
order_idare dropped. Watch it run in CI on a PR. - Turn on branch protection for
mainif you skipped it - require the PR and require CI green. This one setting is most of what "governed" means. - Add a status badge to your README from the Actions tab. A green badge on the repo front page is a small, real signal of a healthy project.
- Optional reading: the GitHub Actions quickstart and pre-commit's docs - now you can read them as a review of what you already built.
What this session covers
This track teaches DataOps on an open-source stack from the tools' official docs, with AWS mapping notes. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · What does continuous integration (CI) actually guarantee?
CI is the automated gate on the way in - install, lint, test on every PR. Deploying is CD (continuous delivery), which comes later in the track.
2 · Why trunk-based development over heavy GitFlow for most data teams?
Data teams ship continuously and are usually small. One releasable main plus short branches gated by CI fits far better than develop/release/hotfix branches.
3 · Why should data/ be in .gitignore?
Never commit data. It bloats the repo forever (history keeps it), risks leaking confidential records, and mixes two things that version differently. Code in Git, data in storage.