Builder session 5 · MLOps I: experiment tracking and the registry

Part 0

Where RetailPulse is tonight

Four sessions in, RetailPulse is a real data product: a clean_sales pipeline that writes clean_sales.parquet, an Airflow DAG that runs it on a schedule, a Great Expectations suite plus a data contract guarding quality, a Postgres sales table with versioned migrations, and green CI on every pull request. It has never had a model. Tonight it gets one - and we treat that model with the exact same discipline as the code: reproducible, reviewed, gated.

Live - presented in session Self-study - read after class ★ Try it now command Open-source stack + AWS notes

★ What you walk out with today A demand-forecast model trained on RetailPulse sales history, with every run logged to MLflow - params, metrics (MAE/RMSE), the model artifact, the data version, and the git sha. The best run promoted into the MLflow Model Registry at the Staging stage. And a model-CI script that retrains, re-evaluates, and blocks the merge if RMSE regresses past a threshold. That is RetailPulse v0.5.

Part 1 · covers the reproducibility crisis + what to log

Why experiments need tracking 7 min live

Modelling is search. You try a feature set, a learning rate, a different date window - dozens of runs, each with its own params, data, and metrics. Do that in a notebook and the truth of "which run was best, and can I rebuild it" lives in cell outputs you have already overwritten. A tracking ledger fixes that: one immutable record per run that you can sort, compare, and reproduce.

🔍 Click to zoom - scattered notebook runs versus one comparable, reproducible ledger

LiveThe reproducibility crisis, and exactly what to log3 min▶

A model result is reproducible only if you can rebuild the same number from the same inputs. In a notebook, the inputs drift silently: you re-ran a cell, the data got a new row, you nudged a hyperparameter and forgot. Six weeks later "the good model" cannot be recreated. Tracking makes reproduction a lookup, not an archaeology dig. Log these on every run:

Params: every hyperparameter and every choice - learning rate, tree depth, feature list, train/test split date.
Metrics: the numbers you compare on - for a regressor, MAE and RMSE on a held-out set.
Artifacts: the serialized model itself, plus any plots (residuals, feature importance).
Data version: which snapshot of clean_sales trained this - a parquet hash, a table version, or a date cutoff.
Git sha + seed: the exact code commit and the random seed. Together with the data version, these make a run rebuildable byte-for-byte.

★ Try it now (your terminal)pip install mlflow scikit-learn pandas pyarrow mlflow --version && python -c "import sklearn; print(sklearn.__version__)" # Both should print. MLflow will store runs under ./mlruns by default.

Real world

The forecast nobody could rebuild. A demand team shipped a model that cut stockouts noticeably, then spent a quarter unable to reproduce it - the winning notebook had been re-run, the data had grown, the seed was never set. When a regulator asked how the number was produced, the honest answer was "we are not sure." One line - mlflow.start_run() - would have made that answer "here is the run, the data version, and the commit."

Self-studyTracking is not just for training runs2 min read▶

The same ledger discipline pays off across the model's life. Log the evaluation run that promoted a model, the batch-scoring job that used it, and the drift report that flagged it for retraining (you build that in b6). When every step writes to the same tracking store, "what happened to this model" becomes one query instead of a Slack thread. The registry - covered in Part 2 - is where those runs graduate from "a number I logged" to "the version serving production."

Part 2 · covers MLflow Tracking + the Model Registry

MLflow: tracking + the model registry 6 min live

MLflow is the open-source default for this. A training run logs its params, metrics, and model to MLflow Tracking. When a run wins, you register its model into the Model Registry, where it gets a version number and moves through stages - None, Staging, Production, Archived. The registry becomes the single answer to "what is serving prod, and where did it come from."

🔍 Click to zoom - runs log to Tracking; the winner is registered and promoted through stages

LiveThe four MLflow components - focus on Tracking and Registry3 min▶

MLflow has four parts. Two are the workhorses tonight; the other two are good to know:

Tracking (core tonight): the API and UI for logging params, metrics, and artifacts per run. mlflow.start_run() opens a run; mlflow.log_param / log_metric / log_model fill it; or mlflow.autolog() captures most of it automatically for scikit-learn.
Registry (core tonight): a versioned store of named models with stage transitions (Staging, Production, Archived). This is what serving and CI point at, not a random file path.
Models: a standard packaging format so any framework's model loads the same way (mlflow.pyfunc). It is what makes the registry framework-agnostic.
Projects: a convention for packaging runnable training code with its environment. Useful later; not needed to log a run.

What to log for RetailPulse	MLflow call	Why it matters
Hyperparameters, feature list, split date	`log_param` / autolog	Reproduce and compare runs
MAE, RMSE on held-out weeks	`log_metric`	Pick the winner objectively
Trained regressor	`log_model`	The artifact you register and serve
Data snapshot version / hash	`set_tag("data_version", ...)`	Ties the model to its training data
Git sha, random seed	`set_tag` / `log_param`	Byte-for-byte rebuildability

Part 3 · covers reproducibility + model CI

Reproducibility and model CI 3 min live

In b1 you built CI for code: every PR runs lint and tests, and red blocks the merge. A model deserves the same gate. Model CI trains and evaluates on a pull request and fails if the key metric regresses below a threshold. But a metric gate is only meaningful if the run is reproducible - fixed seed, pinned data, params in config - otherwise the number wobbles for reasons that have nothing to do with your change.

LivePin the three things, then gate on the metric2 min▶

Reproducibility is three habits, then the gate becomes trustworthy:

Fixed seed: set random_state everywhere (the split, the model). Same inputs, same output.
Pinned data snapshot: CI trains on a fixed, versioned slice of clean_sales - not "whatever is in the table today." Pin by date cutoff or a committed sample.
Params in config: hyperparameters live in a config.yaml read by both training and CI, so the run is defined by a file, not by memory.

Model CI then does exactly what code CI does: on a PR, a GitHub Actions job trains the model, evaluates RMSE on the held-out weeks, and exit 1 if RMSE > threshold. A change that quietly makes the forecast worse gets blocked before it ever reaches the registry. That is the whole idea - the machine, not a reviewer, catches the regression.

Threshold, not perfection Set the RMSE threshold a little above your current best, not at it - models fluctuate run to run even with a fixed seed once data grows. The gate exists to catch real regressions, not to punish noise. Revisit the number as the model improves.

Self-studyData versioning: the piece people skip2 min read▶

Code versioning is solved - Git. Data versioning is the part teams neglect, and it is why models become irreproducible. Options range from simple to serious: a date cutoff plus a content hash of the parquet (enough for RetailPulse tonight); a committed small reference sample for CI; or a dedicated tool like DVC or lakeFS that snapshots datasets alongside commits. Whatever you choose, the rule is the same as b1's "never commit data": store the pointer, not the payload. Tag every MLflow run with the data version so a run always knows what it trained on.

Demo 1 of 2

Track a demand-forecast run ★ 12 min · everyone builds

Build features from RetailPulse sales history, train a gradient-boosting regressor to predict next-period units per product and region, and log the whole run to MLflow - params, MAE/RMSE, the model, and the data version. Keep the model simple; the point is the tracking, not the leaderboard.

Add a train.py under src/retailpulse/. Read clean_sales.parquet, aggregate to units sold per product, region, and period (say weekly), and build lag features - last week's units, a rolling mean - as predictors of next week's units.

Split by time, not randomly: train on the earlier weeks, hold out the most recent ones. A random split leaks the future into training and inflates your metric.

Open an MLflow run with mlflow.start_run() and turn on mlflow.sklearn.autolog(). Fit a GradientBoostingRegressor with a fixed random_state.

Compute MAE and RMSE on the held-out weeks and log_metric both. Tag the run with the data version and git sha so the run is reproducible.

Run mlflow ui and open the run. Change one hyperparameter, run again, and compare the two runs side by side. You have a ledger.

★ RetailPulse v0.5 - train.py, tracked training runimport subprocess, mlflow, mlflow.sklearn, pandas as pd from sklearn.ensemble import GradientBoostingRegressor from sklearn.metrics import mean_absolute_error, mean_squared_error SEED = 42 def build_features(path: str) -> pd.DataFrame: df = pd.read_parquet(path) df["period"] = df["order_date"].dt.to_period("W").dt.start_time g = df.groupby(["product", "region", "period"])["quantity"].sum().reset_index() g = g.sort_values("period") g["lag_1"] = g.groupby(["product", "region"])["quantity"].shift(1) g["roll_3"] = g.groupby(["product", "region"])["quantity"].shift(1).rolling(3).mean() return g.dropna() def main(path: str = "data/clean_sales.parquet", threshold_week: str = "2026-06-01") -> None: feat = build_features(path) X_cols = ["lag_1", "roll_3"] train = feat[feat["period"] < threshold_week] test = feat[feat["period"] >= threshold_week] sha = subprocess.getoutput("git rev-parse --short HEAD") mlflow.set_experiment("retailpulse-demand-forecast") mlflow.sklearn.autolog() with mlflow.start_run() as run: mlflow.set_tag("data_version", "clean_sales@2026-07") mlflow.set_tag("git_sha", sha) model = GradientBoostingRegressor(random_state=SEED) model.fit(train[X_cols], train["quantity"]) pred = model.predict(test[X_cols]) mae = mean_absolute_error(test["quantity"], pred) rmse = mean_squared_error(test["quantity"], pred) ** 0.5 mlflow.log_metric("mae", mae) mlflow.log_metric("rmse", rmse) print(f"run {run.info.run_id} MAE={mae:.2f} RMSE={rmse:.2f}") if __name__ == "__main__": main()

Data tip Train only on synthetic or public retail history in this repo - the same rule from b1. And never commit clean_sales.parquet or the mlruns/ store; both go in .gitignore. The tracking store holds artifacts, not source data.

Demo 2 of 2

Register the model + a CI gate ★ 10 min · build your own

Take the winning run, register its model as demand_forecast, and transition it to Staging. Then add a model-CI script that trains, evaluates, and exits non-zero if RMSE breaks the threshold - wired into GitHub Actions so a regressing model cannot merge.

Register the best run's model with mlflow.register_model, giving it the name demand_forecast. MLflow assigns it a version number.

Transition that version to Staging with the MlflowClient. Staging means "evaluated, candidate for prod" - it does not serve yet.

Write model_ci.py: retrain on the pinned slice, evaluate RMSE, and sys.exit(1) if it exceeds the threshold. This is the model equivalent of pytest.

Add a model-ci job to .github/workflows/ci.yml that runs python -m retailpulse.model_ci on every PR. Red blocks the merge, exactly like a failing test.

Prove it: push a change that makes the forecast worse (drop a feature). Watch the model-CI job go red and block the merge. Restore it, see green, merge.

★ Register the winning run + promote to Stagingimport mlflow from mlflow.tracking import MlflowClient client = MlflowClient() runs = mlflow.search_runs( experiment_names=["retailpulse-demand-forecast"], order_by=["metrics.rmse ASC"], max_results=1, ) best_run_id = runs.iloc[0]["run_id"] mv = mlflow.register_model(f"runs:/{best_run_id}/model", "demand_forecast") client.transition_model_version_stage( name="demand_forecast", version=mv.version, stage="Staging", ) print(f"registered demand_forecast v{mv.version} -> Staging")

★ model_ci.py - the metric gateimport sys from retailpulse.train import build_features from sklearn.ensemble import GradientBoostingRegressor from sklearn.metrics import mean_squared_error RMSE_THRESHOLD = 45.0 # a little above current best; catches real regressions SEED = 42 def evaluate() -> float: feat = build_features("data/reference_sales.parquet") # pinned CI slice X_cols = ["lag_1", "roll_3"] train = feat[feat["period"] < "2026-06-01"] test = feat[feat["period"] >= "2026-06-01"] model = GradientBoostingRegressor(random_state=SEED).fit(train[X_cols], train["quantity"]) pred = model.predict(test[X_cols]) return mean_squared_error(test["quantity"], pred) ** 0.5 if __name__ == "__main__": rmse = evaluate() print(f"model-CI RMSE={rmse:.2f} threshold={RMSE_THRESHOLD}") if rmse > RMSE_THRESHOLD: print("REGRESSION: model got worse. Blocking merge.") sys.exit(1) print("OK: model within threshold.")

★ Add the model-CI job to .github/workflows/ci.yml model-ci: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: { python-version: "3.11" } - run: pip install -e ".[dev]" - run: python -m retailpulse.model_ci # exits 1 if RMSE regresses

Real world

The refactor that quietly cost 8% accuracy. An analyst "cleaned up" a feature-engineering function and shipped it - all the unit tests passed, because the code still ran. What the tests could not see was that the forecast got measurably worse. A team with model CI would have caught it on the PR: RMSE crossed the threshold, the job went red, the merge was blocked. Code tests check that it runs; model CI checks that it still works.

☁️ AWS mapping MLflow is the OSS default here. On AWS, SageMaker Experiments plays the Tracking role and the SageMaker Model Registry plays the registry role, with model packages moving through approval states instead of stages. Or run MLflow itself on SageMaker and keep this exact code. Either way the concept is unchanged: track everything, register the winner, gate promotion on a metric.

Homework

Try it yourself - this week ◐ 40-55 min total

Finish both demos if you did not complete them live - especially the deliberately-regressing PR. Everyone should feel model CI block a merge at least once.
Run a small sweep: train three versions with different tree depths, compare them in the MLflow UI, and register the best. Note the data version tag on each.
Add a second metric - MAPE - to both train.py and the run comparison. Decide which metric you would actually gate on for a demand forecast, and why.
Pin a real reference slice for CI: export a fixed, small reference_sales.parquet committed as a sample (synthetic only), so the model-CI number is stable across runs.
Optional reading: the MLflow Tracking and Model Registry docs - now a review of what you already built by hand.

Source material

What this session covers

This track teaches DataOps on an open-source stack from the tools' official docs, with AWS mapping notes. This page covers:

✓Experiment tracking concepts + what to logPart 1 · reproducibility, params/metrics/artifacts/data/sha

✓MLflow Tracking + Model Registry docsPart 2 + Demo 1/2 · logging, autolog, register, stages

✓Model CI + reproducibility disciplinePart 3 + Demo 2 · seed, pinned data, config, metric gate

◐Data versioning (DVC / lakeFS)Part 3 self-study · concept + options, not a full tutorial

◐SageMaker Experiments / Model Registry mappingDemo 2 sidebar · concept parity, not a full AWS tutorial

Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · Why log the data version and git sha on every training run?

A metric is only trustworthy if you can recreate it. Data version + git sha + seed pin the three things that otherwise drift and make a run impossible to reproduce.

2 · What is the difference between MLflow Tracking and the Model Registry?

Tracking answers "which run was best." The Registry answers "which version is live." Serving and CI point at the registry, not at a random run's file path.

3 · What does model CI add that code CI (pytest) does not?

Unit tests check that code runs. Model CI checks that it still works - a refactor can pass every test while quietly making the forecast worse. The metric gate catches that.

Builder session 5 cheat sheet · pin this

Why trackModelling is search - dozens of runs. A ledger makes "which was best, can I rebuild it" a lookup, not archaeology.

What to logParams, metrics (MAE/RMSE), the model artifact, data version, git sha, and the random seed. Every run.

MLflow Trackingstart_run() + log_param/log_metric/log_model, or autolog() for scikit-learn. Compare runs in the UI.

Model RegistryRegister the winner as a named model; move versions through None → Staging → Production → Archived.

ReproducibilityFixed seed + pinned data snapshot + params in config. Without them the metric gate wobbles on noise.

Model CITrain + evaluate on the PR; exit 1 if RMSE > threshold. Code tests check it runs; model CI checks it still works.

AWS mappingSageMaker Experiments (tracking) + SageMaker Model Registry, or MLflow on SageMaker. Same loop: track, register, gate.

Running projectRetailPulse v0.5 lives: tracked + registered demand-forecast model with a model-CI gate. Next: b6 deploy + monitor.