Where RetailPulse is tonight
Recap: RetailPulse v0.1 is a repo with clean_sales(in_path, out_path) that reads a raw sales CSV, drops null order ids, coerces the date, and writes clean_sales.parquet. It has a test, pre-commit, GitHub Actions CI, and a Dockerfile. Solid. But it is inert - it does nothing until a human runs it. Real data products run on their own, recover from failures, and let you see what happened. That is orchestration, and it is the single biggest jump from "I wrote a script" to "I run a data platform".
From a script to a scheduled DAG 6 min live
A script is a one-shot. You run it, it works or it does not, and if it fails at 3am you find out from an angry stakeholder. An orchestrated DAG is the same logic wrapped in a system that schedules it, orders its steps, retries the flaky ones, lets you re-run history, and shows you a green-or-red grid of every run. Same code, industrial packaging.
LiveThe five things an orchestrator gives you3 min▶
Whatever tool you pick, orchestration buys you five capabilities you would otherwise hand-build badly with cron and prayer:
- Scheduling: "run this every day at 6am" or "every hour" declared once, in code, not scattered across crontabs on a box nobody can find.
- Dependencies: load must not start until clean finishes, and clean must not start until extract finishes. The orchestrator enforces the order and stops load if clean failed.
- Retries: a network blip should not page a human. Say "retry twice, two minutes apart" and transient failures heal themselves.
- Backfills: you added RetailPulse today but you have 90 days of history. One command runs the pipeline for every past day, in order.
- Visibility: a UI that shows every run, its logs, its duration, and whether it is green or red. When something breaks you see it, with the log, instead of guessing.
The cron job nobody could find. A retail team's nightly sales load ran from a crontab on an engineer's personal EC2 box. She left. Three months later the load silently stopped, reports went stale, and it took two days to even locate the machine. Moving that job into an orchestrated DAG - versioned in the repo, visible in a UI, owned by the team - is not bureaucracy. It is the difference between a system and a liability.
Self-studyWhy not just cron?2 min read▶
Cron schedules. That is all it does. It cannot express "B depends on A", it will not retry, it has no backfill, and it gives you no run history beyond whatever you logged by hand. For a single independent job cron is fine. The moment you have steps that depend on each other, or you care about missed runs, or you want to see what happened last Tuesday, you have outgrown it. Orchestration is cron plus dependencies plus retries plus history plus a UI - which is why every data platform ends up with one.
The Airflow mental model 6 min live
Airflow has a big surface, but you use maybe 20% of it every day. That 20% is five words: DAG, task, operator, scheduler, executor. Learn what each one is and the docs stop feeling like a foreign language.
LiveFive words: DAG, task, operator, scheduler, executor3 min▶
The whole daily vocabulary:
- DAG (Directed Acyclic Graph): your pipeline as a graph of steps with a direction and no loops. It carries the schedule ("@daily") and the dependency wiring. One DAG per logical pipeline - RetailPulse gets one.
- Task: a single node in the DAG - extract, or transform, or load. A task is one unit of work Airflow schedules, runs, retries, and shows you the log for.
- Operator: the template a task is built from.
PythonOperatorruns a Python function; there are operators for Bash, SQL, S3, and hundreds more. The modern TaskFlow@taskdecorator is just a friendlier way to make a PythonOperator. - Scheduler: the brain. It reads your DAGs, works out which runs are due, and hands ready tasks to the executor. It is what makes "@daily" actually fire.
- Executor: the muscle. It runs the tasks - locally, on a pool of workers, or on Kubernetes. You rarely think about it until you need scale; for learning, the default is fine.
@dag / @task TaskFlow API. It lets you write plain functions, pass return values between tasks directly, and skip most of the boilerplate. That is what both demos tonight use.
Self-studyAirflow vs Dagster vs Prefect - an honest comparison2 min read▶
You will be asked "why Airflow and not X" in an interview or a planning meeting. A fair, short answer:
| Tool | Model | Best when |
|---|---|---|
| Airflow | Task-oriented; you wire steps | Ubiquitous, huge ecosystem, most jobs and hires - the safe default |
| Dagster | Asset-oriented; you declare data assets | You think in tables/models and want lineage + data-awareness built in |
| Prefect | Pythonic, dynamic flows | Highly dynamic pipelines, lighter setup, Python-native feel |
All three do scheduling, retries, and visibility. We teach Airflow because it is the one you are most likely to inherit or be hired for - but every concept tonight (DAG, task, schedule, backfill, idempotency) transfers directly. If your shop runs Dagster, you are learning the ideas, not just the buttons.
Idempotency and backfills 4 min live
Here is the property that separates a pipeline you can trust from one you cannot: a task must be safe to run twice. Re-running yesterday's job - because it failed, or you fixed a bug, or you are backfilling history - must produce the same result, not double the rows. That is idempotency, and the trick to getting it is partitioning by date.
LiveIdempotent tasks: safe to re-run, always3 min▶
An idempotent task gives the same output no matter how many times it runs. The failure mode you are avoiding is the append: a task that tacks yesterday's rows onto a file every run, so a single retry silently doubles your data. Three rules keep you safe:
- Partition by the run date: each day's output lands in its own folder, e.g.
clean_sales/date=2026-07-01/. The date comes from the run's data interval, notdatetime.now(). - Overwrite, do not append: a task processing 2026-07-01 replaces that partition wholesale. Run it once or ten times - the partition is identical.
- Read your window explicitly: a task should read exactly the slice of input for its date, so it never depends on what happened to be lying around.
Get this right and backfills become trivial: "run every day from 2026-04-01 to today" just replays the same idempotent task across 100 dates, each writing its own partition. Get it wrong and a backfill is a data-corruption event.
datetime.now() in a task
Use Airflow's logical/execution date (the data interval). now() means a re-run of an old date processes today's data - the opposite of idempotent. The whole backfill machinery depends on the task honoring the date it was handed.
Wrap RetailPulse in an Airflow DAG ★ 12 min · everyone builds
Take the clean_sales function you already have and wrap it in a three-task TaskFlow DAG: extract pulls the raw CSV, transform calls clean_sales, load confirms the parquet landed. One new file, dags/retailpulse_dag.py, on a daily schedule. Your pipeline logic does not change - you are adding the shell.
Add an dags/ folder to RetailPulse and point Airflow at it (set AIRFLOW__CORE__DAGS_FOLDER to that path, or drop the file in ~/airflow/dags while learning).
Import your existing clean_sales from retailpulse.pipeline - do not reimplement it. The DAG orchestrates the code you tested in b1; it does not duplicate it.
Write dags/retailpulse_dag.py using the @dag / @task TaskFlow API from Prompt A: three tasks, wired extract -> transform -> load, schedule="@daily".
Open the Airflow UI at localhost:8080, find retailpulse_daily, un-pause it, and trigger one run. Watch the three tasks go green in order in the grid view.
Click the transform task -> Logs. You are looking at the exact output of clean_sales, captured and timestamped. That visibility is half of what you came for.
Schedule, config, and a backfill ★ 10 min · build your own
A daily schedule is only useful if each run processes its own day and you can replay history. Now you parameterize the DAG by execution date, pull config from Airflow Variables instead of hardcoding, make the write idempotent by partition, and run a real backfill across a date range.
Read the run's data interval inside the task (Prompt B) and use it to build a dated input path and a dated output partition. No datetime.now() anywhere.
Move the raw-data location into an Airflow Variable (retailpulse_raw_prefix) or an env var, so dev and prod differ by config, not code. Read it with Variable.get.
Make the write idempotent: the transform overwrites clean_sales/date=YYYY-MM-DD/, so re-running a date replaces rather than appends. Confirm by running the same date twice and checking the row count is stable.
Set catchup=True and a start_date two weeks back, or run the CLI backfill in Prompt C for an explicit range. Watch the grid fill in one green column per day, oldest first.
Commit the DAG on a branch, open a PR, let b1's CI run. RetailPulse is now v0.2: orchestrated, scheduled, backfillable, still green.
The backfill that saved a launch. A retail analytics team shipped a fixed currency-conversion bug on a Thursday. Ninety days of history were wrong. Because the pipeline was idempotent and partitioned by date, the fix was one backfill command overnight - every day recomputed cleanly, no duplicates, no manual surgery. The team that had append-based, non-idempotent jobs next door spent a week reconstructing theirs by hand. Same bug, wildly different Friday.
Try it yourself - this week ◐ 40-55 min total
- Finish both demos if you did not live - especially the backfill. Everyone in this track should watch a grid fill with green columns at least once.
- Break a task on purpose (raise an exception in transform) and watch the two retries fire two minutes apart before the run finally goes red. Feel the retry safety net.
- Add a fourth task -
row_count_check- that fails if the cleaned partition has zero rows. This is a preview of b3's data testing. - Prove idempotency: run
2026-07-05three times and confirm the partition row count never changes. If it grows, your write is appending - fix it to overwrite. - Optional reading: the Airflow "TaskFlow API" tutorial and the "Data Interval" concepts page - now a review of what you already built.
What this session covers
This track teaches DataOps on an open-source stack from the tools' official docs, with AWS mapping notes. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · What does an orchestrator give you that a plain script does not?
The logic is unchanged - orchestration is the shell that schedules it, orders its steps, retries failures, replays history, and shows you every run in a UI.
2 · In Airflow, what is the difference between a task and an operator?
Operator = the reusable template (PythonOperator, BashOperator...). Task = one node built from an operator that the scheduler runs and retries. The scheduler and executor are separate again.
3 · Why must a task use the run's data interval instead of datetime.now()?
With now(), re-running 2026-07-01 would process today's data - the opposite of idempotent. Honoring the handed-in date lets you overwrite the right partition and backfill history cleanly.