Where we are in the stack
b2 gave you durable bytes in an open table format. b3 gave you a catalog that decides who may read them and in what order commits land. Neither of those moves a single row on its own. Layer three is the layer that makes data arrive - reliably, on a schedule, and observably enough that somebody notices when it does not. It is also the first layer in the ladder that unblocks a named human: the data engineer starts working here, three layers and roughly $2,178 a month into the build.
Airflow 3 and the security break 7 min live
Airflow describes itself, in its own docs, as "an open-source platform for developing, scheduling, and monitoring batch-oriented workflows." Hold on to the word batch-oriented - it is the honest scope statement that most comparison blogs quietly drop. Airflow is the default because it got there first and because the provider ecosystem is enormous, not because its programming model won an argument.
LiveThe 3.0 break was about security, not features4 min▶
Airflow 3.0, released 2025-04-22, is the first architectural break in five years - and the reason is security, not features. That framing matters because it tells you what was structurally wrong with the previous design.
The change is the Task Execution Interface plus the Task SDKs: tasks no longer need direct access to the metadata database. Read that as an architecture statement rather than a release note. In Airflow 2, running a task meant handing that task a live connection to the system of record for every DAG, every connection and every variable in your deployment. Every worker was inside the blast radius. Remove that dependency and two things become possible that were previously unsafe:
- Multi-cloud execution - workers in a network that cannot reach your metadata database.
- Untrusted-tenant execution - running somebody else's task code without handing them the keys to the platform.
This is the chokepoint lesson from b1 applied to orchestration. The metadata database was the narrow waist, and in Airflow 2 the waist was reachable from everywhere.
LiveDAG versioning, and what the top feature request tells you4 min▶
3.0 also shipped DAG Versioning. The behaviour, stated plainly: "A DAG will run through to completion based on the version at start, even if a new version has been uploaded."
Read the feature backwards and it becomes a diagnosis. This was the top survey request for years, which tells you the real, lived pain of task-centric orchestration: a long-running DAG mutating mid-flight because somebody pushed to main. Half your run executed yesterday's logic and half executed today's, the UI showed green throughout, and the resulting table is a blend nobody can describe.
Notice that this is not a bug anyone introduced. It is what happens when the unit of execution is a task and the definition of the graph is whatever file happens to be on disk when the scheduler next looks. Part 2 is about an orchestrator that took a different swing at the same problem.
Self-studyThe adoption numbers, and who counted them3 min read▶
Project-published adoption figures (Apache Software Foundation, 2025-04-22):
- 30M monthly downloads - 30x since 2020.
- 80,000 organizations, up from 25,000.
- 30% use it for MLOps, 10% for GenAI workflows.
Those come from the project itself, which is the strongest provenance available for an open-source project. The survey behind the workload split deserves a label though.
Disclosure you should repeat every time you quote it: the Airflow Survey 2025 - 5,818 responses from 122 countries - is sponsored and run by Astronomer, the commercial Airflow vendor. That does not make it false, and the sample is genuinely large. It does mean the respondent pool skews toward people who are engaged enough with commercial Airflow to answer a vendor survey, and that a vendor chose which questions were asked. Quote the number with the sponsor attached, or do not quote it. This course also does not use the version-adoption percentages from that report, because they appear in Astronomer's write-up rather than on the ASF page.
What green actually means 8 min live
Dagster is asset-centric. Its docs define an asset definition as "a description, in code, of an asset that should exist and how to produce and update that asset." That sentence reads like a modelling preference. It is not. It changes what failure means, and once you see it you cannot unsee it in an Airflow UI.
LiveThe key insight: lineage as a by-product4 min▶
The asset model changes what failure means. In a task graph, green means the code exited zero. That is a statement about a process, and it is compatible with an empty output table, a partial load, or yesterday's data written twice.
In an asset graph, a materialization means "this table now exists at this version, from these upstream inputs." That is a statement about the world. And it has a consequence most teams do not expect when they adopt Dagster:
Lineage stops being a separate catalog project and becomes a by-product of the orchestrator. If every run declares which named artifact it produced and which named artifacts it read, the graph you needed a lineage tool to reconstruct is simply the definition you already wrote. Compare that with the b3 pattern, where lineage is inferred after the fact by parsing query logs and hoping.
This is the single strongest reason to consider Dagster, and it is not a UI argument.
LiveDagster deprecated its own founding primitive2 min▶
Dagster launched around ops - composable computations wired into graphs. Its own documentation now says ops "have largely been replaced by assets."
Two things follow, and the second one costs people real time:
- The team changed its mind in public, which is a reasonable signal of intellectual honesty and also a signal that the category is still moving.
- Older Dagster tutorials actively mislead. A 2021 blog post teaching ops and graphs is not merely dated, it teaches a primitive the vendor has since demoted. Check the publication date on every Dagster tutorial before you follow it, and prefer the current docs.
LivePrefect: the graph is discovered at runtime2 min▶
Prefect's own line is that it "turns your Python functions into production-grade data pipelines." Flows and tasks are decorated Python functions, and the structural claim is this: workflows "can branch, loop, and create tasks at runtime, rather than DAGs declared up front."
That is a genuinely different position, not a syntax preference. Airflow and Dagster both want a graph they can see before anything runs. Prefect is comfortable with a graph that only exists once the code is executing - which is exactly right for fan-out over a list whose length you do not know until you query it, and exactly wrong when you want to validate a change in CI.
One historical note so you can read old material: "negative engineering" was Prefect's founding thesis - the idea that most pipeline code exists to handle failure rather than to do work. It is absent from the current site. Treat it as history, not as current positioning.
Choosing honestly ★ 24 min · everyone does it
Open a text file and list your five most important pipelines. For each one, answer a single question before you look at any tool: when this pipeline succeeds, what exists that did not exist before? If the answer is a named table at a version, you are in asset territory. If the answer is "an email went out" or "a payment posted," you are not. That one question does more work than any feature matrix.
| Tool | Mental model | Strongest when | The honest cost |
|---|---|---|---|
| Airflow | a scheduled graph of tasks | you need a connector to something obscure, and you need to hire people who already know it | green means exited zero; the moat is the provider ecosystem and the hiring pool, not the programming model |
| Dagster | a graph of assets that should exist | pipelines produce durable named artifacts and you want lineage without a second project | the asset model only pays off when there are durable named artifacts; for workflows whose output is a side effect you are forcing an asset shape onto a genuine task |
| Prefect | Python functions, graph discovered at runtime | fan-out and branching depend on data you only have at execution time | the runtime-discovered graph buys real dynamism and costs static analysability - you cannot fully validate before it runs |
Sort your five pipelines into artifact or side effect. An artifact is a table, a file, a model, a feature set - something with a name and a version. A side effect is an API call, a notification, a payment posted. Most real platforms have both, which is why most real platforms end up running more than one orchestrator whatever the standardisation memo says.
For each artifact pipeline, write what "done" should assert. Not "the job ran" but "orders_mart exists at today's date, built from orders_clean v41." If you cannot write that sentence, the asset model will not help you yet - the missing piece is naming, not tooling.
For each side-effect pipeline, check whether it is idempotent. Asset semantics assume re-running produces the same artifact. Re-running a payment does not. This is the cleanest line in the whole decision.
Now check what your candidate pins at run start. Airflow 3 pins the DAG version. Ask the same of everything else you shortlist, and ask specifically about a run that takes four hours while somebody merges to main.
Last, and only last, look at popularity - and read it correctly. The table below is real data from the GitHub API, and it still does not mean what people use it to mean.
| Repository | Stars | Forks | Forks per star |
|---|---|---|---|
| apache/airflow | 46,403 | 17,536 | 0.38 |
| PrefectHQ/prefect | 23,565 | 2,446 | 0.10 |
| dagster-io/dagster | 15,940 | 2,231 | 0.14 |
A critical-reading exercise you can run in ninety seconds. Prefect's site claims "50.7k+ GitHub stars." The GitHub API says the PrefectHQ/prefect repository has 23,565. The gap is presumably an organization-wide aggregate across every repo they own, which is a defensible thing to count and a misleading thing to label. Cube's homepage runs the same pattern in the other direction, claiming 18,000 stars when the repo has 20,564 - a stale number nobody updated. The lesson is not that these vendors are dishonest. It is that vendor stat blocks are frozen marketing artifacts, not telemetry, and checking one against its own public API takes less time than reading the paragraph around it.
Try it yourself - this week ◐ 30 min total
- Take your longest-running production DAG and find out what happens today if someone merges a change to it mid-run. If you cannot answer from documentation, you have found an unpinned dependency.
- Open your orchestrator's UI and pick any green run from last week. Write down what you can prove exists because it is green. Then check whether it is true.
- Classify your pipelines into artifact-producing and side-effect-producing. Count them. The ratio tells you honestly whether an asset-centric tool fits your actual work or only your aspirations.
- Pick any vendor page in your current shortlist and verify one number on it against the primary source - GitHub API, the project's own release notes, a billing feed. Note how long it took and whether the number held.
- If you run Airflow 2, list which of your workers currently have network reach to the metadata database. That list is your blast radius, and it is the thing 3.0 exists to shrink.
Official sources covered
Every definition on this page is quoted from the project's own documentation, and every number carries its fetch date. Where a source has a commercial sponsor, the sponsor is named next to the number rather than in a footnote. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Airflow 3.0 was the first architectural break in five years. What drove it?
In Airflow 2 every worker held a live connection to the system of record for the whole deployment, so every worker sat inside the blast radius. Removing that dependency is what makes it safe to run tasks in a network that cannot reach the metadata database, or to run code you do not fully trust.
2 · In an asset graph, a materialization asserts something a green task cannot. What?
Green in a task graph means the code exited zero, which is compatible with an empty table, a partial load, or yesterday's data written twice. Because a materialization names the artifact and its inputs, lineage stops being a separate catalog project and becomes a by-product of the orchestrator.
3 · A shortlist doc ranks orchestrators by GitHub stars. What is the correction?
Airflow 46,403 stars and 17,536 forks is 0.38 forks per star, against 0.10 to 0.14 for Prefect and Dagster - forking implies somebody cloned it to run or change it. And check the vendor's own figure against the API: Prefect's site claims "50.7k+" while the repo shows 23,565.