How this track works
Ten sessions, one deliverable: a reference architecture for your organization's five data personas - data engineer, analyst, data scientist, AI engineer, and the business client who just wants an answer. This is not a vendor tour. Each session takes one layer, states the job it does, names the real options with honest cost bands, and identifies the chokepoint - the narrow waist where that layer's truth is decided and where lock-in actually lives. b2 through b8 walk the stack bottom to top; b9 is the evidence session on self-serve; b10 assembles and costs your architecture. Every price on these pages was fetched from a vendor's own billing feed with the date attached, because this subject goes stale faster than anything else on the shelf.
The stack, bottom to top 8 min live
Every enterprise data platform, whatever the logos on it, is these eight jobs. Not eight products - eight jobs, some of which one product may cover and some of which need three. Learn the jobs and the vendor landscape stops being intimidating, because every pitch you ever hear is a claim about which of these eight boxes a product fills.
LiveReading the stack in both directions4 min▶
Read it bottom-up and it is a dependency chain: nothing above works without the layers beneath it. Read it top-down and it is a list of business outcomes, which is how it gets funded. Most platform failures come from an organization funding top-down and building top-down at the same time - buying the outcome without the dependencies.
Two observations that will keep recurring:
- Layers 1 and 2 unblock nobody by themselves. Nobody's job title is "uses the object store." That makes them the easiest layers to underfund and the most expensive to retrofit.
- The gap between layers 4 and 5 is where most organizations actually live. Data lands, SQL runs, and every dashboard defines revenue slightly differently. That is not a tooling problem, it is a missing layer.
Airbnb published this exact progression over six years. Dataportal in 2017 solved discovery ("where is the data?"). Minerva in 2021 solved metric definitions - 12,000+ metrics, 4,000+ dimensions, 200+ data producers - because teams kept reporting conflicting numbers for the same question. Metis in 2023 added governance and lineage, with a graph of 100M+ nodes and 300M+ edges serving 1,000+ weekly users. Three artifacts, one company, published on their own engineering blog. Nobody starts with layer 8.
Self-studyWhy "just buy a platform" does not collapse the list3 min read▶
Databricks, Snowflake and Microsoft Fabric all sell something they call a unified platform - and all three genuinely cover several of these layers. But read their own words and they are unifying different axes:
- Databricks unifies workloads - BI and ML on one copy of the data.
- Microsoft Fabric unifies organizational storage - one OneLake per tenant, and their docs are blunt about it: "You can't delete OneLake or create multiple OneLakes."
- Snowflake unifies workload types including transactional - running OLTP alongside analytics.
The right question in any evaluation is not "is this platform unified?" but "which axis does my actual pain sit on?" A team drowning in duplicate departmental lakes has a different problem from a team whose ML and BI numbers disagree, and the two buy differently.
Every layer has a chokepoint 7 min live
Here is the frame that turns eight disconnected topics into one course. In every layer there is a narrow waist - a single place where plural, messy inputs become a singular, consequential output. That waist is where truth gets decided, where governance is possible, and where lock-in actually lives. Learn to find it, and you can evaluate any product in the category in about ten minutes.
| Layer | The chokepoint | What it decides |
|---|---|---|
| Storage | egress economics and retention terms | what it costs to read your own data, and whether you can leave |
| Table format + catalog | the catalog | access control, credential vending, commit sequencing |
| Orchestration | the asset graph | what "this ran successfully" actually means |
| Engine | the compute contract | who pays for a badly written query |
| Semantic layer | metric definitions | what "revenue" means, everywhere |
| MLOps | the model registry | what is in production and how to roll it back |
| LLM gateway | the gateway | which models run, at what cost, with what audit trail |
| Self-serve | the semantic layer, again | whether an answer can be trusted |
LiveThe chokepoint test: is it the only door?4 min▶
A chokepoint only governs if it cannot be bypassed. This is the single most useful evaluation question in the course, and it applies to every layer:
- A model registry that developers can route around by deploying an artifact straight from object storage provides zero governance, regardless of how good its UI is.
- A semantic layer that analysts can bypass by writing raw SQL in the BI tool defines metrics for exactly the people who were already following the rules.
- An LLM gateway that teams can skip by calling the provider API directly gives you cost dashboards covering the compliant half of your spend.
So when a vendor demos a governance feature, ask the same question every time: what stops someone from going around this? If the answer is "policy" rather than "architecture," you do not have a chokepoint - you have a suggestion.
Self-studyThe same pattern twice: registry and gateway3 min read▶
Notice something in the table above: layers 6 and 7 have structurally identical chokepoints. The model registry is the narrow waist for models you train; the AI gateway is the narrow waist for models you rent. Both answer exactly the same four questions:
- What is running in production right now?
- Where did it come from?
- Who approved it?
- How do I roll it back?
The strongest evidence that these are one pattern and not two: MLflow, the canonical MLOps tool, now ships an AI gateway. Same team, same governance instinct, applied to a new era of models. If you understand b7, b8 is half-learned already.
Climb the stack, watch the personas wake up ★ 14 min · everyone runs it
Below is your platform, empty. Switch on one layer at a time, bottom to top, and watch two numbers move together: how many of your five people can actually do their job, and what the bill says. Every dollar is computed live from published rate cards fetched on 6-7 August 2026 - AWS S3 storage and egress tiers, AWS p5 three-year reserved GPU pricing, NVIDIA AI Enterprise per-installed-GPU list price.
Nothing on: $0, and all five people are blocked. Obvious, but note who is blocked and why - each card names the specific job they cannot do.
+ Storage, + Catalog: $1,578/month, and still zero personas unblocked. Two layers of spend with nobody's job improved yet. This is the stretch that gets cut in budget reviews, and cutting it is why the layers above later fail.
+ Orchestration: $2,178, and your data engineer starts working - first person unblocked, three layers in.
+ Engine, + Semantic layer: $5,878, analyst unblocked. Note the analyst needed five layers, not one BI licence.
+ MLOps: $9,395, data scientist unblocked - and watch the jump. This is the most expensive single layer, and most of it is GPU compute and per-installed-GPU licensing (b7 breaks that bill down).
+ Gateway, + Self-serve: $11,295/month, $135,540/year, all five working. Sit with that number - it is the honest price of the platform your organization keeps describing as "just a dashboard tool."
Now break it the way real organizations do ★ 10 min · everyone runs it
With all eight layers on, switch off the semantic layer and watch the business-client card. They stay unblocked - the self-serve tool still answers - but the card turns red. That is not a bug in the simulator. That is the single most common failure in enterprise data, rendered honestly.
Turn Semantic layer + BI off while Self-serve stays on. The client card reads: "Answering confidently from ungoverned metrics - plausible SQL, wrong business logic." They are not blocked. They are getting answers. The answers are just not trustworthy.
Read the verdict line and the number in it: Microsoft's engineering team benchmarked Databricks Genie at 9.50% accuracy with an empty configuration, rising to 69.23% once column descriptions existed. Same product, same benchmark - the difference was metadata.
Now try the dashed trap lever, analytics reads direct from cloud storage. The bill jumps by $1,784 a month - $21,402 a year - on AWS's published egress tiers, for exactly the same work. b2 is entirely about why.
Say the sequencing rule out loud: self-serve is layer eight. Buying it first does not fail loudly. It fails quietly, by producing confident wrong answers that nobody flags.
Databricks' own best-practices page is the honest product spec. It states that "Quality table and column descriptions in Unity Catalog are critical for Genie accuracy," that "Genie Agents support up to 30 tables or views," and recommends you "Aim for five or fewer tables. The more focused your selection, the better." Five tables is a curated data mart. Set that against any vendor framing of "ask any question of all your data" - including from the same vendor. The tool is real and useful; the framing is not.
Try it yourself - this week ◐ 30 min total
- Map your own organization onto the eight layers. For each: what product fills it, or is it empty? An empty layer is not necessarily wrong - but an empty layer with a persona depending on it is a diagnosis.
- For each of your five personas, write the one sentence describing what they cannot currently do. Those sentences are your roadmap, in priority order, and they are far more persuasive to a budget holder than a tool list.
- Pick any governance feature your organization already owns and apply the chokepoint test: what stops someone from going around it? Write down whether the answer is architecture or policy.
- Run the ladder with your real numbers - storage volume and egress in the demo inputs - and note where your spend actually concentrates. It is rarely where people assume.
Official sources covered
This track is built on vendors' own documentation and billing feeds, published engineering blogs from companies that built these platforms, and peer-reviewed work where it exists - with marketing claims labelled as such. Every price carries its fetch date. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · You switch on storage and catalog. The bill reads $1,578 a month and zero personas are unblocked. What is the correct reading?
Nobody's job title is "uses the object store." Foundation layers pay off indirectly, through every layer above them - which makes them the hardest to justify in a budget review and the most expensive to retrofit later.
2 · With self-serve on and the semantic layer off, the business client is unblocked but flagged red. Why is that worse than being blocked?
Microsoft's evaluation measured this: 9.50% accuracy on an empty configuration versus 69.23% with column descriptions. The failure mode is not broken SQL, it is syntactically valid SQL carrying wrong business logic - which nothing catches.
3 · A vendor demos a model registry with approval workflows. What is the chokepoint test?
A chokepoint governs only if it is the only door. If the answer is "policy" rather than "architecture," it is a suggestion, not a control - and it will govern exactly the people who were already following the rules.