How the leader track works
Six sessions, forty-five minutes each, and every one of them ends with decisions rather than homework. There is no code on these pages by design. The deliverable is a reference architecture for five personas - your data engineer, your analyst, your data scientist, your AI engineer, and the business client who just wants an answer they can put in a board pack. Sessions a2 and a3 give you the map and the build-versus-buy argument, a4 gives you the cost model, a5 gives you real launches and honest failures, and a6 turns all of it into a funding sequence you can defend. The practitioner track running alongside this one builds what you are learning to fund, which means the two tracks meet in the middle: you learn what to ask for, they learn what it takes to deliver.
Eight jobs, five people 15 min live
Whatever logos are on your architecture diagram, the platform underneath is these eight jobs. Not eight products - eight jobs, some of which one product covers and some of which quietly need three. This is the most useful thing a sponsor can hold in their head, because every vendor pitch you will ever sit through is a claim about which of these boxes the product fills, and most pitches are vague about it on purpose.
LiveThe five people you are actually buying for5 min▶
Layers are abstract. People are not, and people are what a budget conversation is really about. Every platform decision in this course resolves to one of five personas, and the discipline is to name the person before naming the product:
- The data engineer ingests, moves and cleans. Their job is that data arrives, on time, in a shape someone else can use, and that they find out when it does not.
- The analyst builds dashboards, writes SQL and answers business questions. Their job is a number that survives being challenged in a meeting.
- The data scientist trains, deploys and maintains models. Their job includes the unglamorous half: knowing what is in production and how to roll it back.
- The AI engineer ships LLM applications. Newest of the five, and the one most likely to be building on infrastructure nobody approved.
- The business client wants to ask a question in plain language and get an answer they can act on. They do not want a platform. They want an answer, which is precisely why they are the hardest to serve honestly.
Write those five job descriptions down before your next vendor meeting. Then, when the demo happens, ask which of the five just got unblocked and which layer the product assumes you already have. The second half of that question is where the honest conversation starts.
LiveThe two layers nobody defends in a budget review4 min▶
Look again at the bottom of the diagram. Layers 1 and 2 - storage, and the table format plus catalog that sits on it - unblock nobody on their own. Nobody's job title is "uses the object store." No business case has ever been written that opens with "we need a catalog."
That is not a curiosity. It is the structural reason platform programmes go wrong, and it plays out the same way in every organisation:
- The layers with no visible beneficiary get trimmed first, because trimming them appears to cost nothing this quarter.
- Everything above them is then built on foundations that were sized for a pilot.
- Retrofitting a catalog underneath live workloads costs several times what installing it first would have cost, and the bill arrives at the exact moment the platform is finally being used.
So the leader's move is not to defend the catalog on its own merits. It is to fund it as part of the outcome above it. You are not approving a catalog; you are approving the analyst's trustworthy number, and the catalog is what that number stands on.
Airbnb published this exact progression over six years on their own engineering blog. Dataportal in 2017 solved discovery, because nobody could find the data. Minerva in 2021 solved metric definitions - 12,000+ metrics, 4,000+ dimensions, 200+ data producers - because teams kept reporting conflicting numbers for the same question. Metis in 2023 added governance and lineage, a graph of 100M+ nodes and 300M+ edges serving 1,000+ weekly data users. Three artifacts, one company, in that order. Nobody starts at layer 8, including the companies whose talk-to-your-data demos look effortless.
Self-studyWhy "just buy a platform" does not collapse the list3 min read▶
Databricks, Snowflake and Microsoft Fabric each sell something they call a unified platform, and all three genuinely cover several of these layers. But read their own documentation and they are unifying different axes:
- Databricks unifies workloads - BI and ML running on one copy of the data.
- Microsoft Fabric unifies organisational storage - one OneLake per tenant, and their docs are blunt about the consequence: "You can't delete OneLake or create multiple OneLakes."
- Snowflake unifies workload types including transactional - running operational workloads alongside analytics.
None of these is wrong. They are answers to different pains. So the evaluation question is never "is this platform unified?" - it is "which axis does my actual pain sit on?" A group drowning in duplicate departmental lakes has a completely different problem from a group whose ML and BI numbers disagree, and the two should buy differently. If your team cannot state which axis hurts, the procurement is being driven by the demo rather than the diagnosis.
What actually changed in 2025-2026 24 min live
If you last formed a view on this market in 2023, one thing has changed and it is not the thing the headlines suggest. The models got better, yes. But look at what the platform vendors shipped, and every significant launch is about the same problem: giving an agent enough governed context to answer correctly. That is a coordinated shift in what the platform is for, and it is worth twenty minutes of a leadership team's attention.
| Launch | Date | What it is really about |
|---|---|---|
| Unity Catalog Metrics, GA (Databricks, at DAIS) | 2026-06-16 | KPIs become governed objects in the catalog, not definitions scattered across dashboards. One place to ask "what does revenue mean here?" |
| Snowflake's two semantic layers - Horizon Context and Cortex Sense (at Summit) | 2026-06 | Business meaning shipped as a first-class product surface, so an agent has something authoritative to read before it writes SQL |
| Apache Polaris graduates to Top-Level Project | 2026-02-19 | The open catalog stops being one vendor's project. Catalog neutrality becomes a governable choice rather than a promise |
| MCP donated to the Linux Foundation | 2025-12-09 | How an agent reaches your tools and data becomes shared infrastructure with neutral governance, not a vendor protocol |
| Databricks Lakebase, GA | 2026-02-03 | Managed Postgres inside the lakehouse - operational and analytical state stop living in separate universes |
| Amazon S3 Tables, GA (announced 2024-12-03) | 2025-03-13 | The table format moves into the storage service itself. Open tables become the default rather than a project |
| Iceberg v3 - deletion vectors, row lineage, VARIANT and geospatial types (Databricks shipped v3 GA plus managed Iceberg in Unity Catalog 2026-05-28) | spec 2025 | Row-level change tracking becomes a query rather than a nightly diff job, which is what agent-era freshness needs |
LiveThe shift, in one sentence6 min▶
Here is the line to take away from this session:
Consider how differently those two questions procure. The first is answered by a storage and warehouse decision, and once made it stays made for years. The second is answered by the quality of your metric definitions, your table and column descriptions, your access rules and your lineage - all of which are work your own people do, not a product you install. That is why the vendors spent eighteen months shipping context and semantic products: they are selling you the container for work only you can do.
Three consequences for how you sponsor this:
- Metadata quality became a capital item. Descriptions and definitions used to be hygiene. They are now the input that determines whether an AI answer is right, which makes them fundable.
- Neutral governance of the connective tissue is now real. Polaris at the Apache Foundation and MCP at the Linux Foundation both moved the same way in the same window. Ask any vendor how their agent access path relates to those two.
- "Unified" moved up the stack. The unification fight in 2023 was about storage and formats. In 2026 it is about meaning. That is a much harder thing to buy and a much better thing to own.
LiveWhat this does not mean5 min▶
A market moving in one direction is not the same as a problem being solved, and a leader's job here is to separate the two.
- The format war is over; the catalog contest is not. Open table formats have effectively converged in practice - the arguments moved to the catalog, because the catalog is what controls access, vends credentials and sequences commits. The vendors know this. It is why the last three years of acquisitions and donations all cluster there.
- Interoperability translates metadata, not data. When a vendor says their format talks to another format, what crosses is the metadata dialect. The underlying files were always compatible. This is a smaller claim than it sounds and worth checking every time you hear it.
- Announced is not shipped. Roadmap convergence between the major table formats has been proposed, not delivered. Treat proposals as proposals in your planning, however confident the keynote was.
- Nobody has solved talk-to-your-data. Session a2 shows you the failure mode and the practitioner track spends a whole session on the evidence. For now, hold this: the tools are real and useful, and the framing of "ask anything of all your data" is not.
The healthy posture is neither dismissal nor enthusiasm. It is: the ground under the context question genuinely moved, our own semantic groundwork is the part that determines whether we benefit, and that groundwork is not something a purchase order can produce.
Self-studyThe dates worth remembering, and why3 min read▶
You do not need to memorise a product timeline. You need three anchors so that you can tell whether a vendor is describing 2023 or 2026:
- 2025-12-09, MCP to the Linux Foundation. If someone pitches you a proprietary agent connectivity story, this is the date that makes you ask why.
- 2026-02-19, Polaris to Apache Top-Level Project. If someone tells you catalog neutrality is theoretical, this is the date that says otherwise.
- 2026-06, the semantic launches. Unity Catalog Metrics GA on 2026-06-16 and Snowflake's Horizon Context and Cortex Sense in the same month. Two competitors solving the same problem in the same weeks is the strongest available signal that the problem is real.
One honest caveat to carry with all of it: this subject goes stale faster than anything else you sponsor. Every date and price in this course carries the date it was checked, and you should demand the same discipline from anyone briefing you.
Five questions that unstick a stalled platform conversation ◐ 6 min read
Most platform discussions stall because they are being held at the wrong altitude. The room is arguing about which product to buy when the actual disagreement is about which layer is missing. These five questions move the conversation from "which product" to "which layer", and they work in a steering committee, a vendor meeting or a corridor.
1 · "Which of the five people does this unblock, and what can they not do today?" A product that cannot be tied to a named person doing a named job is a product looking for a problem. If two people in the room name different personas, you have found the real disagreement.
2 · "Which of the eight layers does this fill, and which layers does it assume we already have?" The second half is the one vendors skip. Almost every disappointing platform purchase was a layer-8 product bought by an organisation missing layers 2 and 5.
3 · "If two teams answer this question differently today, where would we go to settle it?" If the answer is a person rather than a system, you are missing the semantic layer, and no amount of AI will fix it - it will only produce disagreement faster and more confidently.
4 · "What stops someone going around this control?" The chokepoint test, which session a2 builds out in full. If the answer is "policy" rather than "architecture", you have a suggestion rather than a control.
5 · "What would an agent read before answering this question, and who wrote it?" The 2026 question. It converts an abstract AI ambition into a specific, assignable piece of work: someone has to write the definitions, and the honest answer is often "nobody has."
Three things to do this week ◐ 30 min total
- Map your own organisation onto the eight layers. For each one: which product fills it, or is it empty? An empty layer is not automatically wrong. An empty layer with a persona depending on it is a diagnosis, and it is the first slide of your funding case.
- For each of the five personas, write one sentence describing what they cannot currently do. Those five sentences are your roadmap in priority order, and they persuade a budget holder far better than a tool list does.
- Pick the last platform product your organisation bought and answer question 2 about it in writing: which layer did it fill, and which layers did it assume? Bring the answer to a2. If it assumed a layer you do not have, you already know why adoption is disappointing.
Official sources covered
This track is built on vendors' own documentation and billing feeds, published engineering blogs from the companies that built these platforms, and peer-reviewed work where it exists - with marketing claims labelled as such. Every price and product date carries the date it was checked. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · A vendor tells your board their platform "covers the whole stack." What is the eight-layer frame actually telling you to check?
One product can cover several jobs, and some single jobs need three products. Every pitch is a claim about which boxes get filled, and the revealing half of the question is always the assumed layers rather than the delivered ones.
2 · Storage and the catalog are the first two layers to get trimmed in almost every budget review. Why?
Nobody's job title is "uses the object store." Foundation layers pay off indirectly, through every layer above them. Fund them as part of the outcome they enable rather than on their own merits, or they will lose every time they are argued in isolation.
3 · Unity Catalog Metrics went GA on 2026-06-16, Snowflake shipped Horizon Context and Cortex Sense at Summit in the same month, and MCP went to the Linux Foundation on 2025-12-09. What is the common thread?
Two direct competitors solving the same problem in the same weeks, plus neutral governance arriving for both the catalog and the agent connectivity protocol, is the strongest available signal that the industry agrees on where the bottleneck is. The uncomfortable part for a sponsor: the context itself is work your own people have to do.