The lakehouse engine

The argument in one sentence: turn data lakes built on open formats into systems that provide warehouse-grade performance and management, while keeping direct file I/O for advanced analytics. That last clause is the part people skip, and it is the whole reason the data scientist ends up on this layer rather than fighting a warehouse.

The paper names four enablers, and they map cleanly onto the layers you have already built:

• A transactional metadata layer - this is b2 and b3. Open table formats and the catalog.
• Caching - keeping hot data close to compute rather than paying to read it from object storage repeatedly.
• Auxiliary data structures - statistics, indexes, and the metadata that lets a scan skip files it does not need.
• Data layout optimization - clustering, partitioning and file sizing, so the physical arrangement matches the query pattern.

Notice that three of the four are performance engineering, not new abstractions. The lakehouse claim is not that a new kind of database was invented. It is that with a transaction log and enough layout work, a lake can be fast enough that you stop needing a second copy of everything in a warehouse.

Use the vendor's own glossary rather than a blog paraphrase, because the vendor's version is more careful than the folklore:

• Bronze - "land all the data from external source systems... as-is." The stated purposes are change data capture, historical archive, lineage, and the ability to reprocess without going back to the source.
• Silver - data that is "matched, merged, conformed and cleansed," so the organization gets "an 'Enterprise view' of all its key business entities." Four verbs, and every one of them is real work.
• Gold - consumption-ready "project-specific" databases, typically de-normalized and read-optimized.

That is a coherent design. Bronze buys you re-playability, silver buys you agreement about entities, gold buys you query speed for a known consumer.

The most common failure mode is a silver layer that is just bronze with renamed columns. Nobody matched, nobody merged, nobody conformed anything to a shared entity definition. The medallion diagram on the wall says the enterprise view exists; the tables say a rename happened. Because nothing enforces the convention, nothing catches this - it is caught by a person, in a review, or not at all.

The second is a gold layer that has quietly become 40 unmaintained project-specific tables. "Project-specific" is in the definition, so each one was legitimate on the day it was built. The failure is cumulative: no owner, no retirement, no register of which of the 40 anybody still queries. This is the same shape as the governance-debt pattern from the data-lake literature, arriving through the front door rather than the back.

Both failures are diagnosable in an afternoon. For a sample of silver tables, ask which entities were conformed and against what. For gold, pull query logs and count how many tables were read in the last 90 days.

The Fabric documentation is admirably blunt, and this verbatim line is the most useful sentence in the whole comparison:

"You can't delete OneLake or create multiple OneLakes, and there's no infrastructure to provision or manage."

Both halves of that sentence are the same design decision seen from two sides. Nothing to provision is a genuine operational gift, especially for an organization currently drowning in departmental storage accounts nobody can inventory. And it is also the statement that your storage topology is now Microsoft's tenant model - a shape you did not choose and cannot change.

Fabric does provide reach outward, and it is worth knowing what kind of reach:

• Shortcuts to external sources including S3, ADLS and Iceberg data - references rather than copies.
• Mirroring from Snowflake and Databricks.
• Metadata virtualization, so that "Iceberg tables can be read as Delta Lake tables."

Apply the b2 rule here: that last one translates metadata, not data. The Parquet files were always compatible. What differed was the metadata dialect, and a translation layer is a second source of truth that can drift from the first.

Ask which axis your actual pain sits on before you choose anything. Three organizations, three different right answers:

• "Our ML features and our BI dashboards disagree about what a customer is." That is a workload problem.
• "We have 60 storage accounts, no inventory, and three teams paying for copies of the same data." That is an organizational storage problem.
• "We run a nightly export from the application database and every analytics answer is a day old." That is a workload type problem.

A team drowning in duplicate departmental lakes and a team whose ML and BI numbers disagree will sit through the same three demos and should buy differently. If you cannot state your pain in one of those three sentences, the honest conclusion is that you are not ready to pick a platform yet - which is a cheaper finding to reach in a workshop than in year two of a migration.

"Unified" is a claim with a hidden argument place: unified along what. Every time you meet the word, finish the sentence for the vendor before they do. Then ask the follow-up that separates a platform claim from a product claim: what is now impossible to do separately?

With OneLake the answer is concrete - you cannot create a second one. That is a strong form of unification and you should price it as such. Where a vendor's answer is "nothing, you can still do it the old way," the unification is a default rather than a constraint, which is a very different purchase.

Every layer in this course has a narrow waist where truth is decided. For storage it was egress economics; for the table format it was the catalog. For the engine it is the compute contract - and what it decides is who pays for a badly written query.

That sounds like a finance question and it is really a design question, because the answer shapes behaviour:

• If compute is a shared pool with no attribution, a single unbounded scan is somebody else's dashboard timing out, and nobody can name the culprit.
• If compute is attributed per team, the cost of a bad query lands on the team that wrote it, and query hygiene becomes self-correcting without a policy document.
• If compute is attributed per query and visible to the author before they run it, you get the strongest version - feedback at the moment of the decision.

Apply the chokepoint test from b1 as well: what stops someone going around this? If any team can spin up its own warehouse or cluster outside the attribution scheme, your compute contract governs exactly the people who were already being careful.

The direction of travel is real and worth naming, but it is easy to overstate, so here is the careful version.

What has shipped: Snowflake, historically the warehouse, now positions itself around "transactional applications, analytics and AI on one unified platform" - which is a warehouse reaching outward into workload types warehouses used to refuse. Fabric reads external Iceberg and ADLS data through shortcuts, mirrors from Snowflake and Databricks, and virtualizes metadata so that "Iceberg tables can be read as Delta Lake tables." Engines are learning to read each other's tables, and that part is not marketing.

What has not shipped: the deeper format convergence is a set of proposals, not a delivered capability. Do not teach it as accomplished fact, and do not let a roadmap slide enter a decision document as a current feature. The b2 rule applies without exception: translation layers move metadata, not data, and derived metadata is a second source of truth that can drift from the authoritative one.

The practical reading for an architect: assume you will be reading other people's tables, plan for it, and assume you will still be choosing one authoritative writer per table for the foreseeable future.