The battleground moved to the catalog

Before the REST spec, catalog access was a per-engine integration problem. Spark talked to Hive Metastore one way. Trino talked to Glue another way. Every engine needed a bespoke connector for every catalog, which meant the number of things to build and maintain was engines times catalogs, and any new capability had to be implemented independently in each cell of that grid. In practice that meant new capabilities did not arrive.

The REST spec replaced all of that with plain HTTP. The catalog became a service with a documented API rather than a library you linked against. That sounds like plumbing. Look at what it made possible, because each item is a governance capability that could not exist before:

• Credential vending. The catalog hands the engine a scoped, temporary storage credential instead of the engine holding a long-lived key to the whole bucket. Access decisions move from cloud IAM, which knows about prefixes, to the catalog, which knows about tables.
• Remote signing. The catalog signs individual storage requests, so the engine never holds a credential at all.
• Server-side commit conflict resolution. Two writers racing on the same table get sequenced by the catalog rather than by whichever client's retry logic happens to be better.
• Scan planning - and this is the one to underline. The catalog applies row filters and column masks BEFORE returning a plan.

Sit with that last one, because it is the difference between governance and a suggestion. If filtering happens in the engine, then governance is only as strong as every engine's willingness to apply it, and a new engine is a new hole. If the catalog applies the filter during scan planning, the restricted rows are never in the plan the engine receives. The engine cannot read what it was never told exists. That is architecture, not policy - and by the b1 chokepoint test, that is the difference between a control and a suggestion.

A format is a file layout. If you dislike it, translation is metadata-sized and cheap, as b2 established - XTable and UniForm rewrite nothing. That is precisely why the format war de-escalated: the switching cost turned out to be low, so owning the format stopped being worth fighting for.

A catalog is not a file layout. It holds your access policies, your role hierarchy, your table-level and row-level and column-level rules, your audit trail, and the identity mappings that connect all of it to your organization. None of that is standardized by the REST spec. The spec standardized how an engine asks for a table. It did not standardize how you write "analysts in the EU may see these rows."

So the industry standardized the cheap-to-switch layer and left the expensive-to-switch layer proprietary. That is not a conspiracy, it is the ordinary result of standardizing what everyone agrees on first. But it does mean the interoperability story you are sold at the format layer does not automatically extend one layer up, and Part 3 is about exactly that gap.

Group them by the question they answer and eight options become four decisions:

• "I want the open standard, self-hosted" - Polaris or Lakekeeper. Polaris has the widest published engine interop; Lakekeeper's OpenFGA authorization model is the more interesting choice if fine-grained permissions are your actual pain.
• "I want it managed, and I have already picked a platform" - Snowflake Open Catalog, Unity Catalog, Glue / S3 Tables or BigLake Metastore. Here the catalog decision is mostly downstream of a decision you already made, which is worth naming out loud rather than pretending it is an open evaluation.
• "I need multi-table branching and merge workflows" - Nessie, with clear eyes about the missing access control. Teams adopt Nessie for its development workflow and then discover governance is a separate project.
• "I already have several catalogs and cannot consolidate" - Gravitino. Federation is the honest answer to an organizational fact, not a technical preference.

Take these into your next evaluation. Each maps to something on this page.

• "Show me a row filter being applied during scan planning." Not in the engine. In the plan the engine receives. If it is applied downstream, every new engine is a new hole.
• "What exactly does an engine hold - a long-lived key, a vended temporary credential, or nothing because you sign requests?" Three very different blast radiuses when a client is compromised.
• "If I write my row-level and column-level policies here, what is the export path?" Expect discomfort. Part 3 explains why.
• "What is your availability target, and what happens to reads and writes when you are down?" Answered honestly, this is a tier-1 conversation.
• "Which capabilities exist in the managed product but not in the open-source one?" Ask specifically about governance features, which is where the gap usually sits.

Any vendor who tells you this layer is solved is selling. These three are genuinely open, and the useful posture is to name them and design around them rather than wait for someone to close them.

• 1 · Governance portability. No standard exists for expressing row filters, column masks, role hierarchies or tag-based policies across catalogs. Practical mitigation: author your policies as code in your own repository and apply them to the catalog, so that the source of truth is a file you own rather than a state you accumulated by clicking. That does not make migration free, but it turns a rewrite into a re-render.
• 2 · The open-versus-managed parity gap. The open-source variant of a managed catalog routinely lags the managed one, and it lags most in governance features specifically - which are the expensive ones to reimplement. Evaluating the open-source version and deploying the managed version, or the reverse, produces a surprise in month four rather than in week one.
• 3 · Operational reliability. The catalog is a tier-1 service: if it fails, all lakehouse reads and writes halt. Not degrade - halt. Every engine resolves every table through it. Many teams deploy a catalog with the operational care of a metadata service and then discover they have made it the availability floor of the entire data platform. Ask for the SLO, the failure modes, the backup and restore procedure for catalog state, and whether anyone has ever practised the restore.

Now apply the b1 chokepoint test explicitly: what stops someone from going around the catalog? On a lakehouse, the honest answer is often "a direct read from the object store with cloud credentials." If any engine, notebook or service can read the Parquet directly with bucket-level IAM permissions, every row filter and column mask in your catalog governs only the well-behaved path. The catalog is a genuine chokepoint only when storage-level access has been closed off in favour of catalog-vended credentials. Credential vending is not a convenience feature. It is the mechanism that makes the chokepoint real, and without it you have a suggestion with excellent documentation.

Lineage is the record of which inputs produced which outputs, at table and ideally column granularity. Four things it buys, all of them boring until the day you need them:

• Impact analysis - before you change a column, know who breaks.
• Root-cause analysis - a dashboard is wrong, and lineage turns "somewhere upstream" into a short list.
• Regulatory evidence - showing where a personal-data field came from and where it went, on demand.
• Deprecation - the only way to retire a table with confidence rather than by deleting it and waiting for complaints.

The trap is treating lineage as a procurement item: buy a lineage tool, run parsers over your SQL, produce a graph that is 70% accurate and stale by Thursday. It is expensive and it decays, because it is reconstructing after the fact something the system knew at the time and threw away.

The alternative is structural. In an asset-based orchestrator, you declare the assets and their dependencies as the way you author work in the first place, so a successful run means "this table exists at this version, produced from these inputs." Lineage falls out of that as a by-product rather than a separate project, and it is accurate by construction because it is the same declaration the scheduler executed. That is a change in what "the pipeline succeeded" even means, and it is the heart of b4, where we take orchestration properly and choose between tools by failure semantics rather than by feature list.