Buy vs build the transformation layer

Here is the sentence, and it is worth being blunt about it in front of a procurement conversation: the commercial service does not make your models better, faster or more correct. It makes them somebody else's operational problem.

That is a real and often excellent thing to buy. Managed scheduling means nobody on your team is paged at 3am for a runner that ran out of disk. A browser editing environment means an analyst who is fluent in SQL and uninterested in local tooling can contribute a model this month rather than next quarter. A permissions model that already exists means you are not inventing one.

But notice what none of those are. None of them is a model you could not otherwise have built. If someone tells you the paid tier produces better transformations, they have moved from operations to marketing, and you should ask them to point at the capability.

Option 3 is the one most often chosen by accident. Your loader already runs, it grew a transformation feature, and using it is one click cheaper than any alternative. That is a legitimate reason to start there. It stops being legitimate when nobody has asked the exit question.

• Where do the transformations physically live? If they exist only inside the vendor's interface, you do not have a codebase, you have a configuration. Session a2's whole argument about review and rollback quietly does not apply.
• Can you read them without logging in? Files in your own repository are portable. Definitions inside a product are not, however good the product is.
• What happens to your transformation roadmap? It becomes a subset of their roadmap. If they de-prioritise the feature you need, your only options are waiting or leaving.

None of that makes the option wrong. It makes it a decision with a term attached, and the honest version is: "we are choosing convenience now, and we accept that changing our mind is a rewrite."

Below about fifteen models with a single editor, hand-written SQL plus a scheduler is not a compromise. It is correct. There is no build step to explain, no dependency to install, no templating layer, and one person can hold the whole thing in their head. Introducing a transformation tool there is pure cost.

The problem is that the failure mode above that line is invisible. Nothing breaks loudly. Instead: the run order lives in a document that is slightly wrong, a validator exists that somebody sometimes remembers to run, two scripts compute the same number differently, and onboarding a new analyst takes six weeks because the knowledge is in one head. Every one of those is a tax, none of them is an incident, and taxes do not generate decisions.

This course's own project is the measured version of that. The same marketplace warehouse exists twice: by hand as 8 SQL files plus a 27-check Python validator that somebody has to remember to run, and as a dbt project of 23 models with 100 tests that run on every build. Both produce settled net revenue of 111,906.48 over 957 orders, average order value 116.93, identical to the cent. The tooling changed nothing about the answer and everything about who has to remember what.

Model count is the number people quote, because it sounds like scale. Editor count is the number that actually decides, because it is the number that makes coordination a cost.

With one editor, everything a transformation tool provides can be substituted by memory. She knows the run order. She knows which script is stale. She knows the number is wrong on Mondays. It is not written down and it does not need to be, and the estate runs fine right up until she takes annual leave.

With two editors, memory stops working, because two memories disagree and nothing arbitrates. That is the moment you need the change to be visible before it ships, which is review, which needs a diff, which needs the definitions to be files. Everything in session a2 follows from the editor count going from one to two.

Fifteen is not a magic threshold, it is where three separate costs happen to cross at once for most teams.

• Build order stops being obvious. With eight steps, the order is a paragraph. With twenty-three, it is a graph, and this course's project has 47 dependency edges between 32 nodes. Nobody maintains that by hand accurately, and the failure is silent: a model built from yesterday's input still returns rows.
• Manual checks stop being run. The hand-built version of this warehouse has a validator with 27 checks. It works perfectly and it is optional, which over a year means it runs about as often as anyone remembers.
• Nobody can answer "what breaks if I change this". Which turns every rename into either a three-week discovery process or a decision not to rename, and estates that stop renaming things accumulate names that lie.

If two of those three are already true for you, you are past the line regardless of your model count.

Ask a team to adopt a transformation tool and you have usually asked them to learn three unrelated skills in the same quarter:

• Version control as a daily habit, not an occasional backup. Branches, review, merge conflicts, and the discipline of not editing production directly. Session a2 is about why this is worth it; it is still a real learning curve.
• A templating layer inside their SQL. Analysts who are genuinely fluent in SQL now see syntax that is not SQL in the middle of it. This is the thing they complain about, and the complaint is fair for the first month.
• A layering convention. Which model belongs where, what staging is allowed to do, what a mart is for. This is the part that determines whether the estate is maintainable in two years, and it is the part with no error message.

You can sequence these. Version control first, on the existing scripts, before any tool arrives. Then the tool. Then, once models are landing, tighten the conventions. Teams that try all three in one sprint produce a project that technically builds and that nobody wants to touch.

Documentation and lineage. Not cheap: zero, once the models are in the tool. Descriptions live beside the columns they describe, and the dependency graph is derived from the code rather than drawn by a person. This course's project has a lineage graph of 32 nodes and 47 edges, read from the project's own build output, and nobody maintains it.

That is worth pricing into this decision honestly, because it is the benefit leaders systematically undervalue. A diagram somebody drew is out of date the week after the offsite. A graph derived from the code cannot be out of date, because if it were wrong the build would not have worked. Session a6 is entirely about that difference, and it is the strongest single argument in the business case.

The prompt. "Before this meeting ends, I want two numbers on the whiteboard. First: how many distinct transformations do we have that produce something a person or a dashboard actually queries? Second: how many different people edited at least one of them in the last ninety days? Not estimates. Actual counts, and if we cannot produce them today, say so."

What a good answer sounds like. Somebody says a number, somebody else says "it depends what you count as a transformation", and the room spends four minutes agreeing a definition. That argument is productive: it is the first time anyone has drawn a boundary around the estate. You end with two numbers and a shared definition, and often with the discovery that the count is roughly double what the leadership team assumed.

The failure mode to listen for. Instant confident agreement. "About twenty, and it is basically just me and Ravi." If the numbers arrive with no argument and no source, they are recalled rather than counted, and the recalled version is always smaller and always missing the scripts on somebody's laptop. Ask where the list is written down. If the answer is nowhere, your first number is not a count, it is a hope.

The prompt. "Four pieces of infrastructure sit under this decision: a place to store and review code, a runner that executes things on a trigger, a warehouse, and a scheduler. Which of those four do we already pay for and already use, today, for something else? For each one we own, tell me who administers it."

What a good answer sounds like. Three of the four named, each with an owner: "the platform team runs the code host and the CI runner, the warehouse is ours, and there is a scheduler running the loads that the data engineering team owns." That answer moves you decisively toward self-hosting, because the thing the commercial service sells you is operations you have demonstrably already bought.

The failure mode to listen for. "We have a CI runner but nobody has permission to add jobs to it." That is not owning it. Shared infrastructure you cannot get a change into behaves exactly like infrastructure you do not have, and it is a common reason a self-hosted plan quietly fails six weeks in. Push until you get either a named administrator who has agreed, or an honest no.

The prompt. "Suppose the person who built our current transformation setup resigns on Friday with no handover. Name the individual who picks it up on Monday. Now, in front of everyone, ask that person what they would need in order to be productive within a week."

What a good answer sounds like. The named person says something concrete and slightly damning: "I would need to know what order things run in, and I would need somebody to tell me which of the two revenue queries is the real one." That is a specification for what your tooling and conventions have to produce, written by the person who will actually need it.

The failure mode to listen for. Laughter, then "honestly, we would just rebuild it." That is a truthful answer and it is also the entire business case, quantified. Rebuilding a working estate because nobody can read it is the most expensive thing on this page, and it does not appear on any invoice.