How this track works
▶ Leaders start here. Six sessions, forty-five minutes each, and you will not write a line of SQL in any of them. This track exists because the people who approve the tooling spend and referee the "why do these numbers differ" meetings need a different thing from the people who build the models. You need to know what the category buys, what it costs, what to fund, who owns what, and which questions to ask a team that knows more about the mechanics than you do.
Today is the why. a2 is what changes when SQL lives in git. a3 reads a test suite as a governance artifact. a4 owns buy versus build, including the Core versus Cloud decision. a5 owns team shape and ownership boundaries. a6 owns documentation and lineage. Nothing here pre-empts those four; when a question belongs to a later session, this page says so and moves on.
Every number quoted in this track comes from one project in this repository, and that project is unusual in a useful way: the same marketplace warehouse exists twice. Once hand-written, as eight SQL files plus a Python validator with 27 checks. Once as a dbt project, with 23 models, 100 tests, one snapshot and eight declared sources. Both produce settled net revenue of 111,906.48 over 957 orders, average order value 116.93, identical to the cent. So when this track compares the two ways of working, it is comparing two things that actually exist rather than a tool against a straw man.
What changed, and what it created 12 min live
The role did not appear because somebody wanted a new title. It appeared because a cost curve moved and the work relocated. For most of the history of data warehousing, transformation happened before the warehouse, inside an extract-transform-load tool, because warehouse storage and compute were the expensive part and you only loaded what you had already shaped. Cloud warehouses inverted that. Storage got cheap and compute became something you rent by the second, so the cheapest thing to do became: load the raw data as it is, and shape it afterwards, in the warehouse, in SQL.
LiveWhy "it is now SQL" is the whole story4 min▶
When transformation lived inside an extract-transform-load tool, it was a vendor product with a graphical interface, a licence, a specialist, and a change process. Slow, expensive, and constrained. Also, quietly, governed: there was a small number of people who could change it and a tool that recorded what they did.
When transformation became SQL in a warehouse, all of those constraints fell away at once. Anybody who can write a SELECT can add a transformation. That is a genuine unlock, and it is why the modern data stack got adopted so fast. It is also why estates degrade so predictably:
- The volume grows faster than anyone expects. A warehouse that starts with four transformations has forty inside eighteen months, because each one is cheap to add and nothing forces a decision about whether it should exist.
- Nothing records intent. A SQL file changed in place tells you what it says today, not what it said last quarter or why somebody changed it. The reason lives in a person's memory, and people leave.
- There is no natural review point. A change to a definition and a change to a formatting preference look identical when both are "somebody edited the script".
Self-studyBazaar, and why this course rebuilt the same warehouse twice3 min read▶
Bazaar is a multi-merchant marketplace: 12 merchants, 60 products, 985 orders, 1,060 payment attempts, 90 days of history. Small enough to hold in your head, complicated enough to be honest, and the domain is the kind most organisations actually have.
The sibling course learn-data-modeling built Bazaar's warehouse by hand: table definitions, a transformation script, four marts, and a Python validator with 27 checks. It works. Nothing about it is a caricature, and the person who wrote it was not careless.
This course rebuilt the identical warehouse as a dbt project: 23 models (8 staging views, 3 ephemeral intermediate models, 12 mart tables), 100 tests, one snapshot, 8 declared sources. A full build reports PASS=121, WARN=0, ERROR=0 in 1.22 seconds on a laptop. Its lineage graph has 32 nodes and 47 edges, derived from the SQL rather than drawn by anybody. It was built on dbt-core 1.12.0 with dbt-duckdb 1.10.1.
The point of doing it twice is that the comparison is measurable rather than asserted. Both versions produce settled net revenue of 111,906.48 over 957 orders, AOV 116.93. Same numbers, to the cent. Whatever the second version bought, it was not accuracy. Hold onto that sentence, because it disqualifies most of what gets said about transformation tooling in a vendor meeting.
Self-studyWhy the same thing happened in software twenty years earlier2 min read▶
None of this is new. Software went through it: as soon as code became cheap to write and deploy, the discipline moved from "who is allowed to change it" to "how do we make changes safe" - version control, review, environments, automated tests. Every practice in this track is borrowed from that period, and the borrowing is deliberate rather than fashionable.
Which is also why the categories outlive the products. dbt is the subject of this course because it is the tool most teams will meet, and this repository's project is a real dbt project. But the durable ideas are: transformation as a reviewable artifact, dependencies derived rather than maintained, assertions that run before anybody sees a number, documentation generated from the thing it documents. Those will survive whatever replaces today's tools, and a leader who learns the categories rather than the buttons does not have to relearn this in five years.
The three jobs that were nobody's 10 min live
Here is the most useful way to think about the role, and it has nothing to do with tools. Between your raw data arriving and a number appearing on a dashboard, three things have to be owned. In most organisations that have not made this hire, all three are partially owned by everybody, which is a specific and recognisable failure mode rather than a vague one.
| The job | What owning it means | The symptom when nobody owns it |
|---|---|---|
| The transformation layer | Every model between the raw tables and the dashboard: what exists, what it is called, what it is built from, when it gets rebuilt, and what gets retired. | One transformation script over fifteen models that three people edit, nobody refactors, and everybody works around by writing their own version. The estate nobody dares change. |
| The definitions | What a word means, once, in one place. Active customer, settled revenue, a month. Including the authority to say no to a second version of an existing word. | Two teams presenting different numbers for the same word, both defensible, resolved verbally in a meeting and then forgotten until the next meeting. |
| The tests | The assertions that must hold before anybody sees a number, plus the judgement call about which failures stop the world and which are worth only a note. | A validation script somebody remembers to run, and finance finding the error before the data team does. Trust that decays without any single visible incident. |
LiveWhy the definitions job is the one that gets skipped4 min▶
Of the three, the transformation layer is the one everyone recognises, because unowned SQL is visible. The tests job gets recognised after the first embarrassing incident. The definitions job is the one that gets skipped, for a reason worth naming out loud: it is the only one of the three that requires authority rather than skill.
Deciding what "active customer" means is not a technical act. It is a decision that costs somebody something, because at least one team is currently using a definition that will lose. A person with no standing cannot make it, and a person with standing usually has more urgent things to do. So the decision does not get made, and instead the warehouse accumulates three definitions of the same word, each locally correct.
This matters to you specifically, because the fix is not the hire. You can hire an excellent analytics engineer who owns the transformation layer beautifully and still lose the definitions job, because nobody backed their right to say no. That is the part only a leader can supply.
Self-studyWhy the transformation layer is the one that gets dropped2 min read▶
The boundaries between roles move between companies and that is fine. What is not fine is a boundary nobody has stated, because the layer that gets dropped is always the middle one, and for entirely reasonable reasons on all sides.
Data engineers reasonably decline to own business meaning: they cannot arbitrate what settled revenue means, and they should not be asked to. Analysts reasonably decline to own a shared table that four other teams depend on: that is a maintenance commitment with no end date, and it competes with the analysis they are measured on. So the models in between get maintained by whoever needed them most recently, which is how you end up with eleven tables and no owner.
This is worth understanding as a structural problem rather than a motivational one. Nobody is shirking. The work sits between two job descriptions, and work between job descriptions does not get done, it gets absorbed unevenly until somebody burns out or leaves.
Where it sits, and what it is not 5 min live
Three lanes, one pipeline. The two mistakes below are common enough to be worth pre-empting out loud, because each of them produces a bad hire, a disappointed team and an unhappy person, and both are easy to make from a distance.
LiveThe two mistakes, and the test that separates the three3 min▶
- Not a renamed analyst. An analyst answers a question. An analytics engineer builds the tables the answer comes from. The overlap is SQL and almost nothing else: an analyst optimises for a defensible answer this week, an analytics engineer optimises for that answer still being defensible in a year, after four other people have edited the model. Retitling your analysts does not create the capability, and it usually costs you analysis, because the new work is genuinely different work and it expands to fill the week.
- Not a junior data engineer. The hard part of data engineering is infrastructure, throughput, cost and failure at scale. The hard part of analytics engineering is business meaning holding still while the SQL underneath it changes. Neither is a subset of the other, they fail in different ways, and they need different reviewers. Treating the role as a stepping stone into data engineering tells your strongest candidates the job has no ceiling, and they will believe you.
A useful test when you are reading a job description, an internal proposal or a vendor's positioning: if the deliverable is a report, it is analysis. If the deliverable is a pipeline or a platform, it is data engineering. If the deliverable is a table other people build on, it is analytics engineering.
Self-studyWhat to do when there is no headcount2 min read▶
The honest answer is that a great many organisations get the capability before they get the title, and that is a legitimate route rather than a compromise. What does not work is leaving it implicit.
The minimum viable version is one named person who owns the transformation layer for one subject area, with a stated fraction of their week protected for it and a stated reviewer for their changes. Not a committee, not a rota, and not "everyone is responsible for data quality", which reliably means nobody is. The named person is doing the job even if their business card says analyst, and the protected time is what stops the job from silently becoming a hobby.
What you are buying with that arrangement is a place for the three jobs to sit. Sessions a4 and a5 are about how far that arrangement stretches before it needs real headcount and real tooling, and both give a number rather than a feeling.
The two-signal audit ★ run this with your own team
No slides and nothing to install. Both exercises below are designed to be run verbally in a meeting you already have on the calendar, and both are diagnostic rather than rhetorical: you are trying to find out whether you have the problem, not to build a case for buying something.
Ask the script question first, and let somebody count out loud. The number matters less than how confidently it arrives. Hesitation is itself the finding.
Then ask the meeting question. Do not ask whether definitions are inconsistent, which invites a reassuring answer. Ask for a specific past incident, which invites a real one.
Write both answers down, verbatim, with the date. If you decide to fund the role, this is the before-picture, and in nine months nobody will remember how bad it was.
LiveExercise 1 · The script question6 min · run it next week▶
The prompt, word for word
"Name the biggest transformation script or workbook we own. Roughly how many tables or views does it produce? And who reviewed the last change that went into it?"
What a good answer sounds like
Specific and slightly boring. "It is the nightly revenue build, it produces about eleven tables, and Priya reviewed the last change on a branch before it merged." That is a team that has already solved this, whatever they call it. A different good answer is honestly bad news delivered clearly: "It produces about thirty things, four of us edit it, and nobody reviews anything because we are the only people who understand it." That is a team telling you the truth, and it is the answer that justifies the hire.
The failure mode to listen for
Vagueness that sounds like competence. "It is all in the warehouse", "it depends on the domain", "the pipeline handles that". Nobody is hiding anything; there is genuinely no answer, because no single person can see the whole of it. Also listen for a count that comes back suspiciously low followed by a correction, and for the phrase "and then there is the other one" - a second script somebody maintains privately is the same finding with a worse ending.
LiveExercise 2 · The meeting question6 min · run it next week▶
The prompt, word for word
"Think of the last time two teams brought different numbers for the same word into the same meeting. Who decided which one was right, and where is that decision written down today?"
What a good answer sounds like
An actual incident, named, with an outcome you can go and look at. "Marketing and finance disagreed on settled revenue in March, we agreed finance's version, and it is now the definition in the revenue model with a description on the column." That is the definitions job being owned. Note that the second half of the answer is the whole test: a decision that exists only in a meeting was not a decision, it was a truce.
The failure mode to listen for
"That does not really happen here." It happens everywhere that has more than one team and more than one dashboard, so this answer usually means the differences are not being surfaced, which is worse than an argument. The second failure mode is a decision everyone remembers and nobody can locate: "we agreed it is the finance one" with no artifact behind it. That definition will drift back within two quarters, and it will drift back silently.
Bazaar is the honest version of this, because both estates exist and both are visible. The hand-written warehouse is eight SQL files and a 27-check Python validator that a human has to remember to run. The dbt version is 23 models and 100 tests that run on every build, in 1.22 seconds. Neither is a caricature and both were written carefully, which is why the comparison is worth anything.
And the result of the comparison is deliberately unglamorous: 111,906.48 of settled net revenue over 957 orders, AOV 116.93, from both versions, identical to the cent. The tooling did not improve the numbers. It changed who could safely change them, how fast a mistake surfaces, and whether the reason for a definition survives the person who chose it. If a vendor tells you their transformation layer will make your numbers more accurate, they are describing something else, and you should ask them which of the three jobs above their product actually owns.
Questions to ask your data team 6 questions
Each of these is answerable in under a minute by a team that has the situation under control, and each one is diagnostic rather than leading. None of them requires you to read SQL.
| Ask this | Why it works |
|---|---|
| "Which of our transformations would break if the person who wrote it left this month?" | Turns an abstract worry into a named list. Teams answer this one honestly because it is not an accusation, and the list is usually shorter and more specific than you feared, which makes it actionable. |
| "Who owns the definition of our single most quoted metric, and can they say no to a second version of it?" | Separates the two halves of the definitions job. Most teams can name an owner; far fewer can say that person has the standing to refuse. The gap between those two answers is the thing only you can close. |
| "How would we find out that a number on a dashboard has been wrong for a fortnight?" | Asks about detection rather than prevention, which is the question that separates a tested estate from a hopeful one. If the answer is "somebody would notice it looks odd", you have no detection, you have luck. |
| "When we change what a word means, what happens to the dashboards that already use it?" | Surfaces whether anybody can see downstream impact before shipping a change. A team that can answer has lineage; a team that cannot is changing definitions blind, and finds out from a stakeholder. |
| "If I asked for one new table, would you rather add it to something that exists or start fresh, and why?" | A wonderfully revealing question. "Start fresh" almost always means the existing estate is too frightening to touch, which is the script signal arriving by a different route and in the team's own words. |
| "Which of these three do we already own: the transformation layer, the definitions, the tests?" | Ask it exactly like that, as a checklist, and let people disagree with each other in front of you. Watching where the disagreement lands tells you more than the answers do, and it makes the gap collective rather than personal. |
Before session a2 ◐ 30-40 min total, all conversations
- Run the script question and the meeting question with your data team, and write both answers down verbatim with the date. Ten minutes, and it is the only before-picture you will ever get.
- Pick your single most quoted metric and ask three different people what it means, separately. Compare the answers. If they differ, you have found the definitions job, and you did it without opening a query editor.
- Look at one job description your organisation has posted for a data role in the last year, and decide which of the three jobs it was actually hiring for. Very often it was written for two of them at once, which is why the hire disappointed everybody.
- Ask your team who reviewed the last change to your most important transformation, and how long the review took. Bring the answer to a2, because review culture is the precondition for everything in that session.
- Optional and worth it: skim learn-data-modeling's front page for twenty minutes so you know what the hand-written version of Bazaar looks like. This whole track compares against it.
Sources covered
This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Your head of data says the warehouse team is fine because both transformation scripts were written by the same careful person. What is the strongest follow-up?
The signal is never carefulness or volume, it is how many people edit shared SQL and whether anybody reads the change. A careful author is exactly the situation that degrades quietly, because it works until they go on holiday or leave.
2 · Rebuilt as a dbt project, Bazaar produces settled revenue of 111,906.48 - identical to the cent to the hand-written version. As a leader, what should you conclude?
Identical numbers are the honest baseline. Anything that claims to improve your numbers is really changing a definition. What actually changed is who can safely change the SQL, how fast a mistake surfaces, and whether the reasoning survives the author - and those are the only grounds on which to fund it.
3 · Of the three previously unowned jobs, which one cannot be fixed by the hire alone?
At least one team is currently using a definition that will lose, so the decision costs somebody something. A skilled analytics engineer with no backing gets overruled quietly and stops trying. That backing is the part only a leader supplies, and session a5 works out how to structure it.