Why this decision lands on your desk
You will not write the pipeline code. But you will approve the budget, sign the contract, and answer to finance when the data-tooling bill arrives. This session is about spending that money well: understanding what each layer of a modern data stack actually does, when to pay a vendor to run it, and when running open source or building your own is worth the headcount it costs. The goal is not to make you a buyer or a builder - it is to make you ask the right questions before either.
The modern data stack, in layers 8 min live
The phrase "modern data stack" sounds like one product. It is not - it is a handful of specialist layers, each with its own vendors, wired together. Ingestion tools copy data out of sources. Storage holds it cheaply. A transform layer reshapes it. Orchestration schedules the whole thing. Serving hands it to consumers. Knowing the layers tells you exactly what you are buying, and what you are not.
LiveWhat each layer actually buys you3 min▶
Each layer solves one problem, and you can buy, rent, or build each independently. That modularity is the whole point of the modern stack - you are not locked into one giant suite.
- Ingestion / EL: Fivetran, Airbyte, and dlt copy data out of your sources on a schedule. This is the "get it here" layer - the fragile, unglamorous part most teams underestimate.
- Storage: cheap durable object storage (S3) plus a warehouse (Snowflake, BigQuery, Databricks, or DuckDB at small scale) where the data lives and gets queried.
- Transform: dbt has become the default here - it turns raw landed tables into clean, tested, documented ones, all in SQL.
- Serving: the layer your business actually sees - dashboards, models, AI features. Everything upstream exists to feed this.
Self-studyWhy orchestration is a different course2 min read▶
Orchestration (Airflow, Dagster) is the copper layer in the diagram: it schedules jobs, wires the layers together, retries failures, and pages someone at 2am. It is essential - but it is an operations discipline, not a data-engineering one, so it lives in the learn-dataops course. In this course we choose and connect the ingestion, storage, transform, and serving layers. How you run them reliably in production is the DataOps job. Keeping that line clear stops your team from re-buying the same capability twice.
Self-studyThe stack is assembled, not bought whole2 min read▶
There is no single "modern data stack" product on a price list. Your team assembles it from best-of-breed pieces - and that is a feature, because it lets you swap one layer without ripping out the rest. The risk is integration: five vendors means five contracts, five bills, and five places a handoff can break. The strategic question is not "which stack" but "how many layers do we pay someone else to run, and how many do we own." That is exactly the build-vs-buy call in Part 2.
The stack that was really a spreadsheet. A mid-size retailer proudly listed a "modern data stack" in board slides. In reality one analyst hand-ran exports into a shared drive. The vendor logos were aspirational. Naming the layers honestly - which are bought, which are manual, which are missing - turned a vanity slide into a real roadmap.
Managed vs open source vs build-your-own 7 min live
For every layer, you have three ways to get it: pay a vendor to run it (managed / SaaS), run the open-source version yourself, or build it from scratch. Each wins in a different situation, and each has a failure mode. Joe Reis's rule cuts through it: avoid undifferentiated heavy lifting - buy the boring parts everyone needs, and build only the thing that is actually your edge.
| Approach | When it wins | Watch out for |
|---|---|---|
| Managed / SaaS | you want speed, few engineers, predictable ops; the layer is not your differentiator | per-row or per-connector pricing that scales with success; data egress fees; lock-in |
| Open source, self-run | scale or cost makes SaaS painful; you need control or have a compliance reason | the "free" tool costs you salaries; upgrades, security, and uptime are now your job |
| Build your own | the capability truly is your competitive edge and nothing on the market fits | the maintenance tail - you own it forever, and it competes with vendor R&D budgets |
LiveThe "undifferentiated heavy lifting" rule3 min▶
Reis borrows the phrase from cloud economics: work that every company has to do but no customer will ever pay you extra for. Copying rows out of a database is undifferentiated - a thousand companies do it identically. Your pricing model, your recommendation logic, your proprietary metric - that is differentiated. The rule: rent the undifferentiated layers, spend your scarce engineers on the differentiated one.
- Ingestion is almost always undifferentiated: this is why Fivetran and Airbyte exist and thrive - buy it unless you have a genuinely unusual source.
- Your edge is usually in transform + serving: the business logic that makes your numbers yours is worth building and owning.
- Managed by default, OSS by justification: prefer managed until scale, cost, or control gives you a concrete reason to run it yourself.
Self-studyReading the true cost of "free"2 min read▶
Open source has no license fee, which makes it look free on the invoice - and expensive on the payroll. Running Airbyte or Airflow yourself means engineers patching CVEs, chasing upgrades, and getting paged at night. Two senior engineers cost far more than most SaaS contracts. The honest comparison is total cost of ownership: license plus egress plus people plus opportunity cost of what those engineers could build instead. Sometimes OSS still wins - at large scale it often does - but only when you have done that full sum, not just noticed the price tag says zero.
The team that built its own ingestion tool. A well-funded startup decided commercial connectors were "too expensive" and wrote its own ingestion framework. Two years later, three of its best engineers spent most of their week maintaining connectors and firefighting source-API changes - work Fivetran does for a subscription. The build was cheaper on day one and far more expensive by year two. They eventually bought the managed tool and redeployed the team onto the product. The tell: they were maintaining plumbing, not shipping their edge.
Ten questions for any data-tooling vendor 6 min live
When a vendor pitches you, the demo always looks great. The cost and the pain show up later - in the egress bill, the pricing that scales with your growth, the migration you cannot afford. These ten questions surface the things vendors do not lead with. Ask them before you sign, not after.
LiveThe ten questions, grouped3 min▶
Group them by what they protect you from - cost surprises, lock-in, and fit.
- Cost that scales against you: Is pricing per-row, per-connector, or per-seat? What is the data egress / transfer cost? What does the bill look like at 3x our current volume?
- Lock-in and exit: How do we leave - can we export everything, and in what format? Do we depend on a proprietary format? Who owns the data while it is in your system?
- Fit with what we have: Does it work with our existing warehouse? What does it not do that we will still need? What is the real setup effort in weeks?
- The tenth question: Show me a customer our size who left. If they cannot, ask why.
Self-studyRed flags in the sales room2 min read▶
A few answers should raise your guard immediately. "We do not publish egress pricing" often means it is where they make their margin. "Everyone stays, no one leaves" means either a great product or high switching costs by design - find out which. "You do not need your old warehouse anymore" is a bid to become your whole stack and your single point of failure. None of these are automatic dealbreakers, but each one is a signal to slow down, get it in writing, and model the three-year cost before you sign.
The egress bill that dwarfed the license. A company signed a cheap analytics tool, then discovered that moving query results back out - egress - cost more each month than the subscription itself. The pricing page was honest; nobody had asked the egress question. One line in a discovery call would have caught it.
Take this back to your team ◐ 20-30 min total
- Draw your own stack as four layers (ingestion, storage, transform, serving) and label each as bought, self-run, or built. Circle anything that is actually manual.
- Pick your most expensive data tool and apply Reis's test: is it doing undifferentiated heavy lifting, or is it your edge? Should it be?
- Find one open-source tool your team runs "for free" and estimate its true cost - how many engineer-hours a month does it really consume?
- Take the ten questions into your next vendor call. Note which ones the salesperson dodges.
- Identify one layer you are building that you could buy. Bring the pushback you expect to session a5.
Official sources covered
This session teaches the tooling-strategy core of Reis & Housley's Fundamentals of Data Engineering and the DeepLearning.AI Data Engineering certificate (Joe Reis), framed for leaders who approve the spend. Verified as of mid-2026 - tooling and pricing move fast, so re-verify vendor specifics before any purchase. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Which layer of the modern data stack belongs to the learn-dataops course, not this one?
Orchestration schedules and wires the other layers, but running pipelines reliably is an operations discipline - it lives in learn-dataops. This course chooses and connects the ingestion, storage, transform, and serving layers.
2 · What does Joe Reis mean by "avoid undifferentiated heavy lifting"?
Undifferentiated work (like copying rows out of a database) is done identically by every company and earns you nothing. Rent it, and spend scarce engineers on the differentiated logic that makes your numbers yours.
3 · A vendor will not give a straight answer on data egress cost and how you would export your data to leave. What is that?
Egress fees and difficult exits are often exactly where a vendor makes its margin and keeps you captive. If leaving is a mystery, the lock-in is the product - slow down and get it in writing.