Governance, before anyone says the word platform
Session a2 covered the leak that fakes a good test score. Session a3 covered the dirty inputs that quietly cap it. This session covers the third way a good result fails to survive contact with production: the feature that arrives at serving time is not the feature the model was trained on. Nobody cheated and nothing is dirty. Two teams simply implemented the same definition twice, and the model can tell.
Training-serving skew 15 min live
Google's Rules of Machine Learning defines it in one line: training-serving skew is "a difference between performance during training and performance during serving." Worth noting who says this - the term is Google's, and the feature-store vendors do not use it. That matters, because it means the problem is older than the product category sold to solve it.
LiveThe three documented causes6 min▶
Google lists exactly three causes, and they fail differently enough that it is worth asking your team which one they are looking at:
- A discrepancy between how you handle data in the training and serving pipelines. This is the diagram above. A batch job and an application service each implement the same definition, and the two implementations disagree in some small way nobody noticed - a rounding rule, a timezone, whether today counts.
- A change in the data between when you train and when you serve. Nothing is wrong with either pipeline. The world moved. The model is still answering the question it was trained on, which is no longer the question being asked.
- A feedback loop between your model and your algorithm. The model's own outputs shape the behaviour that becomes tomorrow's training data. This is the subtlest of the three and the hardest to see from a dashboard, because every number looks healthy while the loop tightens.
LiveRule 29: the fix you can ask for without buying anything6 min▶
The doctrinal answer, stated verbatim in Rules of Machine Learning as Rule 29:
Read what it actually inverts. The intuitive design has the warehouse as the source of truth and serving as a copy of it. Rule 29 says the opposite: serving is the source of truth, because serving is where the model will spend its whole working life. If you train on the exact values serving produced, the two paths cannot disagree - there is only one path, recorded once and reused.
This is worth knowing before any procurement conversation. Rule 29 is an engineering practice, not a product. A logging step and the discipline to train from the log will close most of the gap, and it is a reasonable thing to ask for in a quarter without a platform decision attached.
Self-studyWhy skew is harder to spot than a leak3 min read▶
A leak announces itself with a suspiciously good test score, which at least a sceptical reviewer can catch on the day - that was session a2. Skew announces itself with a good test score that is entirely correct for the data it was measured on. There is nothing to catch in the notebook, because the notebook is not wrong.
The gap only appears once traffic is live, usually as a slow disappointment rather than an incident: the uplift is smaller than the pilot promised, the team blames the model, and a retrain is scheduled that changes nothing because the mismatch is in the plumbing. This is why the question above belongs in the design review rather than the postmortem.
When a feature store pays 24 min live
A feature store is two stores wearing one name. Using Feast, the open-source reference implementation, as the vocabulary: an offline store handles historical extraction for training, and an online store handles low-latency serving. Everything else it does follows from having both halves under one definition.
LiveWhat it solves, in the project's own words6 min▶
Three claims, quoted rather than paraphrased so you can hold a vendor to them:
- "Make features consistently available for training and low-latency serving." This is Rule 29's goal, delivered as infrastructure instead of as a practice.
- "Avoid data leakage by generating point-in-time correct feature sets." The leakage session a2 warned about the value that was not knowable yet. This is that guard, applied automatically at the moment features are assembled.
- A single data-access layer. One place models ask for features, rather than each team writing its own path to the warehouse and its own slightly different definition.
Notice that all three are consistency claims. A feature store does not make features better. It makes them the same everywhere, which is a different and often more valuable thing.
LivePoint-in-time joins, in plain English6 min▶
This is the one technical idea in the session worth a leader's attention, because it is the mechanism behind the leakage claim.
When you build a training set, every row is an event that happened at a specific moment. A point-in-time join reconstructs what was known at that exact moment, not what is known now. It scans backward from each event's timestamp, up to a time-to-live window, and takes the most recent feature value that already existed. A value written after the event is invisible to it, which is precisely the discipline session a2 asked for, enforced by the join rather than by somebody remembering.
The Feast documentation flags the part that trips people up: "TTL is not relative to the current point in time." The window is anchored to each event, not to today. A team reasoning about the window as "the last 90 days" is reasoning about the wrong clock, and that misunderstanding produces training sets that quietly disagree with themselves.
LiveWhat a feature store is not5 min▶
Feast is unusually honest about its own boundaries, and repeating that honesty is the fastest way to keep a platform conversation grounded. It is not an ETL system, not an orchestrator, not a warehouse, and not a database. It sits on top of the things you already have; it does not replace any of them, and a budget that assumed otherwise will be short.
It also does not fully solve three problems people expect it to:
- Lineage tracking. Knowing where a feature came from and what depends on it remains a governance job.
- Drift detection. Noticing that a feature's distribution has moved is monitoring, and monitoring is still yours to build.
- Batch feature engineering. The transformations themselves still have to be written and owned by somebody. A store serves features; it does not invent them.
So the honest framing for a steering committee: a feature store solves consistency and point-in-time correctness. If the pitch in the room is that it solves data quality, lineage, or drift, the pitch has drifted past what the technology does.
LiveWhy there are two halves, and what each one costs5 min▶
The two stores exist because training and serving ask genuinely different questions of the same data:
- The offline store answers a bulk historical question. "Give me the value of this feature for four hundred thousand events, each as of its own moment." Throughput matters; a query taking minutes is fine.
- The online store answers one question at a time, fast. "What is this customer's value right now, in a few milliseconds, while a request is waiting." Latency matters; completeness of history does not.
No single system is naturally good at both, which is why the architecture has two, and why the interesting engineering is in keeping them agreed. That is also where the operating cost sits: two systems to run, a sync path between them, and one shared definition that both must honour. Worth knowing before anyone describes the store as a way to simplify the stack - it adds a layer, and earns its place by removing duplicate definitions rather than duplicate systems.
| Signal in your organisation | Does a store pay yet? |
|---|---|
| One model, one team | No. There is no shared definition to protect. Rule 29 logging plus a written definition gets you the same safety for a fraction of the effort. |
| Many models sharing features | Yes. The moment two teams compute "average spend" separately, you are paying for a store already - in duplicated work and divergent numbers, just not on an invoice. |
| Real-time serving plus historical training | Yes. This is exactly the offline plus online split the architecture exists for, and hand-building both halves is where skew is born. |
| Compliance needing point-in-time proof | Yes. When you must show what was known at decision time, a point-in-time correct join is evidence. A reconstructed query is an argument. |
The order of operations matters more than the choice of tool. A team with one model and no shared definitions who buys a feature store gets a new system to run and the same two implementations of the same feature underneath it. A team who writes the definition down, names an owner, and logs the serving-time features first has already removed most of the skew risk - and if they later outgrow that, the store is a straightforward upgrade because the definitions already exist. Governance first, platform second. The reverse order buys infrastructure to manage a problem it cannot see.
Governance without a store ◐ 10 min read
Everything below is available this quarter, needs no procurement, and is what a feature store would have forced you to do anyway. If a store arrives later, this work is the input to it rather than a detour.
Self-studyFour things to put in place6 min read▶
A shared feature definition document. One page per feature: what it means in a sentence a business reader understands, the source it is computed from, the window it covers, and the edge cases. If two teams read it and implement the same thing, it is good enough. If they do not, you have found a skew source before it shipped.
An owner per feature. A named person, not a team mailbox. Ownership is what makes a definition change get announced instead of discovered, and session a3 showed what silent definition changes cost.
A deprecation process. Rule 22 again: "Clean up features you are no longer using." That needs a route from in use to marked unused to actually removed, agreed before the first feature needs it. Without one, nothing is ever retired and every model inherits the accumulated pile.
The write-time field. For every feature, record when its value becomes known - not when the row was loaded, but the moment the fact was true. This single field is what makes point-in-time correctness checkable by a person, and it turns leakage from a matter of individual carefulness into a property of the definition. It is the organisational version of the guard a feature store automates.
Self-studyWhat to ask before approving a feature-store project3 min read▶
- How many features would go in on day one, and who owns each? If the answer is vague, the definitions do not exist yet and the store has nothing to store.
- Which models will read from it, and are any of them real-time? Without a real-time consumer, half the architecture is unused.
- What are we keeping? The store is not the warehouse, the orchestrator, or the ETL. Everything already running keeps running, plus this.
- What are we still building ourselves? Lineage, drift detection, and the feature transformations themselves. Budget for them separately or they will not happen.
Session a5 turns to the other axis of the same argument: hand-crafted features versus learned ones, and what each costs to own.
Official sources covered
This leader track teaches the decision-relevant core of the canonical sources with no code and no notebooks. Certificates and graded exercises stay with the official providers. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · How does Google's Rules of Machine Learning define training-serving skew?
The definition is deliberately about performance, not about data, because that is the symptom a leader sees. Its three documented causes are a discrepancy between the training and serving pipelines, a change in the data between training and serving, and a feedback loop between the model and the algorithm.
2 · What does Rule 29 prescribe as the best way to make sure you train like you serve?
Rule 29 inverts the intuitive design: serving becomes the source of truth rather than a copy of the warehouse. It is an engineering practice, not a product, which is why it is a reasonable thing to ask for without any procurement attached.
3 · One team, one model, definitions living in a notebook. Does a feature store pay yet?
A store pays when many models share features, when real-time serving sits alongside historical training, or when compliance needs point-in-time proof. Buying one first leaves you running new infrastructure over the same two implementations of the same feature - and remember what it is not: not an ETL system, not an orchestrator, not a warehouse, not a database.