Where this sits
Sessions a1 to a4 were about features you can name: a column someone wrote down, a definition someone owns, a value someone can point at in a warehouse. This session is about the other kind - features a model produces for you, out of text, in numbers nobody chose. That is a genuinely useful capability and a genuinely different governance problem, and the mistake leaders make is treating it as an upgrade instead of a trade.
What an embedding actually buys 15 min live
Google's glossary keeps the definition boringly concrete: an embedding vector is "A relatively low-dimensional vector of floating-point values (typically, between 5 and 500)". That is the whole object. Not intelligence, not understanding - a list of numbers standing in for something that used to be a word, a sentence, or a category with far too many levels. The question a leader should ask is not "is it advanced". It is "what problem does it solve that a column could not".
LiveThe one-hot problem, in Google's own words5 min▶
When a category has thousands of levels, the standard fix - one column per level - has two specific costs, and Google's crash course states both plainly.
- Width costs weights. With M one-hot entries and N first-layer nodes, "the model has to train MxN weights for that layer". That is not a rounding error at scale; it is the training bill.
- Sameness. One-hot encodings "lack meaningful relationships". Every category sits the same distance from every other, so nothing carries across. Google's example is the one to repeat in a meeting: "hot dogs and shawarmas" should sit closer than "hot dogs and salads", and a one-hot column has no way to say so.
An embedding fixes both at once. It is narrower than the one-hot block, and things that mean similar things land near each other in the space. That is the entire buy. Everything else people say about embeddings in a planning meeting is decoration on those two sentences.
LiveThe bridge sentence to remember4 min▶
The clearest one-line framing comes from OpenAI's own guide, and it is worth quoting verbatim because it strips out the mystique:
Read what that sentence does not claim. It does not say the embedding is the model. It does not say it replaces your tabular features. It says encoder - a way of getting a text column into the same matrix as your spend and tenure columns, so an ordinary model can use it. Your existing pipeline stays. One column changes shape.
This matters for how you brief a board. "We are adding embeddings" sounds like a re-platform. "We are turning our support notes into a feature the existing model can read" is what is actually happening, and it is a much easier thing to scope, budget and cancel.
Self-studyWhere the numbers come from3 min read▶
Nobody hand-derives an embedding. You call a model that was trained on a very large amount of text, and it hands back the vector. Two consequences follow, and both are governance facts rather than technical ones:
- The numbers are not yours. They come from a model someone else trained on data you did not choose. If that model version changes, your feature changes, silently, for every row scored after the switch.
- The numbers have no names. Dimension 217 does not mean anything. There is no column definition to write, no owner to assign in the usual sense, and no sentence to put in front of a regulator that describes what it measures.
Neither is a reason to avoid embeddings. Both are reasons to write down which model and which version produced your features, and to treat a model upgrade as a change to your feature definitions - because that is exactly what it is.
When to stop hand-crafting 12 min live
This is the decision your team will bring you, usually framed as a technology choice. It is not. It is a question about what data you have and who has to explain the result. Before the table, the counterweight - because in a room full of enthusiasm it is the part that gets skipped.
| Situation | Hand-craft or embed | Why |
|---|---|---|
| A handful of clean categories | Hand-craft | Nine channels or four product categories fit in a one-hot block without strain. An embedding here adds cost and removes names for no gain. |
| Free text (support notes, product descriptions, reviews) | Embed | There is no hand-crafted column that captures a sentence. This is the case embeddings exist for, and the one where the trade is usually worth it. |
| High-cardinality IDs with business meaning | Hand-craft or target-encode | Postal code, merchant ID, SKU. These have structure your business already understands, and the encoding lesson in the practitioner track handles them with math you can audit. |
| You need to explain each feature to a regulator | Hand-craft | "Dimension 217 of a text embedding" is not an answer. If the model sits anywhere near credit, hiring, insurance or health, the explainability cost is the deciding cost. |
| You have text and no time | Embed | Turning a text column into a usable feature by hand takes real linguistic work. An off-the-shelf embedding is one call. Speed is a legitimate reason, as long as you have said out loud what you traded for it. |
LiveThe question that settles most of these4 min▶
When the table does not settle it, one question usually does: does anybody have to justify a single decision this model makes?
- Yes. A declined application, a flagged transaction, a rejected claim, a triage priority. Hand-crafted features win by default, because the explanation has to survive a person asking "why me".
- No. A ranking, a recommendation, an internal prioritisation queue, a routing decision. The explainability cost is much lower, and the embedding trade gets easier.
Notice this is not a technical question and your data scientists cannot answer it for you. It is a question about your product, your regulator and your appetite. That is precisely why it lands on your desk.
Self-studyRule #17 is not anti-embedding2 min read▶
Rule #17 is often quoted as if it says learned features are bad. It says something narrower and more useful: start elsewhere. The reasoning is about risk, not quality. Directly observed features are available on day one, they keep working when an external model changes, and when something breaks you can point at the column that broke.
The honest reading for a leader: the sequencing is the rule, not the verdict. Try the observed features. Measure. Then, if text is where the remaining signal lives, spend the money on learning from it - with your eyes open about what you gave up.
The cost side leaders must own 12 min live
Your team will scope the engineering. Nobody will scope the storage, the latency, or the explainability, because those land on you. Here are the real numbers so you can do it in the meeting rather than after it.
| Option | Dimensions | Cost signal |
|---|---|---|
text-embedding-3-small | 1536 | 62,500 pages per dollar |
text-embedding-3-large | 3072 | 9,615 pages per dollar |
Sentence Transformers all-MiniLM-L6-v2 | 384 | Local and free, 10,000+ pretrained models available |
Dimensions are a dial, not a fixed cost. OpenAI's dimensions parameter lets you "trade-off performance and cost". You are not stuck with 3072 numbers per row because you chose the larger model.
And the dial is generous. A 3-large embedding "shortened to a size of 256" still outperforms an unshortened ada-002 at 1536. Twelve times narrower than its full width, and still ahead of the previous generation at full width. Shortening is not the compromise it sounds like.
The free option is real. Sentence Transformers runs locally at 384 dimensions with 10,000+ pretrained models to choose from. For a first test on your own support notes, "what would this cost" has an answer of zero, and there is no good reason to approve budget before that test has run.
Then multiply by your row count. One text column at 1536 dimensions across ten million rows is a different storage and serving conversation than the same column at 256 across one hundred thousand. Ask for that arithmetic before, not after.
Embeddings turn one text column into hundreds of numbers. That is storage you now pay for on every row, latency you now pay for on every prediction, and a feature nobody can explain in a review. It buys you signal that genuinely was not reachable any other way. That is a real trade, not a free upgrade - and the job in the approval meeting is to make sure both halves of it get said out loud.
LiveThree costs that never make it onto the slide4 min▶
- Serving latency. If the embedding is computed at prediction time, every request now waits on a model call. If it is precomputed, you have a freshness question instead: how stale can a customer's support-note feature be before it is wrong.
- Version drift. The embedding model gets upgraded. Your features change shape or meaning. Unless someone is watching, this is a silent change to the inputs of a production model, which is the training-serving skew problem from a4 arriving through a new door.
- Review cost. Every future audit, incident review and model risk assessment now has a section that ends in "we cannot say what that dimension measures". That cost is small once and compounding thereafter.
Self-studyWhat the practitioner track does with this2 min read▶
Session b9 of the practitioner track builds the join you saw in the diagram: a text column goes through an embedding model, the resulting vector is attached to the tabular feature matrix, and the same ordinary model trains on the combined table. Your team will see the mechanics, the dimension choice, and where it fails.
What stays on your side of the line: whether the text is worth embedding at all, whether the explainability cost is affordable in your regulatory context, and whether the observed features were genuinely exhausted first. Those are not implementation details, and no notebook answers them.
Before you approve an embeddings project ◐ 10 min read
Four questions. If the answers are not ready, the project is not ready, and asking them costs you five minutes against a quarter of somebody's roadmap.
1 · What text do we actually have? Not "we could collect". Have. How many rows carry it, how long is it, how much of it is empty or boilerplate. A support-note field that is blank on 70% of rows is not a text asset yet, and the embedding will faithfully encode the emptiness.
2 · Does anyone need to explain the model? Regulator, customer, internal risk committee, journalist. If yes, name who, and get their view before the work starts rather than at the review. This is the question from Part 2, and it is the one most likely to kill the project late if asked late.
3 · What is the per-query cost at our volume? Take the published rate, take the dimension choice, multiply by your actual rows and your actual refresh cadence. Then ask what the local 384-dimension option would cost instead. If nobody has done this arithmetic, that is the first deliverable.
4 · Have the cheap hand-crafted features been tried first? Rule #17, as a gate rather than a slogan. What is the held-out number with observed features only? If that number does not exist, there is nothing for the embedding to beat, and you will never know whether it earned its cost.
Self-studyHow to say no without saying never3 min read▶
The useful response to an embeddings pitch is rarely a rejection. It is a sequence:
- "Show me the observed-feature baseline first." Cheap, fast, and it sets the bar the embedding has to clear.
- "Run it locally at 384 dimensions before we buy anything." Free, and it answers whether the text carries signal at all.
- "Come back with the lift and the per-row cost." Two numbers. A decision either becomes obvious or the project was never worth arguing about.
Every step is a real experiment, so nobody feels blocked, and by the end you are approving a known trade rather than a hope.
Official sources covered
The leader track teaches the decision-relevant core of the canonical sources, with every number and quote taken from the official documentation rather than paraphrased. Certificates and graded exercises stay with the official providers. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Rules of ML #17 says "Start with directly observed and reported features as opposed to learned features." What does that mean for a leader approving work?
The rule is about sequencing, not about quality. Observed features are available immediately, keep working when an external model changes, and can be pointed at when something breaks. Reversing the order is how a quarter gets spent on a project a weekend of column work would have beaten.
2 · What does an embedding buy you that a one-hot column cannot?
Both halves come straight from Google's crash course. One-hot encodings "lack meaningful relationships", which is why "hot dogs and shawarmas" cannot sit closer than "hot dogs and salads" in a one-hot block. Narrower, plus relationships. That is the entire buy.
3 · Your team says the large embedding model is too expensive at 3072 dimensions. What is the accurate response?
Shortening is not the compromise it sounds like: twelve times narrower and still ahead of the previous generation at full width. And before any of that, the local 384-dimension option costs nothing, which makes it the right first test rather than a fallback.