Start with what you observed 8 min live
Before a single word about embeddings, the counterweight, and it is Google's own: Rule 17 - "Start with directly observed and reported features as opposed to learned features." That rule is why this session is number nine and not number two. Everything in b2 to b8 is directly observed: a spend figure, a channel, a weekday, a count. Learned features are the ones a model invented for you, and they buy real capability at the price of everything you liked about a hand-crafted column.
So the honest question is not "should we use embeddings?" It is narrower and much easier to answer: is there a column here that hand-crafting genuinely cannot reach? On Lumen there is exactly one. support_note is free text a customer typed. There is no scaler for it, no sensible one-hot, no decomposition. It is either encoded or it is dropped, and for eight sessions we have been dropping it.
LiveWhy one-hot runs out on text5 min▶
Google's crash course gives two reasons, and both matter more for text than for anything you encoded in b3:
- Width. With M one-hot entries feeding N first-layer nodes, "the model has to train MxN weights for that layer." A vocabulary is not nine channels; it is tens of thousands of entries, and the weight count follows it up.
- Meaninglessness. One-hot encodings "lack meaningful relationships." Every category sits the same distance from every other, so "hot dogs and shawarmas" are exactly as unrelated as "hot dogs and salads" - which is the opposite of what you want a model to know about food, or about a customer note.
An embedding fixes both by being small and by placing similar things near each other. Google's glossary is precise about the size: an embedding vector is "A relatively low-dimensional vector of floating-point values (typically, between 5 and 500)." Low-dimensional relative to the vocabulary, not relative to your table - 384 columns is still 384 columns, and Part 3 prices that.
And then the line that makes this a feature-engineering session rather than an NLP session, from OpenAI's embeddings guide: "An embedding can be used as a general free-text feature encoder within a machine learning model." That is the whole idea. The embedding is not the model. It is a transformer that turns a column you could not use into columns you can, and everything downstream stays exactly as it was.
support_note becomes 384 columns 9 min live
The code is shorter than the discussion. Encode the text, stack the result beside the matrix you already had, fit the same estimator. Note where the split sits - first, as always, because an encoder call is still a transform and Part 4 is about what happens when you forget that.
LiveThe build-along, end to end6 min▶
from sentence_transformers import SentenceTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
import numpy as np
X_train, X_test, y_train, y_test = train_test_split( # split FIRST
df, y, test_size=0.4, stratify=y, random_state=42)
encoder = SentenceTransformer("all-MiniLM-L6-v2") # pin this string
E_train = encoder.encode(X_train["support_note"].fillna("").tolist())
E_test = encoder.encode(X_test["support_note"].fillna("").tolist())
print(E_train.shape) # (n, 384)
Z_train = np.hstack([prep.fit_transform(X_train), E_train]) # b2-b8 lane
Z_test = np.hstack([prep.transform(X_test), E_test]) # transform only
clf = LogisticRegression(max_iter=1000).fit(Z_train, y_train)Swap the first two lines for an API call and nothing else changes: you get back a list of float vectors, you np.hstack them, the estimator is untouched. That interchangeability is the practical form of OpenAI's "general free-text feature encoder" line - the encoder is a component, not a commitment.
LiveThree practical notes before you ship this5 min▶
- Dimensionality against rows. Rule 21: "The number of feature weights you can learn in a linear model is roughly proportional to the amount of data you have." 384 new columns against Lumen's 40,000 rows is comfortable. 3,072 columns against 4,000 rows is a different conversation, and the answer there is a smaller encoder or a shortened output, not more regularization bolted on afterwards.
- Cache the embeddings. Text does not change; a support note written in November is the same string every time you retrain. Encode once, store the vectors keyed by a hash of the text, and you pay the cost exactly one time instead of once per experiment. This is the single biggest difference between a cheap embedding pipeline and an expensive one.
- Pin the model version. A new encoder is a different feature space. The columns keep their names and their shape, the numbers mean something else entirely, and a model trained on the old vectors scored against the new ones will degrade quietly. Treat the encoder identifier as part of the feature definition - which is field 4 of the spec you write in b10.
Self-studyWhat the workbench cannot show you2 min read▶
Plainly: this page has no live workbench, because the in-browser workbench hosts Lumen's tabular features only and cannot run a text encoder. The ladder demo and the encode demo return in b10 for the capstone. Tonight the code above is the demonstration, and it runs on your laptop rather than in the slide.
What it costs, in dollars and in explanation 8 min live
Two bills arrive with an embedding feature. The first is money and latency, and it is easy to compare. The second is interpretability, and it is the one that shows up months later in a review meeting.
| Option | Dims | Cost | When it is the right call |
|---|---|---|---|
text-embedding-3-small | 1536 | 62,500 pages per dollar | The default hosted choice. Cheap enough that caching is a convenience rather than a necessity. |
text-embedding-3-large | 3072 | 9,615 pages per dollar | Roughly 6.5x the cost per page and twice the columns. Justify it with a measured lift, not a hunch. |
3-large + dimensions | 256 | same per page, fewer columns | The parameter exists to "trade-off performance and cost". A 3-large embedding "shortened to a size of 256" still outperforms an unshortened ada-002 at 1536. |
all-MiniLM-L6-v2 (local) | 384 | free, but you host it | No per-call bill, no data leaving your network, 10,000+ pretrained models on Hugging Face to choose from. You own the serving. |
LiveThe interpretability bill, stated plainly5 min▶
Here is what actually happened to your feature set. You traded one explainable column for hundreds of anonymous ones. Before: "support_note mentions a gift." After: "dimension 217 is 0.043." No stakeholder will ever ask about dimension 217, and no one can answer if they do.
Three places this bill comes due:
- The feature review. The leader track (a5, a6) teaches executives to ask "what does this feature mean and who owns it?" For 384 dimensions the honest answer is "the encoder means it," which is a defensible answer only if you say it out loud in advance.
- Regulated settings. Anywhere a decision has to be explained to the person it affects, an anonymous 384-column block is a governance problem before it is a modelling one. Hand-crafted features are not merely nostalgic there; they are the compliance artifact.
- Debugging. When performance drifts, a named column tells you where to look. A dimension does not. Permutation importance (b8) still works on the block as a whole, which is usually the granularity you can actually act on: keep the text feature, or drop it.
The leakage rails for embeddings 9 min live
Every rule from b5 still applies, and text gives them new hiding places. sklearn's definition has not changed: information "that would not be available at prediction time." A text column obeys it exactly like a numeric one - the difference is that text feels like context rather than data, so people forget to ask when it was written.
Rail 1 - the text must exist at prediction time. A support_note written after the order is target leakage, the b5 family-1 shape precisely: a column that only exists because the outcome already happened. Ask the same audit question you ask of every other column - "was this value knowable before the thing I am predicting?" - and be strict, because a well-written note is exactly the kind of feature that dominates importance and feels magically good.
Rail 2 - fit any dimensionality reduction on training data only. PCA or SVD over the full embedding matrix, all rows, before the split is family-2 preprocessing leakage. It is invisible in the output, it raises no error, and the test rows quietly voted on their own components. Same fix as always: inside the Pipeline, fit_transform on train, transform on test.
Rail 3 - a fine-tuned encoder can be a leak. If the encoder was fine-tuned on your labels, the vectors carry target information into every row that produced them, including the rows you later call held-out. This is the target-encoding failure from b5 wearing a much larger coat: naive encoding of a random postal code scored train 0.931 and held-out 0.493 for exactly this reason. A pretrained, frozen, off-the-shelf encoder has never seen your target and is safe by construction.
And the splits still matter. Text does not exempt you from family 3 or family 4. If notes accumulate over time, split by time. If the same customer writes several, split by customer - otherwise the model memorizes a writing style and you will not find out until a genuinely new customer arrives.
The resolution summary. A support team embedded a "resolution summary" field into a churn model and got a spectacular offline score - the best the team had ever seen, across every fold. The summary was written when the case closed. It described the outcome in fluent prose, and the encoder faithfully turned that prose into 768 excellent predictors of a thing that had already happened. No cross-validation can catch this: the column is contaminated in every fold, so every fold agrees. The only thing that catches it is field 3 of a feature spec, which is what b10 makes you write.
Try it yourself - this week ◐ 30-40 min total
- Run the build-along on a text column you actually have. Report two numbers: held-out score without the embedding block, and with it. If the gap is not worth the hosting, you have learned something valuable for free.
- Before that, do the Rule 17 pass: write down three hand-crafted features you could derive from the same text in ten minutes. Score those first. Embeddings that cannot beat a keyword flag are not earning their bill.
- Write the write-time answer for your text column in one sentence: who types it, into which system, and at what moment relative to the outcome. If you cannot answer, that is this week's finding.
- Price it honestly: rows to encode, encoder choice, whether caching applies, and what a re-encode costs when you change models. Bring the number to b10, because it belongs in the spec.
Official sources covered
Certificates and graded exercises stay with the official providers (linked each session). This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Why does this course teach embeddings in session 9 rather than session 2?
Google's Rule 17 is the counterweight to the whole session. Directly observed features are cheaper, explainable, and debuggable. Reach for a learned feature when a column genuinely cannot be hand-crafted - on Lumen that is exactly one column, support_note.
2 · What do embeddings buy you that one-hot encoding of text cannot?
Google's two objections to one-hot on text: with M entries and N first-layer nodes "the model has to train MxN weights for that layer", and every category sits equidistant from every other, so "hot dogs and shawarmas" are no closer than "hot dogs and salads". An embedding is low-dimensional - "typically, between 5 and 500" - and similar text lands nearby.
3 · Your encoder was fine-tuned on your own labels before being used to embed the training text. What is the problem?
A correct split does not save you here, exactly as it did not in b5's Demo 2: naive target encoding of a random postal code scored train 0.931 and held-out 0.493. A pretrained, frozen encoder has never seen your target and is safe by construction.