learn-economics-with-phoebe / Deep-dive session 2 of 10
Learn Economics with Phoebe · Deep-dive session 2 of 10

Why the naive number lies

Deep-dive session 1 left you with 2,216 product-month rows for 203 products and a reason not to trust any one product's slope. Tonight fits those rows as one line, and the first three ways of doing it give +0.008, -0.108 and -1.788. Nothing about the rows changed between those numbers. What changed is the question each fit asks: does a cheaper product sell more than a dearer one, or does a product sell more in the months it is cheaper? The first is a fact about the catalogue and the second is an elasticity, and the pooled line quietly answers the first while looking like the second. Then a month effect, three robustness checks, and a placebo that shuffles the prices inside each product and watches the slope fall to zero.

🟡 Core Analysts, pricing and growth teams Python, pandas, numpy 45 min
0-3 · Welcome 3-18 · Three rungs, one fixed effect, one placebo 18-40 · Build-along: rungs 1 to 5 on the panel 40-45 · Q&A
Part 0

Where we start

A line through logs has a slope, and last time the slope was fitted one product at a time and came out as noise. The fix is not more data per product; it is a different unit. Fit all 2,216 rows at once and the slope steadies. But "all at once" hides a choice: whether each product is allowed its own level. Without that allowance, the line is comparing a bath towel that sells 40 a month at R$ 30 with a watch that sells 4 a month at R$ 300, and calling the difference an elasticity. With it, the line only sees how each product's own months differ from each other. Every standard error tonight is a naive standard error: conventional OLS, unclustered, and the page says so each time one appears.

🎫 Clear all three quiz questions to stamp your passport Live = presented in the 45 minutes Self-study = the full depth, after class ★ Build-along = run the code yourself

★ What you walk out with tonight. Five fits of the same panel with the number on each; the one sentence that explains why +0.008 and -1.788 are both correct answers to different questions; a demeaning step you can write in one line of pandas and defend in one line of English; a placebo loop that tells you how far from zero "far" is; and the honest label for the result: a within-product association with a placebo behind it, not a causal elasticity.

Part 1 · covers the cross-section across products

Rung 1: cheap products are not popular products

5 min live

The first thing most people fit is the obvious one: one dot per product, its mean price against its total units, a line through all of them. Across 4,731 Olist products with at least five units sold, that line has slope +0.008 with a naive standard error of 0.011. Zero, in other words. Inside each of the eight biggest categories the slope sits between -0.035 and +0.089; none is below -0.04. Price tells you nothing about how much a product sells when the products are different products.

SAME 2,216 ROWS, THREE LINES each cluster is one product's months; five sketched of 203 LOG UNITS LOG PRICE rung 1 · across products · +0.008 (naive se 0.011) rung 2 · pooled, no controls · -0.108 (naive se 0.028) rung 3 · within product · -1.788 (naive se 0.149) a dear product that sells a lot sits up here a cheap one that sells little sits down there one slope, 203 intercepts The flat lines ask whether cheap products outsell dear ones across the catalogue. The amber lines ask whether a product sells more in its cheaper months. Only the second is an elasticity.
🔍 Click to zoom - the two navy lines are almost level because they compare products with other products; every amber line is steep because it compares a product with itself
LiveWhat a cross-section across products is actually measuring3 min▶

One dot per product. Mean price on the x axis, total units on the y axis, 4,731 dots, a line. The slope of +0.008 with a naive standard error of 0.011 says the line is flat. The reading people reach for is "demand on this marketplace is perfectly inelastic", and that reading is wrong, because the line never saw a price change. It saw 4,731 different products at 4,731 different prices, and the reason a R$ 300 watch sells fewer units than a R$ 30 towel is that it is a watch, not that it is dear.

Inside a category it does not improve. Bed, bath and table +0.022; sports and leisure -0.013; health and beauty -0.026; furniture and decor -0.028; housewares +0.089; computers and accessories +0.066; telephony -0.009; watches and gifts -0.035. Eight categories, none below -0.04. Two towels at different prices are still two different towels: one has better reviews, a bigger seller, a photo that loads. Their price difference is tangled with everything else that differs between them, and the slope cannot pull the price part out.

The bench's own version. On the 203 eligible products alone, the same cross-section gives -0.001 with a naive standard error of 0.058. Same answer on the rows the browser can recompute in deep-dive session 10, which is why the build-along tonight prints that one and quotes the 4,731-product figure from the offline run.

Self-studyWhy a flat cross-section is not evidence of inelastic demand2 min▶

An elasticity is the answer to "if this product's price moved, how would its units move". A cross-section holds 4,731 answers to a different question, "which products are cheap and which are popular", and those two properties are set by different things. Popularity comes from category, brand, reviews, the seller's reach; cheapness comes from cost, positioning, the seller's margin. There is no reason for them to line up, and on Olist they do not.

The mistake has a name in econometrics, omitted variable bias, and its cure is not a bigger cross-section. Ten times the products would give ten times the dots and the same flat line with a smaller standard error, which would make the wrong answer look more certain. The cure is a within comparison, which is Part 2.

Real world

The catalogue slide. A pricing review opens with a scatter of every SKU's price against its volume and a fitted line that is nearly flat. The slide concludes that customers are not price sensitive and recommends a 10 percent rise across the board. Every dot on that slide is a different product; the slide has measured the shape of the catalogue and presented it as the shape of demand. On these rows the within-product number says the rise loses about 7 percent of revenue, not gains 9, and Part 3 shows the two predictions side by side.

Part 2 · covers pooled versus within, and what a fixed effect removes

Rungs 2 and 3: the same rows, compared within product

5 min live

Now use the panel. Rung 2 fits one line through all 2,216 product-months with no allowance for which product a row belongs to, and gets -0.108 with a naive standard error of 0.028. Rung 3 fits the same 2,216 rows after subtracting each product's own mean of log price and mean of log units, and gets -1.788 with a naive standard error of 0.149. The second fit is the first fit with 203 intercepts, one per product, and those intercepts are a product fixed effect. What they remove is everything about a product that does not change from month to month.

BEFORE: EACH PRODUCT AT ITS OWN LEVEL AFTER: EACH PRODUCT'S MEAN SUBTRACTED LOG UNITS LOG PRICE product A product B product C + its mean three of 203, sketched; the levels differ, the tilt does not subtract the mean LOG UNITS, DEMEANED LOG PRICE, DEMEANED 0 -1.788 naive se 0.149 2,216 rows, each measured from its own product's mean; one line, one slope The subtraction keeps only how each month differs from that product's usual month. A product that is always cheap and always slow contributes nothing to the slope; one that was cheaper in March and sold more in March contributes everything.
🔍 Click to zoom - the amber crosses are what the fixed effect removes; after they are subtracted every product sits on the same origin and the only thing left to fit is the tilt
LivePooled is not within, and demeaning is the whole trick3 min▶

Rung 2, pooled. One line through 2,216 rows, no intercepts. Its slope of -0.108 looks like a small elasticity and is not one, because the line is still mostly comparing products with other products: 203 products at 203 levels contribute most of the spread in both price and units, and the month-to-month movement inside a product is a small part of the picture. The pooled number is a blend of the flat across-product story and the steep within-product story, weighted toward the flat one.

Rung 3, within. For each product, subtract its mean log price from every month's log price and its mean log units from every month's log units. Then fit one line through the 2,216 demeaned rows. This is exactly the regression with 203 product dummies, and the demeaned version is the one you can draw: every product now sits at the origin and the only variation left is within it. The slope is -1.788 with a naive standard error of 0.149.

What the fixed effect removes. Everything about a product that is constant across its months: its category, its seller, its photo, its reviews as of the panel, its usual price level and its usual volume. What it cannot remove is anything that changes month to month inside the product, which is where Part 3 starts.

Self-studyWhy 203 dummies and one demeaning step give the same slope2 min▶

Regressing log units on log price plus 203 product indicator columns, and regressing demeaned log units on demeaned log price with no indicators, give the identical slope. This is the Frisch-Waugh-Lovell result: partialling a set of regressors out of both sides first and then fitting the rest is the same as fitting everything at once. Demeaning within product is the partialling-out step for product dummies, and it is why the build-along never constructs a 2,216 by 204 design matrix.

One thing the shortcut does change is the naive standard error's degrees of freedom: the demeaned regression thinks it estimated one parameter when it estimated 204. The correction is small at 2,216 rows, and the honest move anyway is to cluster by product, which conventional OLS does not do. That is the reason every standard error on this page is called naive.

Real world

The subscription tier that "proved" price does not matter. A SaaS analyst pools three years of accounts across a starter, a team and an enterprise tier and finds that higher prices go with more seats. Of course they do; enterprise customers buy the enterprise tier. Compare each account with itself across its own renewals, where the price actually moved, and the sign flips. The pooled fit was a cross-section across customers wearing a panel's clothes.

Part 3 · covers month effects, robustness, and the placebo

Rungs 4 and 5: the month, the checks, and the shuffle

5 min live

A product fixed effect does nothing about November. If the marketplace grew, or every seller discounted in the same month, then price and units moved together across all products at once, and the within-product slope carries a share of that. Rung 4 removes each month's mean as well as each product's and gets -1.854 with a naive standard error of 0.150: month effects move it a little, not a lot. Three robustness rows move it inside a band. And a placebo tells you what the fit gives when there is nothing to find: shuffle log price within product, refit 200 times, and the slope averages -0.002.

200 SHUFFLES, 200 SLOPES, AND THE REAL ONE log price shuffled within each product, then the rung 3 fit repeated; bars sketched, shape as computed -1.788 -0.5 0 +0.5 -1.788 the real slope more than 11 placebo sd from zero widest of 200 draws: 0.444 in absolute value mean -0.002 sd 0.159 seed 20180901 If the within-product slope were an artefact of the fitting, shuffling prices inside each product would leave it standing. It does not; the slope collapses to zero every time. -1.788 is not the method.
🔍 Click to zoom - the bell is what the fit produces when price carries no information; the amber marker sits eleven bell-widths to its left
LiveMonth effects, three checks, and the placebo3 min▶

Rung 4, product and month effects. Subtract each product's mean, then each month's mean, and repeat until nothing moves (alternating projections, fifty passes is plenty). The slope is -1.854 with a naive standard error of 0.150, against -1.788 without the month effects. The marketplace grew across 2017 and 2018 and prices drifted, and removing that shared movement changes the answer by 0.07. A small move is itself a finding: the within-product slope is not being driven by the calendar.

Three ways of being wrong that do not change the sign. Weight each row by its units, so big-selling months count more: -2.216 (naive standard error 0.143). Use the freight-inclusive price the buyer paid instead of the seller's: -1.881 (naive standard error 0.154). Drop the months where a product sold a single unit, which leaves 1,850 rows: -1.480 (naive standard error 0.142); the single-unit months pull the slope more negative. All three are rung 3 on the same panel with one decision flipped, and the answer stays between -1.5 and -2.2.

Rung 5, the placebo. Inside each product, shuffle the log prices across its months so that every product keeps its own prices but in a random order, then run the rung 3 fit. Do it 200 times. The slopes have mean -0.002, standard deviation 0.159, and the widest is 0.444 in absolute value. The real -1.788 is more than eleven of those standard deviations from zero. That is what "the slope is not the method" looks like in a number.

Self-studyWhat none of it removes: the seller chose the price after seeing demand3 min▶

Product fixed effects remove "this product is cheap and popular". Month effects remove "the marketplace grew". The placebo shows the slope is not a fitting artefact. None of the three touches the thing that makes an observational price panel hard: a seller cuts price when stock is high, or when a competitor moves, or when units have been slow; and a seller raises price in the last months of a product's life, which is also when it is going out of stock. The price was chosen after looking at demand, so price and units are tied together by the seller's decision as well as by the buyers' response, and the panel cannot tell the two apart.

The honest label for -1.788 is therefore a within-product association with a placebo behind it, not a causal elasticity. It is the number to publish, with its range, and it is a much better number than +0.008 or -0.108. It is not the number an experiment would give. The steps from association to cause, confounding, what a control can and cannot do, and how to read a diagram of the mechanism, are the subject of learn-causal-inference-with-phoebe. The experiment that would settle the Olist question, a price test where the seller's choice is replaced by a random draw, is designed in learn-experimentation-with-phoebe. Deep-dive session 3 uses the number as it is, with the label on.

Real world

The clearance that looked like elasticity. In the headline product from deep-dive session 1, the last three months carry prices of R$ 89.90 and R$ 103 and units of 2, 2 and 3. A within-product fit sees a rise and a collapse and reads it as a response. The seller most likely raised the price because the product was ending, and units collapsed for the same reason. Fixed effects cannot see that, month effects cannot see that, and the placebo cannot see that; only knowing why the price moved can. The number survives all three checks and still carries the label.

Build-along

Rungs 1 to 5 on the saved panel

★ 22 min · everyone builds

Start from panel_2018-09-01.csv, the file deep-dive session 1 wrote, with columns product_id, month, units, price, freight, cat, lp, lq. Seven steps in pandas and numpy, each printing the map's number to three decimals. Every standard error is the conventional OLS one, unclustered, so the code calls it naive_se and the comments do too.

FIVE RUNGS, ONE PANEL, THE NUMBER ON EACH rung 1 · across products rung 2 · pooled, no controls rung 3 · product effects rung 4 · product + month rung 5 · placebo, 200 draws +0.008 -0.108 -1.788 -1.854 -0.002 naive se 0.011 · on the 203: -0.001 naive se 0.028 · 2,216 rows naive se 0.150 sd 0.159 · max abs 0.444 rung 3, one decision flipped weighted by units-2.216 price incl. freight-1.881 months with 2+ units-1.480 naive se 0.143 · 0.154 · 0.142 Climb it in order. Each rung is one line of pandas more than the last, and the number changes by a factor of sixteen between rung 2 and rung 3.
🔍 Click to zoom - the amber rung is the one to publish, with the three flipped-decision numbers beside it as its range

Load the panel and write the slope function once. The slope of y on x, optionally weighted, with the conventional standard error. It is the same function for every rung, which is the point: the rungs differ in what goes in, never in the fit.

★ Step 1 · load and slopeimport numpy as np import pandas as pd panel = pd.read_csv("panel_2018-09-01.csv") # from deep-dive session 1 print(len(panel), panel["product_id"].nunique()) # 2216 203 def slope(x, y, w=None): """OLS slope of y on x and its naive (conventional, unclustered) standard error.""" x = np.asarray(x, float); y = np.asarray(y, float) w = np.ones_like(x) if w is None else np.asarray(w, float) xm = np.average(x, weights=w); ym = np.average(y, weights=w) sxx = np.sum(w * (x - xm) ** 2); sxy = np.sum(w * (x - xm) * (y - ym)) b = sxy / sxx resid = y - ym - b * (x - xm) naive_se = np.sqrt(np.sum(w * resid ** 2) / (len(x) - 2) / sxx) return b, naive_se

Rung 1: the cross-section on the 203. One dot per product, total units against mean price. The 4,731-product version (+0.008, naive standard error 0.011) needs the full item table and is quoted from the offline run; on the 203 the answer is the same: nothing.

★ Step 2 · rung 1, across productscs = panel.groupby("product_id").agg(units=("units", "sum"), price=("price", "mean")) b, se = slope(np.log(cs["price"]), np.log(cs["units"])) print(f"rung 1 across the 203 products: {b:+.3f} (naive se {se:.3f})") # rung 1 across the 203 products: -0.001 (naive se 0.058) # across all 4,731 products with 5+ units (offline, needs the item table): +0.008 (naive se 0.011)

Rung 2: pooled. All 2,216 rows, one line, no intercepts per product. The number looks like a small elasticity. It is a blend.

★ Step 3 · rung 2, pooledb, se = slope(panel["lp"], panel["lq"]) print(f"rung 2 pooled, no controls: {b:+.3f} (naive se {se:.3f})") # rung 2 pooled, no controls: -0.108 (naive se 0.028)

Rung 3: demean by product. One helper, one line per column, then the same fit. This is the regression with 203 product dummies, without building them.

★ Step 4 · rung 3, product fixed effectsdef demean(df, col, by): return df[col] - df.groupby(by)[col].transform("mean") lp_w = demean(panel, "lp", "product_id") lq_w = demean(panel, "lq", "product_id") b, se = slope(lp_w, lq_w) print(f"rung 3 product fixed effects: {b:+.3f} (naive se {se:.3f})") # rung 3 product fixed effects: -1.788 (naive se 0.149)

Rung 4: product and month effects by alternating projections. Subtract product means, subtract month means, repeat. Fifty passes is far more than the panel needs and costs nothing.

★ Step 5 · rung 4, two-way effectsdef twoway_demean(df, col, a, b, iters=50): v = df[col].astype(float).copy() for _ in range(iters): v = v - v.groupby(df[a]).transform("mean") v = v - v.groupby(df[b]).transform("mean") return v panel["m"] = panel["month"].astype(str) b, se = slope(twoway_demean(panel, "lp", "product_id", "m"), twoway_demean(panel, "lq", "product_id", "m")) print(f"rung 4 product + month effects: {b:+.3f} (naive se {se:.3f})") # rung 4 product + month effects: -1.854 (naive se 0.150)

Three decisions flipped, one at a time. Each is rung 3 with a single change. Print all three before quoting any one of them; together they are the range.

★ Step 6 · the robustness rowsb, se = slope(lp_w, lq_w, w=panel["units"]) print(f" weighted by units: {b:+.3f} (naive se {se:.3f})") # weighted by units: -2.216 (naive se 0.143) p2 = panel.assign(lpf=np.log(panel["price"] + panel["freight"])) b, se = slope(demean(p2, "lpf", "product_id"), demean(p2, "lq", "product_id")) print(f" price incl. freight: {b:+.3f} (naive se {se:.3f})") # price incl. freight: -1.881 (naive se 0.154) p3 = panel[panel["units"] >= 2] b, se = slope(demean(p3, "lp", "product_id"), demean(p3, "lq", "product_id")) print(f" months with 2+ units: {b:+.3f} (naive se {se:.3f}), {len(p3)} rows") # months with 2+ units: -1.480 (naive se 0.142), 1850 rows

Rung 5: the placebo. Shuffle log price inside each product, demean, refit, 200 times. The mean and sd are "about" because a different generator order gives slightly different draws; the offline run with this seed gave -0.002 and 0.159, and the browser's own generator in deep-dive session 10 gave -0.003 and 0.170.

★ Step 7 · rung 5, shuffle within productrng = np.random.default_rng(20180901) bs = [] for _ in range(200): sh = panel.groupby("product_id")["lp"].transform( lambda s: pd.Series(rng.permutation(s.values), index=s.index)) lp_sh = sh - sh.groupby(panel["product_id"]).transform("mean") bs.append(slope(lp_sh, lq_w)[0]) bs = np.array(bs) print(f"rung 5 placebo: mean {bs.mean():+.3f}, sd {bs.std():.3f}, max abs {np.abs(bs).max():.3f}") # rung 5 placebo: mean about -0.002, sd about 0.159, max abs about 0.444 print(f"real slope is {abs(-1.788 / bs.std()):.1f} placebo sd from zero") # about 11

Print the ladder as one block, every time. Five rungs and three flipped decisions, eight lines, before a single number leaves the notebook. A reader who sees only -1.788 has to trust you; a reader who sees +0.008 above it and -0.002 below it can see for themselves what the number is and what it is not.

Homework

Before deep-dive session 3

30 min
  1. Run the seven steps on the saved panel and check every printed number against the comment beside it. Any three-decimal difference means the panel is not the panel; find which of deep-dive session 1's three decisions moved.
  2. Flip a fourth decision. Restrict rung 3 to one category (bed, bath and table has 22 products and 241 rows) and print the slope with its naive standard error. Write one sentence on whether it is distinguishable from the all-category number.
  3. Run the placebo with a different seed and a different draw count (500). Write down the mean and sd, and how many placebo standard deviations -1.788 is from zero this time.
  4. Write the label. One sentence that says what -1.788 is and what it is not, including the words "within product", "placebo" and "not causal". That sentence goes above every chart that uses the number.
Check yourself

Three questions

Clear all three and deep-dive session 2 gets stamped in your passport.

1 · Across 4,731 products, a line through mean price and total units has slope +0.008 with a naive standard error of 0.011. What does that say about demand?

An elasticity answers "if this product's price moved, how would its units move". A cross-section holds thousands of answers to "which products are cheap and which are popular", and those are set by different things. Inside each of the eight biggest categories the slope stays between -0.035 and +0.089.

2 · Rung 2 (pooled) gives -0.108 and rung 3 (product fixed effects) gives -1.788 on the same 2,216 rows. What does the fixed effect do?

Demeaning within product is the same regression as one with 203 product dummies. Option D describes the month effect of rung 4, which moves the answer from -1.788 to -1.854; the product effect is what moves it from -0.108 to -1.788.

3 · The placebo shuffles log price within each product 200 times and the refitted slope averages -0.002 with sd 0.159. The real slope is -1.788. What is the honest label for -1.788?

-1.788 is more than eleven placebo standard deviations from zero, so it is not noise and not the method. It is also not an experiment: a seller cuts price when stock is high and raises it when a product is ending. The number to publish is the range -1.5 to -2.2 with the label on.

Sources covered

What this session teaches, and where it came from

✓Rung 1: cross-section across 4,731 products +0.008 (naive se 0.011); by category, none of eight below -0.04; on the 203 eligible products -0.001 (naive se 0.058)Computed from the Brazilian E-Commerce Public Dataset by Olist (Kaggle, CC BY-NC-SA 4.0) for these pages
✓Rung 2 pooled -0.108 (naive se 0.028); rung 3 product fixed effects -1.788 (naive se 0.149); rung 4 product and month effects -1.854 (naive se 0.150), all on 2,216 rowsComputed; standard errors are conventional OLS, unclustered, and called naive throughout
✓Robustness: weighted by units -2.216 (naive se 0.143); freight-inclusive price -1.881 (naive se 0.154); months with 2+ units -1.480 (naive se 0.142) on 1,850 rows; the two-scope statement -1.5 to -2.2, about -1.8Computed
✓Placebo: log price shuffled within product, 200 draws, seed 20180901: mean -0.002, sd 0.159, max abs 0.444; the real slope more than 11 placebo sd from zeroComputed; a different generator order gives slightly different draws, so the build-along says "about"
✓Fixed effects as demeaning; Frisch-Waugh-Lovell; omitted variable bias in a cross-sectionStandard econometrics, no single source
◐Confounding, the steps from association to cause, and the price experimentNamed and linked to the causal inference and experimentation courses; not taught here

Every fact on this page, and its verification tier, is recorded in the course's source map.

Deep-dive session 2 cheat sheet · pin this

Across products = the catalogue+0.008 on 4,731 products. Cheap products are not popular products. Not an elasticity.
Pooled = a blend-0.108 on 2,216 rows. Mostly still comparing products with products.
Within product = the elasticity-1.788, naive se 0.149. Subtract each product's mean, fit one line. 203 intercepts.
Month effects move it a little-1.854. Weighted -2.216, freight -1.881, 2+ units -1.480. Publish -1.5 to -2.2.
Placebo = what nothing looks likeShuffle price within product, 200 times: mean -0.002, sd 0.159. Real slope 11 sd away.
The labelWithin-product association with a placebo behind it. The seller chose the price. Not causal.