Multivariate testing, and the interactions a stack of A/B tests can't see

The clean A/B discipline of b4 - change exactly one thing so the effect is unambiguous - has a blind spot. Real product changes travel in bundles: a new hero image and new CTA copy and a new layout ship together, and their effects are not always additive. When the effect of one factor depends on the level of another, that is an interaction, and a series of isolated A/B tests is structurally incapable of detecting it. Multivariate testing (MVT) tests multiple factors simultaneously in a factorial design, so a single experiment estimates each factor's main effect and the interactions between them. The catch: the number of cells grows with every factor, and your traffic has to fill them all.

A full factorial tests every combination of every factor. With k two-level factors that is 2^k cells: 2 factors = 4 cells, 3 factors = 8, 5 factors = 32. Its virtue is completeness - it can estimate every main effect and every interaction, up to the k-way. Its cost is traffic: your users split across all those cells, and each cell needs enough volume to be powered. A fractional factorial runs a carefully chosen subset of the cells - halving or quartering the design - to save traffic. The price is aliasing (also called confounding): some effects become mathematically indistinguishable from one another, so you trade the ability to estimate certain (usually higher-order) interactions for a smaller, cheaper design.

The fastest way to see an interaction is a picture. Plot the outcome (conversion) on the y-axis, one factor across the x-axis, and one line per level of the other factor. If the lines are parallel, the effect of each factor is the same regardless of the other - no interaction, the model is additive. If the lines are non-parallel (converging, diverging, or crossing), the effect of one factor depends on the level of the other - an interaction is present. Crossing lines are the dramatic case: the best hero image actually flips depending on which CTA you pair it with.

We build a Lumen test with four cells - hero {A,B} × CTA {A,B} - and bake in a real interaction: hero B only helps when paired with CTA B. Then we fit ols with an interaction formula and read anova_lm to recover it.

import numpy as np, pandas as pd
rng = np.random.default_rng(42)

# per-cell TRUE conversion rate - note B/B is higher than additivity predicts
rates = {("A","A"): 0.032, ("A","B"): 0.034,
         ("B","A"): 0.031, ("B","B"): 0.045}   # interaction lives in B/B

rows = []
n_cell = 8_000
for (hero, cta), p in rates.items():
    y = (rng.random(n_cell) < p).astype(int)
    rows.append(pd.DataFrame({"hero": hero, "cta": cta, "convert": y}))
df = pd.concat(rows, ignore_index=True)
print(df.groupby(["hero","cta"]).convert.mean().round(4))

import statsmodels.formula.api as smf
import statsmodels.api as sm

model = smf.ols("convert ~ C(hero)*C(cta)", data=df).fit()
anova = sm.stats.anova_lm(model, typ=2)
print(anova[["F", "PR(>F)"]].round(4))
# C(hero), C(cta), and C(hero):C(cta) each get an F and a p-value
# the interaction row C(hero):C(cta) should be significant here

The ANOVA gives you a p-value; the interaction plot gives you the intuition. We plot cell-mean conversion with CTA on the x-axis and one line per hero level - and watch the lines refuse to stay parallel.

cell = (df.groupby(["hero","cta"]).convert.mean()
          .reset_index())
pivot = cell.pivot(index="cta", columns="hero", values="convert")
print(pivot)
#          hero        A        B
#  cta A               0.032    0.031
#  cta B               0.034    0.045   <- B jumps only here

import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(5, 4))
for hero in ["A", "B"]:
    ax.plot(pivot.index, pivot[hero], marker="o", label=f"hero {hero}")
ax.set_xlabel("CTA copy"); ax.set_ylabel("conversion rate")
ax.set_title("Interaction plot - non-parallel = interaction")
ax.legend(); fig.tight_layout(); plt.show()
# hero A is roughly flat; hero B climbs sharply from CTA A to CTA B
# the two lines diverge -> the interaction you just tested, visualized

Add factors and the full factorial explodes: 5 two-level factors = 32 cells. A fractional design runs a fraction of them to save traffic - but you pay in aliasing. We generate a half-fraction with pyDOE2 and show exactly which effects become indistinguishable.

from pyDOE2 import ff2n, fracfact
import numpy as np

full = ff2n(3)                 # full 2^3 = 8 runs, factors A B C in -1/+1
print("full factorial runs:", full.shape[0])   # 8

# half-fraction: define C as the product of A and B (generator C = AB)
frac = fracfact("a b ab")      # 2^(3-1) = 4 runs
print("fractional runs:", frac.shape[0])        # 4  -> half the traffic

A, B, C = frac[:, 0], frac[:, 1], frac[:, 2]
AB = A * B
print("C  :", C.astype(int))     # e.g. [ 1 -1 -1  1]
print("A*B:", AB.astype(int))    # e.g. [ 1 -1 -1  1]  -> identical!
print("aliased?", np.array_equal(C, AB))   # True

• Add a third factor. Extend the Lumen test with a layout factor {A,B}, making a 2³ = 8-cell full factorial. At ~8,000 users per cell, that is 64,000 users; at ~4,000 daily checkout sessions, work out the runtime. Feel the cell count bite.
• Count the traffic cost. Tabulate cells and required N for k = 2, 3, 4, 5 factors (2^k cells). Plot cells vs k and mark where Lumen's daily traffic makes the test infeasibly long - that is your practical ceiling on full MVT.
• A/B/n vs MVT decision. Take a real Lumen question - say, "which of 4 hero images and 3 CTAs is best?" - and decide: run one big MVT (12 cells), a sequence of A/B tests, or an A/B/n on the single highest-leverage factor. Justify it by expected traffic, whether you suspect interactions, and how fast you need the answer.
• Optional: refit build-along 1 without the interaction term (convert ~ C(hero) + C(cta)) and compare the fitted B/B prediction to the true 4.5%. See the additive model underpredict exactly the cell where the interaction lived.

Factorial design and interactions are the working core of classical DOE. We teach ~80% of the practical MVT toolkit here - factorial vs fractional, interactions, two-way ANOVA, and aliasing; response-surface optimization and the deeper DOE theory live in the sources below.