learn-ai-marketing-with-phoebe / Builder session 9 of 10
Learn AI + Marketing with Phoebe · Builder track · Session 9 of 10

Analytics & optimization: read the numbers, design the next test

The Cadence campaign is live and the dashboards are filling up. Tonight you turn AI into an analysis co-pilot: paste the metrics you gathered, get a clean readout of what worked and what did not, then design one sharp next experiment. The discipline underneath: AI helps you see patterns and draft the story, but it never sources the numbers and never decides the spend - that stays with you.

🟠 Builder track Practitioners: growth · analytics · lifecycle ChatGPT or Claude 45 min
0-3 · Welcome 3-21 · AI as an analysis co-pilot 21-42 · Build-along: readout + next test 42-45 · Q&A
Part 0

Where this sits

You have built the Cadence campaign end to end - positioning, content, SEO, ads, email, social, and creative. Now it runs, and the question shifts from "what should we make?" to "what did it do, and what do we try next?" AI is genuinely useful here as a fast, patient analyst: it summarizes, spots patterns, and drafts the readout in your voice. It is also exactly where over-trust bites - so we keep b1's first rail front and centre: the model is not the source of truth for a single figure.

Live - presented in session Self-study - read after class ★ Try it now prompt Official sources covered
★ What you walk out with today An AI-assisted readout of the Cadence campaign - what worked, what did not, and the caveats - plus a clean next-experiment design (hypothesis, one variable, metric, decision rule) you can run next week. Note: deep multi-touch attribution modeling is the sibling course learn-marketing-attribution-with-phoebe. Here we read results and design experiments; we do not build attribution models.
Part 1 · covers Harvard DCE predictive analytics (reading, not modeling)

AI as an analysis co-pilot 6 min live

Think of AI as a sharp junior analyst who reads fast and never gets bored - but who has no access to your accounts. You supply the numbers; it summarizes them, surfaces patterns, and drafts the readout. It does two things it must never do: source the figures itself, and decide where the money goes. Keep those two off its desk and it earns its seat.

Metrics you supply CTR, CPA, conversions, opens, spend - copied from the platform AI reads + patterns summarizes, spots what moved, drafts the readout - never sources numbers You decide the spend + next move - AI advises, it never decides the budget AI reads the numbers you supply and verify; it never sources them and never decides the spend.
🔍 Click to zoom - the readout pipeline: you supply and verify, AI summarizes, you decide
LiveWhat a good AI readout gives you - and what it must not3 min▶

Paste a small, clean metrics table and a good prompt turns it into a story a human can act on: the headline, the two or three things that moved, and the honest caveats. That is real leverage - it is the "restructure and reason over my brief" strength from b1, pointed at numbers instead of copy.

  • Summarize + prioritize. Which channel, ad, or subject line pulled its weight, and which quietly burned budget. AI is fast at ranking and grouping.
  • Spot patterns you would miss. "Opens are strong but clicks collapse on mobile" is the kind of cross-cut it surfaces in seconds.
  • Draft the readout in your voice. A tidy paragraph plus a bullet list your stakeholders can read - not a wall of raw rows.
  • But it must not source figures. If a number is not in what you pasted, the answer is "[need data]", never an invented one. This is b1's first rail, in the analytics chair.
★ Try it now (any chat AI)Here are last week's Cadence email numbers (I am supplying them; use only these): sent 4,200; opens 1,510; clicks 190; signups 34. Summarize what stands out in three bullets, and if you need a figure I did not give you, write "[need data]" instead of guessing.
The one rule that keeps this safe Every figure in the readout must trace back to something you pasted, which traces back to the platform. If AI produces a number you did not supply, treat it as fabricated until you verify it in the source dashboard.
Self-studyBring your own numbers - AI has no dashboard2 min read▶

A chat AI cannot see your Google Ads, Meta, HubSpot, or GA4 account. It only knows what you paste. That is a feature: it forces you to pull real figures from the source of truth before the conversation starts, and it keeps the analysis grounded exactly like b1's slop-vs-grounded demo.

AI is good at (once you supply data)Only your platform can do
Summarizing a metrics table into a readoutReport the actual numbers - the source of truth
Ranking channels and flagging outliersAttribute a conversion to a touchpoint
Drafting stakeholder-ready commentaryConfirm spend, budget caps, and pacing
Suggesting what to test nextRun the test and record the result
Part 2 · the growth loop that compounds

From readout to next experiment 5 min live

A readout that ends in "interesting" is wasted. The point of reading results is to earn the next test. Every signal in the readout hints at a hypothesis; a good experiment turns one hypothesis into a clean test that changes exactly one variable. AI is great at drafting a dozen hypotheses fast - you prioritize the one worth a week.

Read results the AI-assisted readout Form hypothesis AI drafts many, you pick one Run one test change one variable, set a decision rule loop back: the result becomes next week's readout One variable per test - change two and you will not know which one moved the metric.
🔍 Click to zoom - the growth loop: read, hypothesize, test one thing, read again
LiveTurn a signal into a clean, one-variable test3 min▶

A clean experiment has four parts and one discipline. The four parts: a hypothesis (if we change X, metric Y improves because Z), the one variable you change, the single metric that decides it, and the decision rule you set before you look. The discipline: change one thing at a time, so the result actually teaches you something.

Signal in the readoutExperiment it suggests (one variable)
Strong opens, weak clicks on the Cadence emailTest the CTA wording only; hold subject + body constant
Ad A beats Ad B on CTR but not signupsTest the landing-page hero only, same ads and audience
Teams-plan page converts far below solo planTest one proof point on the Teams page, nothing else
Mobile signups lag desktop badlyTest the mobile form length only, desktop untouched
★ Try it now - draft hypotheses to prioritizeGiven this readout (paste yours), draft five one-variable experiments for the Cadence campaign. For each: the hypothesis, the single variable changed, the metric that decides it, and a suggested decision rule. Rank them by likely impact vs effort - I will pick which one we run.
Real world

The test that proved nothing. A team changed the subject line, the send time, and the offer in one email "to move faster". It won - and they had no idea why, so they could not repeat it. The next three "copies" of that win flopped. One variable at a time is slower for a week and far faster over a quarter.

Part 3 · the rails, applied to analytics

Guardrails: verify, assist-not-decide, correlation is not cause 4 min live

Analytics is where a confident AI does the most quiet damage, because numbers feel objective. Three guardrails keep the co-pilot honest.

LiveVerify every figure, and let AI advise not decide the spend2 min▶

Two habits carry most of the risk. First, every number that reaches a stakeholder is checked against the platform - the dashboard is the source of truth, not the chat window. Second, AI can recommend where to shift budget, but a human signs off on the move. That is b1's rail, worded for a growth team: AI assists, you decide the spend.

  • Verify at the source. Re-open the dashboard and confirm the figures the AI summarized. A transposed digit in your paste becomes a wrong decision.
  • Assist, do not decide. "Shift 20% from Ad B to Ad A" is a recommendation to weigh, not an instruction to execute. Money moves on human sign-off.
LiveWatch for over-confident causal claims2 min▶

AI will happily say "the new subject line caused the lift" from two weeks of data. Often it is a coincidence, a seasonal bump, or something else that moved at the same time. Correlation is not causation, and a co-pilot that forgets this will talk you into repeating flukes.

  • Ask for the caveat. A good readout names what could explain the change besides your test - seasonality, a price change, a PR spike.
  • Causal certainty needs a real experiment. A clean A/B test earns a causal claim; a before-and-after glance does not.
Prompt the humility in End your readout prompts with "flag anything besides our change that could explain this, and say where the data is too thin to draw a conclusion". You get analysis, not overconfidence. Deep attribution - untangling which touchpoint truly drove the sale - is the sibling course learn-marketing-attribution-with-phoebe.
Demo 1 of 2

AI-assisted readout of the Cadence campaign ★ 13 min · everyone builds

Give AI a small, supplied Cadence metrics table and turn it into a structured readout you would send a stakeholder - what worked, what did not, and the honest caveats. Then you verify every figure. This is the whole session in one build.

Open any chat AI. Use the supplied Cadence numbers below (or paste your own real figures - never customer PII, rail two). You are the source of every number.

Ask for a structured readout: a one-line headline, "what worked", "what did not", and "caveats + what I cannot conclude yet". Require a "[need data]" tag for anything you did not supply.

Read it critically. Does every figure trace back to your table? Is any claim more confident than two weeks of data allows? Push back in the chat and ask it to soften over-reaches.

Verify. Pick three figures from the readout and confirm them against the numbers you pasted (and, in real life, the platform). This step is non-negotiable.

Save the readout. It becomes the input to demo 2, and the template you reuse every reporting cycle.

★ The readout prompt (supplied Cadence numbers)You are my analytics co-pilot. Use ONLY the figures I supply below - do not add or estimate any number. If you need a figure I did not give, write "[need data]". Cadence campaign, last 14 days: - Blog + SEO: 6,400 visits, 512 free signups - Paid ads: spend $3,000, 1,900 clicks, 141 free signups - Email nurture: 4,200 sent, 1,510 opens, 190 clicks, 34 Teams-plan trials - Social: 22,000 impressions, 610 clicks, 48 free signups Write a stakeholder readout: one-line headline, "what worked", "what did not", and "caveats + what we cannot conclude yet". Flag anything besides our own work that could explain a change, and do not claim causation from this data alone.
Real world

The readout that caught a typo. A marketer pasted a metrics table, got a crisp readout, then ran the verify step - and found she had dropped a zero on ad spend. The AI had faithfully summarized a wrong number into a confident (and wrong) conclusion. The tool did its job; the verify step did the rest. Garbage in, confident garbage out.

Demo 2 of 2

Design the next experiment ★ 8 min · build your own

Take the readout from demo 1 and turn its strongest signal into one clean test you could launch next week. Four parts, one variable, one decision rule set before you look.

Point AI at your readout and ask for five one-variable experiments, ranked by impact vs effort. Let it draft; you are the one who prioritizes.

Pick one. Sharpen it into the four parts: hypothesis (if we change X, Y improves because Z), the one variable, the deciding metric, and the decision rule (e.g. "ship the variant if signups rise 15%+ with enough sample").

Pressure-test that exactly one thing changes. If your "test" also moves the audience, the offer, or the timing, split it - you would not learn which lever worked.

Write the one-line plan: variable, metric, rule, and how long it runs. That is a runnable experiment, and the loop that will make the next campaign better than this one.

★ The next-experiment promptHere is my Cadence readout (paste demo 1 output). Its strongest signal is [name it]. Design ONE clean experiment from it. Give me: 1) Hypothesis: if we change X, metric Y improves because Z. 2) The single variable we change (everything else held constant). 3) The one metric that decides it. 4) A decision rule set before we look at results. 5) How long to run it and roughly how much traffic we need to trust the result. Keep it to one variable. If my idea secretly changes more than one thing, split it and tell me.
Why demo 2 matters most A readout tells you about last week; an experiment earns you a better next week. This loop - read, hypothesize, test one thing, read again - is how a good campaign compounds instead of just running.
Homework

Try it yourself - this week ◐ 30-45 min total

Source material

Official sources covered

Taught from public AI-marketing curricula and vendor documentation. Certificates and paid tooling stay with the official programs. This session covers:

◐Harvard DCE - AI Marketing: predictive analytics (CLV / churn)Parts 1-2 · reading results, patterns, experiment design (reading, not modeling)
✓2026 AI-marketing guides - analytics + optimization modulePart 1-2 · AI co-pilot readouts, the test-and-learn loop
✓b1 rails, applied to analyticsPart 3 · verify every figure; AI assists, you decide the spend
◐Deep multi-touch / marketing-mix attributionNot here by design → sibling course learn-marketing-attribution-with-phoebe
Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · Your AI readout includes a conversion-rate figure you do not remember pasting. What do you do?

AI is a co-pilot, not the source of truth. Any figure you did not supply is unverified - confirm it in the dashboard before it reaches a stakeholder. This is b1's first rail in the analytics chair.

2 · You have a solid readout. What is the point of it?

Reading results is only half the loop. The readout's job is to surface a signal you turn into a hypothesis and a one-variable test, so the next cycle beats this one. Read, hypothesize, test, read again.

3 · AI recommends shifting 20% of budget from Ad B to Ad A. What is the right move?

AI assists, you decide the spend. A budget recommendation is input to a human decision, not an instruction to execute. Money moves on human sign-off, every time.

Builder session 9 cheat sheet · pin this

AI's analytics rolea fast analyst with no dashboard - it summarizes, ranks, and drafts the readout from numbers you supply.
Bring your own numbersa chat AI cannot see your accounts. Paste real figures from the source of truth; that keeps analysis grounded.
Never sources figuresanything the model produces that you did not supply is fabricated until verified. Require a "[need data]" tag.
The readout shapeheadline → what worked → what did not → caveats + what you cannot conclude yet.
The growth loopread results → form a hypothesis → run one clean test → read again. That is how a campaign compounds.
One variablechange one thing per test, set the decision rule before you look, or you will not know what moved the metric.
Assist, not decideAI recommends where budget could shift; a human signs off. AI assists, you decide the spend.
Correlation ≠ causemake AI flag other explanations; a clean A/B test earns a causal claim, a before-after glance does not.