learn-course-design-with-phoebe / Delivery session 5 of 6
Learn Course Design with Phoebe · Delivery track, session 5 of 6

Measuring behaviour change

Almost all training measurement is a form asking whether people enjoyed it, collected while they are still in the room and still being polite. It is the easiest thing to measure and it predicts almost nothing about whether anybody does anything differently. Tonight: the four levels and what is actually wrong with them, how to pick the one behaviour that already leaves a trail, why you should write your prediction down first, and how to be honest about attribution without losing the argument.

🟠 Hands-on Anyone who has to justify a training spend Bring your b4 transfer plan 45 min
0-3 · Welcome 3-18 · Four levels, and the one data trail 18-40 · Build-along: pick one measure 40-45 · Q&A
Part 0

Where we are

b4 gave you a model of what should happen. Tonight is about finding out what did - with the resources you actually have, which are usually none.

🎫 Clear all three quiz questions to stamp your passport Live = presented in the 45 minutes Self-study = the full depth, after class

★ What you walk out with tonight. One behaviour to measure, an existing data trail to measure it with, a written prediction, and a sentence about attribution that is honest and still persuasive.

Part 1 · covers the levels

Four levels, and the honest critique

6 min live

The four-level model is the standard vocabulary and it is worth knowing for that reason alone. It is also worth knowing what is wrong with it, because being able to name that is what stops you being pushed into measuring the useless one.

Level Measures Measured how often Predicts behaviour 1 · Reaction did they enjoy it nearly always barely 2 · Learning can they do it now sometimes somewhat 3 · Behaviour do they do it at work rarely it is the thing 4 · Results did the number move almost never confounded by everything Three honest objections worth being able to state The levels are not a causal chain - a happy room does not lead to a changed behaviour. Effort is inversely proportional to usefulness, so everybody measures level 1 and stops. Level 4 is almost never attributable, and claiming it is where training loses credibility.
🔍 Click to zoom - the ranking of usefulness is the exact reverse of the ranking of convenience
LiveAim at level 3, and only one behaviour4 min▶

Level 3 is the one worth having and it is the one nobody measures, because it requires knowing what to look at and waiting a month. Both are solvable, and the trick is to be much less ambitious than measurement usually is.

  • One behaviour, not a programme. "Do producers open the contract file when they change a schema?" One thing, countable.
  • Find the trail that already exists. Do not build instrumentation. Pull requests, tickets, commit history, the support queue, the incident log. Somebody is already recording something adjacent to the behaviour you care about.
  • Accept a proxy, and say it is one. "Deprecation notices opened per month" is not the behaviour, it is a shadow of it. Naming it as a proxy out loud costs you nothing and buys you enormous credibility.
Daybreak's measure

Schema-changing pull requests that reference the contract file, as a percentage of all schema-changing PRs. Already in the git history. No instrumentation, no survey, and it can be counted for the three months before the session as well as after - which is worth more than anything you could start collecting today.

Part 2 · covers honesty

Predict first, and be straight about attribution

6 min live
Written before, dated Observed at day 60 Baseline: 18% of schema PRs reference the contract 31% Expected after: about 45% below prediction Would make me revert: no change at all by day 60 not triggered Why the middle row is the valuable one 31 percent is a real improvement and it is below what you predicted. Without the written prediction, 31 is simply a success. With it, you have a genuine question worth investigating.
🔍 Click to zoom - the same result reads as a win or as a finding, depending on whether you wrote the guess down
LiveAttribution, said honestly and still persuasively4 min▶

You will almost never be able to prove your session caused the change. Something else always happened in the same quarter - a reorganisation, an incident, a new tool, a manager who started caring. Overclaiming here is the single fastest way for a training function to lose credibility with anybody numerate.

The honest version is also the more persuasive one:

The sentence

"Contract references went from 18 percent to 31 percent over sixty days. The session is one of three things that changed in that window - the CI check and the definition-of-done line are the others. I would not claim the session caused it. I would say the package is working and I can tell you which part I would drop first."

  • Name the other causes yourself, before somebody else does. It costs you the overclaim and buys you the room's trust for everything else you say.
  • Offer the drop-first opinion. It shows you have thought about relative value rather than just defending your own budget line.
  • If you genuinely need causal evidence, that is a control group and a different discipline - learn-experimentation-with-phoebe. Most of the time it is not proportionate, and saying that is fine.
Build-along

One behaviour, one trail, one prediction

★ 22 min · everyone builds

The behaviour. One. Observable, countable, at work rather than in the room. "Producers reference the contract file when changing a schema."

The trail. What already records something adjacent? Git history, tickets, the incident log, the support queue. If your answer is "we would need to build something", pick a different behaviour.

The baseline. Count the three months before. This is free right now and impossible later, and forgetting it is the most common measurement mistake there is.

The prediction. Written and dated before the session. What do you expect at day 60? Being wrong is the useful outcome.

The revert line. What result would make you stop doing this? One sentence, written while you are still honest about it.

Write your one behaviour. If it contains "understand", "awareness" or "engagement", it is not a behaviour. Rewrite until somebody could count it.

Go and find the trail before you design anything else. Fifteen minutes in git, the ticket system or the incident log. If nothing exists, either pick a different behaviour or accept a proxy and label it.

Pull the baseline now. Three months back. Do this today - the moment the session runs, the clean before-period stops being available.

Write the prediction and date it. Same discipline as the pre-registered guess in learn-product-design-with-phoebe session 2, and for the same reason: without it, every result is a success.

Write the revert line. "If contract references have not moved at all by day 60, this was not the right intervention and I will say so." One sentence, before launch.

Draft your attribution sentence. Name the other two things that changed. Practise saying it, because the temptation to drop that clause when the number is good is considerable.

Decide the reporting date now, and put it in a calendar. Day 60. Measurement that is not scheduled does not happen, and unmeasured training is why the whole function keeps having to justify itself.

If you take one thing from tonight. Pull the baseline today. It is fifteen minutes of work that is free right now and permanently unavailable the moment your session runs, and it is the difference between a number and a comparison.

Homework

Before session b6

30 min
  1. Pick one behaviour and find its existing trail. Fifteen minutes of looking. Most people find something adjacent within ten.
  2. Pull a real baseline for something you are about to run, or for something you ran recently.
  3. Write a dated prediction and put it somewhere you will find it at day 60.
  4. Write your revert line. Then check whether you would actually honour it.
  5. Bring the baseline number to b6, because a business case with a real before-number in it is a different document from one without.
Check yourself

Three questions

Clear all three and this session gets stamped in your passport.

1 · Why does nearly everybody measure level 1 and stop?

A feedback form takes two minutes at the end of the room. Level 3 needs a defined behaviour and a month of patience. And the levels are not a causal chain - a happy room does not lead to a changed behaviour, which is exactly why level 1 cannot stand in for the others.

2 · What makes "schema-changing PRs that reference the contract file" a good measure?

Narrowness is the feature. One countable behaviour with a pre-existing trail beats a comprehensive measurement programme nobody will run - and the fact that the history goes backwards gives you a baseline you could not otherwise reconstruct.

3 · The number moved from 18 percent to 31 percent. What should you say?

Overclaiming causation is the fastest way for a training function to lose credibility with anybody numerate, and staying silent forfeits a real result. Naming the other causes yourself costs you the overclaim and buys trust for everything else - and the drop-first opinion shows you are thinking about value rather than defending a budget line.

Sources covered

What this session teaches, and where it came from

✓The four levels, with the critiqueReaction, learning, behaviour, results - plus the three honest objections: not a causal chain, effort inversely proportional to usefulness, level 4 rarely attributable
✓One behaviour, one existing trailGit history, tickets, incident logs. If it needs new instrumentation, pick a different behaviour
✓Proxies, labelled as proxiesNaming a shadow measure as a shadow costs nothing and buys credibility
✓The baseline and the dated predictionFree now, impossible later. Same discipline as the pre-registered guess in learn-product-design-with-phoebe
✓Attribution, honestlyName the other causes yourself and offer a drop-first opinion. Overclaiming is how the function loses numerate people
◐Control groups and formal evaluationlearn-experimentation-with-phoebe. Usually not proportionate, and saying so is a legitimate answer

Every fact on this page, and its verification tier, is recorded in the course's source map. The Daybreak figures are a teaching scenario; the method is real.

Delivery b5 cheat sheet · pin this

Aim at level 3Behaviour at work. Level 1 is the easiest and predicts the least.
Not a causal chainA happy room does not lead to a changed behaviour. Say so when asked.
One behaviour, existing trailIf it needs new instrumentation, pick a different behaviour.
Pull the baseline todayFree now, impossible the moment the session runs.
Predict, datedWithout it, every result is a success and nothing is a finding.
Name the other causesCosts the overclaim, buys trust for everything else you say.