Where we are
You met Goodhart's Law back in b3 and watched a gamed number climb in the Fair-Metric Lab while the real outcome collapsed. This session is the maintenance manual for that problem. The framing that matters: a broken metric is almost never a people problem. When a number goes wrong, the usual cause is that the system rewarded exactly what it got - and the people responding to it were being rational. So the diagnosis never starts with "who gamed this". It starts with "what did we make worth doing", and the repair changes the incentive, not the person. One more thing to carry in: a metric can be technically accurate and still broken, because it is measuring something that stopped mattering, or being asked to carry a judgment it was never designed for.
Four failure modes, four different repairs 7 min live
These four get lumped together as "bad metrics", but they fail for different reasons and the fixes are not interchangeable. Name the mode correctly and the repair is usually obvious.
LiveGoodhart, Campbell, and the cobra effect3 min▶
Three ways of saying the same uncomfortable thing:
- Goodhart's Law - when a measure becomes a target, it stops being a good measure. The correlation that made it useful was observed on unmanaged behaviour; hold people to it and the behaviour changes underneath the number.
- Campbell's Law - the more any quantitative indicator is used for decision-making, the more it will be corrupted. Campbell was writing about social programmes and schools, and the pattern is identical inside companies: the higher the stakes on a number, the faster it decays.
- The cobra effect - the classic name for an incentive that produces the opposite of its intent, because people optimise the reward rather than the goal behind it.
The practical consequence is not "do not measure". It is that a metric's usefulness has a shelf life that shortens as its stakes rise. Any number wired to pay decays fastest of all - which is the whole reason this course keeps OKR scores decoupled from compensation.
The support queue that got faster and worse. Cartwheel puts a target on average handling time. It falls beautifully for two months. Then reopens climb, CSAT slides, and refunds tick up: agents are closing conversations early to protect the number, which is precisely what the target asked them to do. Nobody cheated. The metric worked exactly as designed, and the design was the fault.
Self-studySurrogation - when the proxy eats the goal2 min read▶
Surrogation is the subtlest of the four, because nobody is gaming anything. Research on strategy measurement (Choi, Hecht and Tayler, "Lost in Translation") describes how managers come to substitute the measure for the strategic construct it was meant to represent - they start managing the proxy as if it were the goal.
- What it looks like: "customer satisfaction" becomes "the CSAT survey score", and improving the survey becomes the work - better timing of the survey, nudging happy customers to respond - while satisfaction itself flatlines.
- Why it happens: the proxy is concrete, visible, and reported weekly. The construct is abstract and slow. Attention flows to whatever is on the dashboard.
- The mitigation: keep the goal in the room. Name the construct every time the number is shown, use more than one proxy for anything important, and periodically ask "if this number improved and the goal did not, would we notice?"
That last question is the surrogation test, and it is worth asking about your most-watched metric this week.
Five steps from symptom to fix 5 min live
When a number starts behaving oddly, resist two instincts: defending it, and deleting it. Run these five steps instead. Most broken metrics end up repaired rather than removed - demoted to context, paired with a partner, or re-pointed at a lever the team actually controls.
LivePairing: the highest-yield repair3 min▶
Most gaming has a signature: one dimension improves while another silently pays for it. Pairing a metric with the thing that would have to suffer makes the trade visible, so gaming stops being free.
- Speed pairs with quality. Handling time with reopen rate or CSAT. Fast-and-wrong stops looking like a win.
- Volume pairs with an outcome. Tickets closed with first-contact resolution. Features shipped with adoption.
- Growth pairs with retention or cost. New signups with 30-day retention. Revenue with refund rate or margin.
- Any number pairs with a health guardrail - a threshold that must not be crossed while chasing the target. Guardrails do not have to be graded to work; they only need to be watched and honoured.
The pair must be genuinely in tension. Two metrics that rise together are a dashboard, not a control. And keep the set small: two or three numbers that constrain each other beat a scorecard of twelve nobody reads.
Self-studyVanity metrics and the "so what" test2 min read▶
Vanity metrics fail differently from gamed ones: nobody is distorting behaviour, the number is simply useless. The tells are consistent:
- It only goes up. Cumulative totals - total registered users, total orders ever - can only rise, so they carry no information about now.
- No decision hangs on it. Ask what you would do differently if it were 20% higher or lower. If the answer is nothing, it is decoration.
- It has no denominator. Raw counts without a base rate flatter constantly. Sessions without conversion rate, followers without engagement rate.
- It is never compared to anything. No target, no trend, no segment, no prior period.
The cost is not the number itself, it is attention: dashboard space and meeting time that a decision-shaped metric could have used. Retiring a vanity metric is a real improvement, and it is the cheapest one available - though expect resistance if someone reported it upward last quarter.
Diagnose three broken Cartwheel metrics ★ 12 min · everyone tries
Three numbers that have gone wrong in different ways. For each: name the mode, find the incentive, then write the repair. Do not skip to the repair - the mode decides which repair works.
Metric A: support average handling time is down 22%, but reopens are up 31% and CSAT is down 6 points.
Metric B: the CSAT survey score is up, while refund requests and negative reviews are also rising. Someone changed when the survey is sent.
Metric C: "total registered accounts" is on the weekly leadership dashboard and has risen every week for three years.
For each, ask the four questions in order: what is the symptom, what did we reward, could the team control it, and what is the smallest repair that removes the incentive?
| Metric | Symptom | Mode | Repair |
|---|---|---|---|
| Average handling time | Number improves, outcome degrades | Gaming (Goodhart) | Pair with reopen rate and CSAT; grade the pair, not the speed |
| CSAT survey score | Proxy improves, the goal does not | Surrogation | Restate the goal; add a second proxy (refunds, reviews); fix the survey timing |
| Total registered accounts | Only goes up, changes no decision | Vanity metric | Retire from the dashboard; replace with 30-day active rate |
Run a metric post-mortem, then set the review cadence ★ 10 min · do your own
Two moves that turn this from firefighting into maintenance: a blameless post-mortem on the metric that broke, and a standing review that audits your measures the way you already review your results.
Open the post-mortem on the metric, not the team. The opening question is "what did this number make worth doing", never "who did this".
Reconstruct the timeline: when the target was set, when the behaviour shifted, when the partner signal moved, when anyone noticed. The gap between the last two is what you are really fixing.
Write the repair, and write what it will now be paired with. Note explicitly whether the metric stays gradable or drops to context.
Set the metric review cadence: quarterly, alongside the OKR review, ask of every measure whether it is still measuring the thing.
Add one standing item: which numbers has anyone stopped trusting? Loss of trust always precedes the data showing a problem.
| Question | Looking for | Action if it fails |
|---|---|---|
| Is it still measuring the goal? | Surrogation drift | Restate the goal; add a second proxy |
| Has anything improved while the outcome did not? | Gaming | Pair it, or re-point at a controllable lever |
| What decision did it change this quarter? | Vanity | Retire it or attach a decision |
| Is anyone graded on something they cannot move? | Controllability failure | Demote to context; grade the lever instead |
| Does anyone quietly distrust this number? | Early warning | Investigate before the data confirms it |
The metric nobody would say was broken. A Cartwheel weekly report carried an on-time-delivery figure at 97% for eight months while customer complaints about late arrivals climbed. The number was accurate - it measured dispatch against the promise date, and dispatch really was on time. What had changed was the carrier's last-mile performance, which the metric never touched. Three people privately did not trust the figure; none raised it, because it was technically correct and defending it was easier than replacing it. The lesson: accuracy is not validity. Ask what a number does not cover, and treat quiet distrust as data.
Try it yourself - this week ◐ 30-45 min total
- Take your team's three most-watched metrics and label each with a failure mode it is most exposed to. Write the pair you would add.
- Run the surrogation test on your headline number: if it improved and the goal did not, would you notice? How?
- Find one vanity metric on a dashboard you own and retire it. Note the resistance you get, and from where.
- Run a blameless post-mortem on one metric that has already gone wrong. Start with the incentive, never the people.
- Put a quarterly metric review on the calendar next to your OKR review, with the five questions as its agenda.
- Ask two people privately which of your team's numbers they do not trust. Bring the answers to b10.
Sources covered (cited)
Taught from the metric-failure canon. This session covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Handling time falls 22% while reopens rise 31%. What is the failure mode, and the repair?
The number improved while the outcome degraded: classic Goodhart. Pair speed with reopen rate or CSAT so fast-and-wrong stops counting as a win. Nobody cheated - the target asked for exactly what it got.
2 · The surrogation test is...
Surrogation is the measure quietly replacing the goal - improving the CSAT survey rather than satisfaction. The mitigation is keeping the construct in the room: name the goal whenever the proxy is shown, and use more than one proxy for anything important.
3 · Why is deleting a broken metric usually the wrong first move?
Diagnose the mode, then remove the incentive. Most broken metrics are repaired rather than deleted: paired with a partner, re-pointed at a controllable lever, or demoted to context on a dashboard.