Where we are
In a3 we decomposed shared goals so each team is graded on a lever it controls. But even a perfectly-owned metric is scored by a human, and humans are not measuring instruments. Today we look at how measurement actually goes wrong: a rating tells you as much about the manager who gave it as the person who received it; a handful of predictable biases skew every score; and the very language of feedback is unequal - women, on the research, receive about twice as much vague, personality-based feedback as men. Then the constructive half: what "fair" actually means (it is mostly about process, not outcome), why development and evaluation must not share a form, and the rule that protects all of it - an OKR score never sets pay.
Most of a score is about the person giving it 7 min live
The single most uncomfortable finding in performance measurement: when you decompose the variance in ratings, the largest chunk is the rater's own idiosyncrasy, not the ratee's performance. The "objective number" is mostly a fingerprint of who happened to hold the pen.
LiveThe five biases that quietly bend every score3 min▶
On top of rater idiosyncrasy sit predictable, well-documented biases. Naming them is the first defence:
- Recency - the last few weeks dominate a whole year's rating.
- Halo - one strong trait (or a good relationship) spills over and inflates everything else.
- Similarity / affinity - raters score people who resemble them more generously.
- Leniency vs central tendency - some managers rate everyone high; others cluster everyone in the safe middle. Two employees with identical work get different numbers from different managers.
None of these are cured by asking managers to "be fair." They are cured by process: more than one rater, calibration across managers, evidence over impressions, and structured rather than free-form scoring.
Self-studyBias hides in the words, not just the score2 min read▶
Even before a number is assigned, the language of feedback is unequal. Correll & Simard (HBR 2016), drawing on the Textio review-bias work, found that women receive markedly more vague, personality-based feedback ("you're great to work with," "a bit abrasive") while men receive more specific, skill-based, actionable feedback ("tighten your financial modelling"). Reported figures put the vague-feedback gap at roughly two to one.
This matters to a leader for a hard reason: vague feedback cannot be acted on. If one group systematically gets feedback they cannot use to improve, the measurement system is quietly widening a gap rather than closing it - regardless of what the final scores say. Fair measurement audits the words, not only the numbers.
People accept a hard outcome from a fair process 6 min live
Ask people what "fair" means and they say "I got what I deserved." The research on organizational justice (Greenberg 1987) says something more useful for leaders: how you decide often matters more than what you decide. A fair process buys trust even when the answer disappoints.
Self-studyDevelopment and ranking should not share a form2 min read▶
Measuring to develop a person and measuring to rank them for reward are opposite activities wearing the same word. Development needs honesty about weaknesses; ranking punishes admitting them. Put both on one form and people manage the number instead of telling you the truth - so you lose the coaching data and the ranking is corrupted by defensiveness.
The linchpin that keeps the whole system honest is Doerr's rule from Measure What Matters: decouple OKR scores from pay and ratings. When an OKR score does not decide compensation, people set ambitious goals and report progress truthfully - because there is no penalty for a stretch goal landing at 0.7. Wire the score to pay and everyone sandbags: they set goals they know they will hit, and your measurement system stops measuring anything real.
None of this means abandoning evaluation - reward decisions still have to be made. It means making them from a broader body of evidence (calibrated ratings, a manager's holistic judgment, sustained results over time) rather than from a raw OKR score. The OKR feeds the conversation; it does not automate the verdict.
Spot the rater, not the ratee, in a Cartwheel review ★ Worked example (Cartwheel) · 10 min
Two Cartwheel managers rate the same analyst on the same quarter. Watch how much of the gap is about them, not her.
Manager A (a lenient rater who saw her strong March) scores the analyst 4.6 / 5. Manager B (a central-tendency rater who remembers a missed April deadline) scores her 3.2 / 5.
Same person, same work, a 1.4-point gap. Most of that gap is rater idiosyncrasy plus recency, not a change in her performance.
The fair response is not to average two noisy numbers - it is to add process: a second reviewer, calibration across managers, and evidence tied to specific outcomes.
Report her result as target + trend + context, never as a single verdict number that hides which rater produced it.
| Signal | Manager A | Manager B |
|---|---|---|
| Score given | 4.6 / 5 | 3.2 / 5 |
| Rater tendency | lenient | central / strict |
| What they anchored on | strong March (recency) | missed April deadline (recency) |
| Actual performance signal | unchanged - the gap is mostly the rater | |
Rewrite vague feedback into something usable ★ Thought exercise · 8 min
Vague, personality-based feedback is not just unfair - it is useless, because it cannot be acted on. The leadership skill is turning an impression into a specific, actionable observation. Watch the move.
Before: "She's great to work with but can be a bit abrasive in meetings." This is personality-based, vague, and impossible to improve against - and it is exactly the kind of language the research finds women receive far more often.
After: "In the last two planning meetings she interrupted colleagues three times before they finished. Next quarter, hold questions until the speaker is done, then summarise their point before adding yours." Specific, behavioural, time-bound, and actionable - the same message made fair.
The test for any line of feedback: could the person do something differently tomorrow because of it? If not, it is describing your impression, not their behaviour.
Strip out the personality label ("abrasive," "great," "not a team player") - it describes the rater's impression, not a behaviour.
Replace it with the specific behaviour, when it happened, and its effect.
Add a concrete, doable next step so the person can actually improve against it.
Try it yourself - this week ◐ 30-45 min total
- Find one recent performance review you wrote or received. Highlight every vague or personality-based phrase ("great," "abrasive," "not strategic enough").
- Rewrite each highlighted line into a specific, behavioural, actionable version: what happened, its effect, and a concrete next step.
- Count the ratio - how much of the original review was actionable versus impression? Note the gap.
- List which of the five biases (recency, halo, similarity, leniency, central tendency) you are personally most prone to as a rater.
- Pick one score in your org that is currently wired to pay. Sketch how you would decouple it - what the score would inform instead, and what evidence would drive the reward decision.
- Optional: read Correll & Simard, "Vague Feedback Is Holding Women Back" (HBR 2016) before a5.
Sources covered (cited)
Taught from the fair-measurement and organizational-justice canon. This session covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · According to Scullen, Mount & Goff (2000), what does most of a performance rating measure?
More than half the variance in ratings is the rater, not the ratee. A single score from a single manager is a weak signal - the fix is multiple raters and calibration.
2 · Greenberg's research suggests that what builds trust most is often...
Procedural justice frequently matters more than the outcome itself. A consistent, unbiased process where the person had a voice builds trust even when the answer disappoints.
3 · Why must OKR scores stay decoupled from pay?
Doerr's linchpin: when an OKR score does not set compensation, people set ambitious goals and report truthfully. Tie it to pay and they set safe goals they know they will hit.