Where we are
Last session you split the conversations so honesty could survive. This one attacks the next leak: even a well-run, well-separated conversation can carry a rating that is mostly noise. Three ideas carry the session. First, a rating is a measurement instrument, and the instrument is a human being - so most of what a rating varies with is the rater, not the rated. Second, bias travels in language, not just in numbers: the words a manager reaches for differ by who is being described, and the difference is measurable. Third, calibration is the fix, but only if it is run properly - done badly it becomes forced ranking with a friendlier name. None of this is a reason to stop measuring. It is a reason to measure with the error in view.
The idiosyncratic rater effect 7 min live
Scullen, Mount and Goff took thousands of multi-source performance ratings apart to see what the variance was actually made of. The uncomfortable answer: the largest share traced to the individual rater's own way of rating, and only a modest share to the ratee's actual performance.
LiveWhat the rater effect does and does not mean3 min▶
The finding is easy to over-read in both directions, so be precise about it:
- It does mean that a single manager's rating is a noisy instrument. Two honest, competent managers rating the same person will often land in different places, and the gap is mostly them, not the person.
- It does mean that one number from one rater should never be treated as an objective fact about a human being.
- It does not mean raters are lying or that performance is unmeasurable. Real differences exist - they are just a smaller share of the variance than we assume.
- It does mean the fix is structural: more than one perspective, criteria fixed in advance, evidence attached to claims, and calibration to level out rater set points.
Two managers, one analyst. A Cartwheel analyst moves from Merchandising to Growth mid-year. Her old manager rates her a 4 (he rates almost everyone 4). Her new manager rates her a 3 (he reserves 4 for exceptional). Nothing about her work changed between those two numbers. Without calibration, the average of two rater set points becomes her "performance" - and follows her into a pay decision.
Self-studyTwo quieter biases: contrast and similarity2 min read▶
Beyond the four classic distortions, two more do damage precisely because they feel like good judgment:
- Contrast bias - you rate the person against whoever you assessed just before them, not against the criteria. Review order quietly becomes part of the score. The fix is dull and effective: rate every person against the written criteria, one criterion at a time, and shuffle the order.
- Similarity bias - people who work, communicate, or think like you read as "strong". This is how homogeneous teams grow while everyone believes they are being meritocratic. The fix is to ask, for each rating, which evidence would look different if this person's style matched yours less.
Both biases share a tell: you can describe the impression clearly but cannot name the evidence. That gap is the signal to slow down.
What the review language gives away 5 min live
Bias rarely shows up as a lower number. It shows up as vaguer praise, more comment on personality, and less concrete, career-building specificity - and those differences have been measured in real review corpora.
LiveCalibration done fairly, and the trap next to it4 min▶
Calibration is a room where managers compare how they are applying the criteria, so one manager's 4 means roughly what another's does. Done well it is the single best correction for rater set points. Done badly it becomes forced distribution wearing a nicer label.
- Fair calibration - criteria agreed in advance, evidence required for every claim, discussion of how the standard is being applied, outliers examined in both directions, and a written record of what changed and why.
- The trap - arriving with a quota ("only 15% can be top"), ranking people against each other rather than the criteria, and letting the most confident voice in the room set the standard. That does not remove bias; it launders it and adds a new one.
A blunt test: if the session's output is a rank order, it was a ranking meeting. If its output is a shared, evidenced standard plus a handful of adjusted ratings with reasons attached, it was calibration.
Self-studyWhy "no rating at all" is not the answer2 min read▶
A tempting response to the rater effect is to scrap ratings entirely. Some organisations tried it, and the honest result is mixed: when the formal rating disappears but pay and promotion decisions do not, judgment simply moves somewhere less visible - hallway reputation, a manager's advocacy, who is remembered in the room. That is not less biased; it is only less auditable.
- Keep a decision record. If a judgment affects someone's pay or level, write down the criteria and the evidence, whatever you call the output.
- Reduce the stakes on any single reading. Multiple perspectives, a longer window, and evidence beat a cleverer scale.
- Separate steering from evaluation (b6's golden rule), so most measurement is low-stakes and honest, and the rare high-stakes judgment is properly evidenced.
The goal is not a perfect number. It is a defensible, contestable, well-evidenced judgment that the person can see and challenge.
Audit a Cartwheel review cycle's language ★ 12 min · everyone tries
Four real-sounding lines from Cartwheel's last review round. Flag the bias pattern in each, then rewrite it as behaviour plus impact plus next step - the b6 shape, now used as a bias control.
Read each line and ask the first audit question: is there any evidence here? If you cannot name the behaviour, the line is an impression.
Ask the second: is this about the work or the personality? Personality words are where bias concentrates.
Ask the third: would this help the person get promoted? Vague praise reads well and advances nobody.
Rewrite each line with behaviour, impact, and a next step. Then compare across the team: who got specifics and who got adjectives?
| Review line | Pattern | Rewritten |
|---|---|---|
| "Very supportive, a real team player." | Vague praise, no evidence | "Onboarded two analysts in Q3; both shipped their first dashboard inside a month." |
| "Can come across as abrasive in reviews." | Personality criticism | "In the last two design reviews, feedback was given without naming what to change. Next: lead with the specific ask." |
| "Had a fantastic quarter, great energy." | Recency plus halo | "Cut checkout errors 18% in Q4 after a flat Q2 and Q3. Full-year trend is upward." |
| "Not quite ready for senior yet." | Unfalsifiable, no criteria | "Senior requires leading a cross-team project; she has led within Growth only. Next: own the returns project in Q1." |
Run a fair calibration session ★ 10 min · do your own
Five Cartwheel managers, one standard to share. Design the room so it corrects rater set points instead of producing a ranking - then run the four-question self-audit before you submit anything.
Before the room: every manager writes their rating with evidence attached, against criteria fixed in advance. No evidence, no claim.
Open with the standard, not the people: what does each level actually look like in observed behaviour at Cartwheel?
Examine outliers in both directions - the unusually generous rater and the unusually harsh one. Set points are the target, not people.
Record every change and its reason. A rating that moves without a written reason is a new bias, not a correction.
Close by naming who got specifics and who got adjectives across the whole cycle, and fix the language before it goes out.
| Rule | Why it exists | Failure mode it blocks |
|---|---|---|
| Criteria fixed before ratings | Stops the standard moving to fit a conclusion | Post-hoc justification |
| Evidence required per claim | Impressions cannot be examined; behaviour can | Halo, similarity bias |
| No quota, no forced curve | Calibration levels set points, it does not allocate | Forced ranking in disguise |
| Outliers both ways | Harsh raters distort as much as generous ones | Leniency and severity |
| Written reason for every change | Makes the decision contestable later | Loudest-voice-wins |
The quota that ate the calibration. A common pattern: leadership tells the calibration room that "roughly 15% should be top box." The moment that lands, managers stop comparing standards and start trading people. The strongest advocate wins, the quietest team loses, and the outcome correlates with manager confidence rather than performance. Calibration levels rater set points; it must never be handed a distribution to hit. If a distribution is genuinely required by a pay budget, keep that decision in a separate, clearly-named step - it is a budgeting call, not a measurement one.
Try it yourself - this week ◐ 30-45 min total
- Pull your last review cycle and count, per person, how many statements name a behaviour with evidence versus how many are adjectives. Look at how that splits across your team.
- Search your own feedback notes for "abrasive," "aggressive," "too nice," "not a culture fit," "great attitude." Rewrite each as behaviour plus impact plus next step.
- Estimate your own set point: are you a generous rater or a harsh one? Ask a peer manager to compare two ratings against the same criteria and see how far apart you land.
- Take one rating you are least sure about and run the four-question self-audit on it. Note what changes.
- If your organisation calibrates, check the session against the five rules above. If it does not calibrate, draft the agenda you would propose.
Sources covered (cited)
Taught from the rating-bias research. This session covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · The idiosyncratic rater effect says the largest share of variance in a performance rating traces to...
Scullen, Mount and Goff reported idiosyncratic rater effects of roughly 53-62% of variance, with a far smaller share traced to the ratee's actual performance. One rater's number is a noisy reading, not a verdict.
2 · A review line reading "great attitude, real team player" is a problem mainly because...
Vague praise reads well and advances nobody. Research reports women receive more of it and less specific, business-linked feedback - so the gap costs careers even when the score looks generous.
3 · What turns a calibration session into forced ranking in disguise?
Calibration exists to level rater set points against a shared standard. Hand the room a distribution to hit and managers start trading people; the loudest advocate wins and a new bias replaces the old one.