Where we are
Nine lenses are now on the bench, and between them they hold well over a hundred principles. They resolve to twenty distinct findings on one screen. That compression is the whole craft: a review instrument is not the union of every standard you respect, it is the smallest set that still catches everything you care about.
★ What you walk out with tonight. A finished, defensible review instrument with a named source behind every check. A severity-ordered fix list rather than a pile of complaints. And a measured answer to the question a stakeholder always asks: why did you not check everything?
LiveWhat each lens contributed, and what it duplicated4 min▶
Measured on the bench, adding one lens at a time, counting only findings no earlier lens had already produced:
| Lens added | Running total | New |
|---|---|---|
| NN/g's 10 usability heuristics | 5 | 5 |
| Few's 13 dashboard pitfalls | 10 | 5 |
| Tufte on graphical integrity | 11 | 1 |
| IBCS SUCCESS rules | 11 | 0 |
| Microsoft's 18 human-AI guidelines | 14 | 3 |
| Google PAIR's guidebook | 14 | 0 |
| Data-as-a-product, FAIR, DAMA | 17 | 3 |
| DataOps and the golden signals | 19 | 2 |
| GQM and HEART | 20 | 1 |
Two lenses contributed nothing on this product. Five of the nine contributed three findings or fewer. And the single most productive lens after the usability set was a white paper from 2006 about dashboards.
A team that adopts three reporting standards at once believes it has tripled coverage. What it has actually built is a review that takes three times as long and produces a report with three citations per finding. The gain from adding IBCS on top of Tufte and Few here, measured, is zero.
Self-studyWhy overlap is structural, not accidental4 min▶
The sets overlap because they were written by people solving the same problem from different chairs. "Do not let colour be the only carrier of meaning" appears as an accessibility concern in the usability literature, as pitfall 12 in Few, as part of Unify in IBCS, and as a perceptual point in the Gestalt tradition. Four independent derivations, one defect.
This has a practical consequence for how you cite a finding. Naming every set that catches it looks thorough and reads as padding. Name the one whose framing makes the fix most obvious to the person who has to do the work, and keep the rest in your notes for the argument you might have later.
An instrument is assembled per layer, not per standard
7 min liveThe reason a usability-only review finds a quarter of what is wrong with a data product is not that its principles are weak. It is that a data product has layers a screen does not have, and no interface principle was ever written to ask who owns a dataset. Assemble by layer and the gaps become visible.
LiveThe four questions that select a check5 min▶
A check earns its place in the instrument only if all four answers hold:
- Is it detectable? Can a reviewer, looking at the product, tell whether it holds? "Be elegant" is not a check. "No measure is shown without its unit" is.
- Is it new? Does any check already in the instrument catch the same defect? If yes, keep the clearer wording and drop the other.
- Does it carry consequence? If it fails, can you name what goes wrong for a reader or a decision? A check with no consequence produces a finding nobody funds.
- Is it sourced? Can you name the set it came from, and the tier you read it at? This is what turns a review meeting from a taste argument into a technical one.
The second question is the one that gets skipped, and it is why instruments bloat. Every new standard a team adopts arrives with its own complete list, and nobody does the deduplication, so the review grows by seven items and its coverage grows by nothing.
Self-studySourcing a finding when the source is a book4 min▶
Four of the most useful sets in the canon were never published as numbered lists by their authors: Norman's design principles, Tufte's principles, the Gestalt laws, and the Kimball and Inmon tenets. Every count you see for these was assigned by a secondary source.
That does not make them unusable. It changes how you cite them. Rather than "this violates Tufte's principle 3", write the concept and the consequence: "the bars start at 40, so the change reads about six and a half times larger than it is, which is the lie factor Tufte named in 1983". The concept is genuinely his, is defensible, and does not depend on a numbering nobody published.
The reverse discipline matters just as much. Where a set genuinely is numbered and primary-verified, quote it precisely. "Microsoft's guideline 2, make clear how well the system can do what it can do" is checkable by anyone in the room, and that precision is what the citation is for.
Twenty findings are not a plan until they are ranked
6 min liveA review that hands over twenty equal findings hands over nothing. The team fixes the three cheapest, and the truncated axis is still lying to the board next quarter. Severity is the step that turns a list into an order, and it takes three factors rather than one.
Four of Tideline's findings are loaded above with a defensible starting rating. Move the sliders and watch the fix order change. Two things are worth doing deliberately:
- Take the four decimal places on MRR and raise its impact to 4. Its severity goes to 3.0 and it jumps above the gauge. That is what happens when a cosmetic finding gets argued up by whoever is annoyed by it, and it is exactly the failure mode severity ratings exist to make visible.
- Now drop its persistence to 0, on the grounds that readers learn to ignore the extra digits within a week. It falls back down. Persistence is the factor that separates cosmetic from structural, and it is the one most reviews never record.
Emails in the clear and the truncated axis both land at 3.7 out of the box, and they are the two findings on Tideline that a reasonable person would escalate. One is a privacy exposure, the other is a number that misleads every reader every time. Neither is a design opinion, and neither can be argued down by whoever owns the dashboard.
Run the full instrument, then break it on purpose
★ 20 min · everyone buildsThe bench holds all nine lenses and the exhaustive pass. Every check reads the rendered Tideline screen below it, so the numbers move because the screen genuinely has those properties.
Press Turn on all nine lenses. Twenty of twenty, zero false flags, 100 percent precision, 66 modelled minutes. Read the finding list top to bottom once, and notice that six of the twenty are things no interface principle asks about.
Look at the three findings that carry a computed value: the retention axis at a lie factor of 6.5 to 1, 13 distinct fills on one screen, and 64px below the fold. Each is measured off the rendered screen rather than asserted.
Now turn off every lens except NN/g. Five findings. This is the review a well-trained interface reviewer produces on this product, working carefully and missing three quarters of it. Say out loud which of the fifteen missing findings would have hurt your organisation most.
Turn the nine back on, then tick Check all 215 principles. The catch count stays at twenty, because there was nothing left to catch. Twelve new findings appear.
Read those twelve. Every one is true of the screen: it uses more than one font weight, panels have rounded corners, a number is set in bold, a chart uses circular marks. None is a defect. Precision falls from 100 percent to 63 percent, and the modelled review cost goes from 66 minutes to 144.
That is the argument for the discard, and it is worth being precise about the harm. An over-large instrument does not produce nonsense, which would be easy to reject. It produces irrelevance dressed as rigour, and the team on the receiving end cannot tell which of your 32 findings are the two that matter.
Untick it. Now write the report: the two findings at severity 3.5 or above, each with one named source and one sentence of consequence, then the rest in a table below. One page. If it runs to three, cut findings rather than words.
The review-cost figure is the one modelled number on the bench, built from a flat per-lens and per-finding estimate. Everything else, including all twenty catches and all twelve false flags, is computed from the rendered screen. Say which is which when you present numbers from a tool, always.
The report is the product, and it has one job
4 min liveA review's output is not a finding list. It is a decision by somebody else to spend engineering time. Everything about the format follows from that.
Self-studyWhere to take the instrument next4 min▶
- Run it with three people, separately. Heuristic evaluation is specified as independent-then-pooled for a reason: a single reviewer finds a modest share of what is present, and different reviewers notice different things. On a data product the expertise is genuinely split, so the person who spots the axis is rarely the person who asks who owns the table.
- Re-run it on a schedule, not on a redesign. Most of the twenty findings on Tideline arrived one at a time, each defensible on the day. Nobody ever shipped that screen; it accumulated.
- Add checks from your own incidents. Every outage that reached a reader is a check the instrument was missing. That is the same derivation Nielsen ran in 1994, on your own data.
- Keep it small. When you add a check, ask what it replaces. An instrument that only grows becomes the 60-item Confluence checklist that nobody has run since the day it was written.
The one that counts
60 min- Run your assembled instrument over your own most-used data product. Record every finding with its source and its severity, using all three factors.
- Write the one-page report: the findings at 3.5 or above first, each with a named source and a one-sentence consequence, then everything else in a table.
- Take it to whoever owns that product and ask for exactly two fixes. Not twenty.
- Afterwards, note which findings they argued with. Those arguments tell you which checks in your instrument are not yet detectable enough, and that is your next revision.
Three questions
Nine lenses holding well over a hundred principles between them produce twenty findings on Tideline. What does that tell you about assembling an instrument?
Four independent traditions each derived "colour must not be the only carrier of meaning". The sets are not wrong and adding more does not help much; the work is deduplication, which is the step teams skip when they adopt a new standard.
Which severity factor separates a cosmetic finding from a structural one, and is most often left out?
Persistence is the third factor and the one most reviews record nowhere. Extra decimal places get ignored within a week; a truncated axis misleads every reader every time, no matter how familiar they are with it.
The exhaustive pass keeps the catch count at 20 and adds twelve findings that are all true of the screen. Why is that a problem rather than a bonus?
All twelve are genuinely true and none is a defect, which is exactly what makes them dangerous. Nonsense is easy to reject; irrelevance dressed as rigour buries the two findings that needed funding, and it costs 144 minutes instead of 66.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map.
Session 9 cheat sheet · pin this
The instrument is finished and it is small on purpose. Take it to your own product this week, ask for exactly two fixes, and note which findings got argued with - those arguments are the next revision. Then add a check for every incident that reaches a reader, which is the same derivation Nielsen ran in 1994, on your own data.