How this course works
Nine sessions, 45 minutes each, one artifact: a review instrument you can point at any data product. It fills in layer by layer - the canon and the method tonight, the decision the product serves in session 2, the data contract in session 3, the platform in session 4, the operating promise in session 5, the surface in sessions 6 and 7, the model in session 8, and the finished review in session 9. The thing under review throughout is Tideline, a subscription-analytics product with twenty real defects in it.
★ What you walk out with tonight. How a heuristic set is actually derived, and therefore how to tell a real one from a listicle. A severity scale that turns twenty findings into a fix order. And a measured, uncomfortable number: what a purely usability-trained reviewer misses on a data product.
LiveWhy heuristics rather than a checklist3 min▶
A checklist enumerates the things you already know go wrong. A heuristic is broader on purpose: it is a rule of thumb general enough to catch problems nobody has written down yet, and vague enough that two reviewers will sometimes disagree about whether it was violated. That vagueness is the feature. A specific check catches a specific defect; a heuristic catches a class.
The cost of that generality is that heuristics do not scale by addition. Adding the 40th principle to a review instrument does not make it 4 percent better, it makes the review longer, the report less readable, and the findings less likely to be acted on. Session 9 measures that cost precisely.
Tideline's team has a 60-item dashboard checklist in Confluence. It has been run twice in two years, both times by the person who wrote it. The screen still shows customer email addresses in the clear, because that was not on a dashboard checklist - it was somebody else's concern, on a list nobody read.
Self-studyThe vocabulary for the whole arc4 min▶
- Heuristic evaluation - a small number of reviewers inspect an interface against a set of principles independently, then their findings are pooled. Nielsen and Molich introduced the method in 1990.
- Lens - one named principle set, applied on its own so you can see what it does and does not catch. Session 9's bench toggles nine of them.
- Severity - a rating combining how often a problem hits, how badly, and whether users learn to work around it. It is what converts a finding list into a fix order.
- Precision - of everything the review flagged, the share that was really a defect. The number that collapses when an instrument gets too big.
- Verification tier - whether a principle set was read from its primary source, from a close secondary, or from a book that never numbered its principles at all. Four of the most-cited sets fall in the last group.
Ten heuristics were not chosen, they were extracted
7 min liveThe ten usability heuristics are quoted everywhere and derived almost nowhere. They came out of an exercise worth understanding, because it is the same exercise that tells you whether any other set deserves your trust: Nielsen took a large body of real, observed usability problems and asked which small group of rules accounted for most of them.
LiveThe ten, in their current wording4 min▶
Published in 1994 and revised in 2020, still the most-applied set in the field. The revision changed wording, not count.
| # | Heuristic | What it asks of a data product |
|---|---|---|
| 1 | Visibility of system status | Can a reader tell how fresh this is, and whether it loaded fully? |
| 2 | Match between system and the real world | Do the measure names match what the business calls them? |
| 3 | User control and freedom | Can a filter be undone without reloading the world? |
| 4 | Consistency and standards | Does green mean the same thing on every panel? |
| 5 | Error prevention | Can someone compare two incompatible measures by accident? |
| 6 | Recognition rather than recall | Are units and definitions on the screen, or in someone's head? |
| 7 | Flexibility and efficiency of use | Is there a fast path for the daily reader and a guided one for the rest? |
| 8 | Aesthetic and minimalist design | Is every element on the screen earning its space? |
| 9 | Help users recognize, diagnose and recover from errors | When a query fails, does the message say what to do? |
| 10 | Help and documentation | Is the metric definition reachable from the number? |
Heuristic 1 is the one that fires on Tideline immediately. There is no timestamp anywhere on the screen. Every reader in the Monday meeting assumes the number is current, and roughly one Monday a month it is not, because a load failed quietly on Sunday night.
LiveSeverity: three factors, one number5 min▶
A finding without a severity is a complaint. Nielsen rates each problem on three factors and combines them:
- Frequency - how often does a reader hit it? A defect on the panel everyone opens first outranks one three clicks deep.
- Impact - when it hits, how much damage? Reading a truncated axis and drawing the wrong conclusion is expensive. Reading four decimal places is merely annoying.
- Persistence - do people learn to work around it, or does it get them every single time? A one-time confusion is cheaper than a permanent trap.
The combined rating runs 0 to 4, where 0 means "not a problem at all" and 4 means "fix before anything else ships". Session 9 puts three sliders on four of Tideline's findings, and the fix order genuinely reorders as you move them.
Persistence is the factor most reviews drop, and it is the one that separates a cosmetic issue from a structural one. A misleading axis never stops misleading, no matter how many times a reader sees it.
Self-studyHow many reviewers, and why not one4 min▶
Heuristic evaluation is specified as an independent-then-pooled activity, and the reason is that a single reviewer finds a surprisingly small share of the problems present. Different reviewers notice different things, so their findings overlap far less than people expect. The standard advice is a handful of evaluators inspecting separately, with the aggregation happening afterwards.
This matters for a data product more than for a website, because the relevant expertise is split across people. The person who spots the truncated axis is usually not the person who notices there is no owner on the dataset, and neither of them is the person who asks what decision the screen is for. Session 9's bench models that split as nine lenses; in a real review, some of those lenses are colleagues.
Thirty-two sets, seven families, and four that do not exist
7 min liveThe principles that bear on a data product were written by people who mostly were not talking to each other: interface researchers, dashboard designers, distributed-systems engineers, metrologists, and machine-learning teams. Knowing which family a principle came from tells you what it is blind to.
LiveThe four sets that everyone cites and nobody wrote5 min▶
Some of the most-quoted "N principles of X" have no such list at their primary source. Quoting a count for these is citing something that was never published:
| Commonly cited as | What the primary source actually contains |
|---|---|
| The principles of the Lakehouse | The 2021 CIDR paper describes an architecture, with themes such as open formats and transactions on object storage. It enumerates no principles. |
| Decision intelligence principles | Kozyrkov defines an interdisciplinary field and separates automatable from human decisions. There is no numbered framework. |
| The North Star framework's principles | One north-star metric plus input metrics. A metric-selection method, not a principle set. |
| The AI hierarchy of needs, with N layers | Rogati's 2017 pyramid is real and its argument holds, but published renderings differ on whether it has four layers or six. |
Three more sets are real but access-gated, so their exact wording travels through secondary sources: Dehghani's eight data-product characteristics live in a paid book, DAMA UK's six data-quality dimensions in a members-only whitepaper, and ISO 9241-110's seven dialogue principles behind an ISO paywall. Cite them by source and say which tier you read.
Self-studyThe full inventory, by verification tier5 min▶
Read from the primary source. NN/g's 10 heuristics. Shneiderman's 8 golden rules. Tognazzini's 19 interaction principles. Few's 13 dashboard pitfalls. Microsoft's 18 human-AI guidelines. Google PAIR's 6 chapters. NN/g's dashboard guidance. Data Mesh's 4 principles. FAIR's 4 principles and 15 sub-points. The DataOps manifesto's 18. The Agile manifesto's 12. AWS Well-Architected's 6 pillars. Google SRE's 4 golden signals. The 12-Factor App. The Reactive manifesto's 4 traits. GQM. HEART's 5. CAP and PACELC.
Real, but read through a close secondary. Dehghani's 8 characteristics. DAMA UK's 6 dimensions. AWS's Analytics Lens design principles. ISO 9241-110's 7.
Book-only, never numbered by their author. Norman's design principles. Tufte's principles. Gestalt's laws of perception. Kimball and Inmon's tenets. All four are genuinely foundational and none of them ships as a canonical numbered list, which is why every secondary source gives a slightly different count.
The point of tracking the tier is not pedantry. A principle you can quote exactly can be argued about precisely; a principle you absorbed from a blog post that misremembered a book cannot.
A usability lens finds a quarter of what is wrong
6 min liveTideline has twenty real defects. Not twenty opinions - twenty properties of the rendered screen that a program can detect, from a genuinely truncated axis to email addresses genuinely printed in the clear. Turning on one principle set at a time shows exactly how much of that each set is capable of seeing.
LiveTwo whole standards that add nothing here4 min▶
IBCS is a seven-rule reporting standard used across European enterprise finance. Google PAIR is a six-chapter guidebook from the team that coined much of the human-centred AI vocabulary. On Tideline, both add zero new findings, because everything they can catch has already been caught by a set that was switched on earlier.
That is not a criticism of either. It is the single most useful property of a review instrument, and the reason instruments must be assembled rather than accumulated: overlap is the norm, not the exception. Nine sets holding well over a hundred principles between them resolve to twenty distinct findings on one screen.
A team that adopts three dashboard standards at once usually believes it has tripled its coverage. What it has actually done is triple its review time and produce a report with three names attached to each finding. The measured gain from adding IBCS on top of Tufte and Few here is nothing at all.
Self-studyWhat the last three lenses see that the others cannot4 min▶
Six of Tideline's twenty defects are invisible to every interface-oriented set, because they are not properties of the interface at all:
- No owner named for the screen or its data. A number nobody owns is a number nobody fixes.
- No lineage for any measure, so when two screens disagree there is no way to find out why.
- Email addresses shown in the clear, on a screen anyone with the link can open.
- No freshness promise, which means the pipeline cannot be late, because nothing was ever promised.
- No evidence the data was tested before it landed, so the first person to notice bad data is the reader.
- No named decision the screen exists to support, which means there is no test for whether it works.
Sessions 2 through 5 are about those six. They are the reason this course exists rather than pointing you at a usability checklist.
Meet Tideline, and switch on one lens
★ 17 min · everyone buildsThe bench below renders Tideline into this page and runs real checks against it. Work through the steps with it open. The full instrument arrives in session 9; tonight you only need to feel the gap between one lens and nine.
Start with NN/g only, which is where the bench opens. Read the five findings. Every one of them is a real property of the screen above: no freshness stamp, thirteen distinct fill colours, red and green carrying meaning with no second cue, no panel highlighted, and no way to correct the AI claim.
Now add Few's 13 pitfalls. The count goes to 10. Notice which panels light up: the four-decimal MRR, the gauge spending a quarter of a panel on one number, and the signup count standing in for a rate.
Add Tufte. One new finding, and it is the most expensive one on the screen: the retention bars start at 40 rather than zero, which the bench measures as a lie factor of 6.5 to 1. The change looks six and a half times bigger than it is. That number is computed from the drawn bar heights, not looked up.
Add IBCS. The count does not move. Sit with that for a moment before moving on.
Press Turn on all nine lenses. Twenty of twenty, no false flags, 66 modelled minutes of review. This is the instrument you will have built by session 9.
Now the trap. Tick Check all 215 principles. The catch count does not improve, because there was nothing left to catch. What changes is that twelve new findings appear which are true statements about the screen and are not defects at all: it uses more than one font weight, it has rounded corners, a number is bold. Precision falls from 100 percent to 63 percent and the modelled review time goes from 66 minutes to 144.
Untick it. Then ask the question the rest of the course answers: if the exhaustive pass is worse than the assembled one, which principles belong in the instrument, and how would you defend each choice to the team whose product you just reviewed?
The false flags are worth reading carefully rather than dismissing. Each one is genuinely true of the screen. That is precisely why an over-large instrument is dangerous: it does not produce nonsense, it produces irrelevance, and irrelevance is much harder to argue with in a review meeting.
Before session 2
20 min- Open the dashboard or data product your team looks at most often. Run NN/g's ten against it, alone, and write down every finding. Do not fix anything yet.
- Give each finding a severity from 0 to 4 using the three factors: frequency, impact, persistence. Note which findings changed rank once you added persistence.
- Then answer the two questions no usability heuristic asks: who owns this data, and what decision does this screen exist to support. If either answer takes more than a minute to find, that is your session 2 material.
- Bring the list. Session 9 reviews Tideline with the finished instrument, and it is much more interesting if you have your own product's findings sitting next to it.
Three questions
Why were the ten usability heuristics reduced from a much larger body of observed problems rather than simply listed?
The reduction was an analysis of which few rules accounted for the most real problems. Breadth is what lets a heuristic catch a class of defect rather than one instance, and the discarded tail was specific and rare, not wrong.
On Tideline, adding the IBCS SUCCESS rules on top of Tufte and Few changes the catch count from 11 to what?
It stays at 11. Overlap between principle sets is the norm, and it is why an instrument has to be assembled deliberately rather than grown by adding every standard someone recommends.
Ticking "check all 215 principles" leaves the catch count at 20. What does it actually cost?
Twelve additional findings appear that are true of the screen and are not defects, so precision falls from 100 percent to 63 and the modelled cost goes from 66 minutes to 144. The damage from an over-large instrument is irrelevance, not error.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map.
Session 1 cheat sheet · pin this
Six of Tideline's twenty defects are invisible to every interface principle, and the first of them is the one that makes the rest hard to argue about: nobody can say what decision the screen exists to support. Session 2 fixes that with GQM and HEART, and asks the question that retires more dashboards than any redesign.