Surveys as evidence

Every survey score is a flow over a stock, like every rate in Foundations session 1. The stock is the population invited, on a date: Apex's 1,022 on the 1 January 2026 payroll. The flow is the people who answered in the survey window: 600, a count chosen for the constructed survey. The response rate is 600 over 1,022, 58.7 percent, and it is the base of every number the survey produces. A score reported without it is a numerator wearing a decimal point.

The 422 are not missing at random. The people who skip a survey can be the people it most needs to hear: the ones who have stopped believing anything will change, the ones who are already interviewing elsewhere, the ones too overloaded to spend ten minutes on a form. It can also run the other way, a team so content it never bothers. Nothing in the survey can tell you which. What you can do is compare the 600 to the 1,022 on everything HR already knows about everyone: department, entity, tenure, grade. If the respondents are older in tenure than the invited, or one department answered at half the rate of the rest, the score leans towards the groups that answered, and the sentence that reports it has to say so.

A high response rate is not immunity. At 58.7 percent the score describes a majority of Apex; it still describes the majority that chose to speak. Deep-dive session 7 plants exactly this bias in the constructed survey on purpose, so that the gap between the respondents' score and everyone's score can be measured, which a real survey never allows.

A single response rate for the company hides the rate for each group, and the groups are where the score will be read. Compute the rate per department from the HR table, which has every invited person in it, and the survey tool, which has every respondent. The department with the lowest rate has the score most shaped by its volunteers, and it is usually the department whose score leadership is most curious about.

Two habits follow. First, print the response rate beside every group's score, not once in a footnote: "Sales: score X, answered Y percent" is a different claim from "Sales: score X". Second, treat a change in response rate as a change in the population. If a department's score rose from one year to the next while its response rate fell, the score may have risen because the unhappy stopped answering. Both years were computed correctly; the two years describe different people. The X and Y are yours to fill from your own survey; no Apex group rate is claimed.

Team A: a fifth answered 3 and four fifths answered 4. Nobody is unhappy and nobody is delighted. Team B: a fifth answered 1, a twentieth each answered 2 and 3, 15 percent answered 4 and 55 percent answered 5. Both average exactly 3.8, and the two teams need opposite conversations. Team A is a mild, stable yes. Team B is a majority who are very positive and a fifth who chose the lowest answer on the scale, and a fifth of a team at the bottom of the scale is a conversation the mean will never prompt.

Three numbers carry what the mean loses: the favourable share (4 or 5), the unfavourable share (1 or 2), and the count behind them. Team A is 80 favourable and 0 unfavourable; team B is 70 and 25. A dashboard that shows those two shares for every group, with the response count beside them, will never show two identical teams that are not. Constructed teams; every survey tool can produce the same two shares from its own export.

Whether a mean of five-point answers is meaningful at all is argued, not settled. One side treats the labels as ordered categories only, where a mean is not defined in principle; the other side points out that averages of several such items behave well in practice and are used everywhere. You do not need to settle it to act on it: show the distribution beside the mean, and the argument stops mattering for the decision in front of you.

An engagement score is usually the average of several items, twelve in the constructed survey, on the claim that they all measure one underlying thing. That claim can be tested. Cronbach 1951, "Coefficient alpha and the internal structure of tests", Psychometrika 16(3) 297-334, gave the number that tests it: coefficient alpha, which is high when the items move together across people and low when they do not. If the items measure one thing, a person who scores high on one tends to score high on the rest.

The constructed survey has one item built on purpose to measure almost nothing: item 9. Averaged into the score, it adds noise to everyone's number by an amount unrelated to anything anybody did. The question to ask of any scale is "what is the reliability of this scale, and which item drags it down?" Deep-dive session 7 writes alpha from the 1951 formula, finds item 9 with it, and shows what dropping an item does to every score, which is a decision about the questionnaire rather than a statistical one.

Alpha says whether items hang together. It does not say they measure what their label claims, and it does not say the score causes or predicts anything. A reliable scale can be reliably measuring the wrong thing.

The threshold is a count, set once and applied before any score is shown. Below it, a group's score is not displayed, because a mean over a handful of people tells the handful's manager who said what. The number itself is a policy choice with no single authoritative source; the constructed survey uses 8, the same stated choice deep-dive session 7 applies in code. Whatever the number, write it down and apply it to every group the tool can produce.

The failure is the subtraction. In the figure, HR's 40 constructed respondents split by entity into 21, 12 and 7. The 7 are below 8 and are held back. But the HR total of 40 is shown, and so are the 21 and the 12, and from those three means anybody can recover the mean of the 7: forty times the total mean, minus the two shown groups' sums, divided by seven. The fix is to hold back a second cell so that the hidden remainder is itself above the threshold: hide the 12 as well and the hidden pair is 19.

The same logic applies to cuts nobody planned. Department by tenure band by sex is a small group even when every margin is large, and a dashboard filter that lets anyone combine cuts freely will produce it on request. The threshold has to apply to what the tool can show, not to what the report designer intended.

The most quoted engagement number in the field comes from Gallup's State of the Global Workplace. Gallup reported 23 percent of employees engaged in its 2024 edition, on 2023 data; its 2025 edition reported 21 percent, and its 2026 edition 20. These are reported figures: they come from Gallup's publications and are not recomputed here.

Two habits follow. Name the edition every time. "Gallup says 23 percent" on a slide made in 2024 was current; the same slide reused in 2026 is two editions stale, and the figure has moved three points since. Do not hold it beside your own score as a comparison. Gallup's figure is the share of people its own items and its own classification call "engaged"; your survey's favourable share on your own items is a different measurement, and the two can be quoted on the same page only as two different things.

"People leave managers, not companies." It is usually quoted as a Gallup finding and usually attached to a survey result about managers. It traces to a book, Buckingham and Coffman, First, Break All the Rules (1999), not to a measured Gallup statistic. It is not stated as fact on any page here. Apex's constructed exit reasons illustrate why it is hard to measure at all: 23 of the 210 leavers carry the category Management/Culture, but that is whatever the person filling the leaving form chose from a dropdown, and a category is a choice, as Foundations session 2 showed with "regretted".

"Replacing someone costs 1.5 to 2 times their salary." It usually arrives beside an attrition figure to make it expensive. SHRM's own range is 50 to 200 percent of salary, which is a spread of four to one, and the 1.5 to 2 times version is attributed to an analyst rather than measured. It is not stated as fact on any page here either.

Neither sentence is necessarily false. Each is unmeasured, and a survey result presented beside an unmeasured sentence borrows the sentence's confidence. The test is one question: what was counted, by whom, and in which year? Gallup's 23 has an answer; the two sentences do not.