AI on people data, and what the law forbids

The decision (GDPR Article 22). It concerns decisions based solely on automated processing that produce legal or similarly significant effects on a person. A score that ranks applicants and a person who then decides is a different situation from a score that rejects applicants with nobody looking, and the difference has to be real: a reviewer who approves every score unread is automation with a signature. The data (GDPR Article 9). Special categories of personal data, health and trade union membership among them, carry stricter conditions for processing at all. The use (EU AI Act Annex III point 4). Systems used for recruitment and selection, for decisions on promotion and termination, for task allocation and for monitoring are listed as high-risk, which brings obligations that attach to the use, whatever the model. The audit (NYC Local Law 144). Enforced from 5 July 2023, it requires an annual independent bias audit of automated employment decision tools, a published summary of it, and impact ratios by sex and by race/ethnicity.

Read them as four gates, not one. An attrition model that only informs a manager may sit outside Article 22 and still be a monitoring use under Annex III. A screening tool that never sees a special category can still owe a bias audit. None of the four is the whole law of anywhere, and the jurisdiction you work in may add its own; the checklist leaves a line for it.

The four rules are named so that a people analyst recognises when a request has crossed into them. Each has exceptions, conditions and safeguards that decide real cases, and those belong with the courses that teach the law. The GDPR course covers Articles 22 and 9 in depth; the PDPA course is the Singapore counterpart; the AI Governance and EU AI Act course covers what a high-risk classification requires; the Data Governance course covers who owns a column and who may grant access to it.

Local Law 144's audit reports impact ratios, and an impact ratio is a selection rate divided by the highest group's selection rate. The four-fifths rule of thumb (29 CFR 1607.4(D)) and a working screening simulator live in the AI + HR course; deep-dive session 4 computed the ratio once on a constructed funnel. The habit carried here is narrower: whenever a ratio is printed, the two rates it divides are printed beside it.

Delete the sex column and a model trained on past hiring can still separate the groups, through anything that differs between them in the pile: a word on a resume, a society, a school, a gap in the dates, a part-time spell. The reported case is the cleanest illustration. Reuters reported on 10 October 2018 that Amazon had scrapped an experimental recruiting tool that penalised resumes containing the word "women's". The penalised item, as reported, was a word, not a demographic field. The report is secondary here, confirmed through a reproduction, and it is quoted as reported, not as a finding about any system now in use.

The consequence for practice is a split that looks odd at first. The model may not see the protected attribute, and the audit must. A proxy can only be found by someone who holds the attribute and checks whether the features predict it, and whether the model's scores differ by it. So the attribute is kept, held apart from the modelling table by someone other than the model builder, and used for exactly that. Local Law 144's impact ratios by sex and race/ethnicity cannot be computed any other way.

The usual wording of Goodhart's law is Marilyn Strathern's, from 1997: when a measure becomes a target, it ceases to be a good measure. People data is where it bites hardest, because the people being measured can see the measure. A time-to-fill target is met by filling the seat with whoever is available. An attrition target is met by moving an exit date across a quarter boundary. A model's risk score, once managers are judged on the number of high-risk people in their team, starts to measure how managers describe their teams.

The check is one question on the checklist: is the score, or the metric the model was trained to predict, also a target somebody is rewarded on? If yes, the score will drift from the thing it was built to measure, and the drift will look like improvement. Foundations session 6 asks the same question of every dashboard row.

Two findings, both primary, from the full text. Introducing the test raised the completed tenure of hires by about 25 percent (0.23 log points). And among hires made once the test existed, those made by overriding it left sooner. On that outcome, managers' discretion was on average worse than the test.

What it does not license is "remove the humans". Tenure is one outcome, chosen by the study; it is not fairness, and a test can raise tenure while selecting unevenly across groups, which is what an impact ratio exists to catch. And Article 22 restricts decisions made solely by automated processing whatever their tenure. The two sources are not in conflict; they answer different questions. The practical synthesis is modest: keep the human, make the override visible, and record every one so that the organisation can ask of its own data what the paper asked of the firms it studied.

An override log has five columns: who was recommended, who was chosen, who chose, the reason in a sentence, and the date. That is enough to compare, a year later, the outcomes of hires made as recommended and hires made against the recommendation, and to compute both groups' selection rates by the audit attributes. It is also enough to see a reviewer who never overrides anything, which is the signature of a human in the loop in name only.

The audit is then a habit rather than an event: the same checklist re-run each year, the same may-not-see list re-applied whenever a column is added to the export, the same impact ratios computed with both rates printed. Local Law 144 makes the annual cadence a legal requirement for tools in its scope; for everything else it is simply the cadence at which a people model drifts.