When ML Bias Becomes a Security Failure

In December 2019 NIST published FRVT Part 3: Demographic Effects, testing 189 face recognition algorithms from 99 developers against four photo collections holding 18.27 million images of 8.49 million people.

Its main result was not the one that travelled. False positive differentials, NIST wrote, are much larger than those related to false negatives and exist broadly across many of the algorithms tested. False positive rates varied between demographic groups by one or two orders of magnitude. False negative effects varied by factors usually much less than three. The follow-up report, FRVT Part 8 in July 2022, put the spread in within-group false match rates at a factor of 720 across demographic cells, at one fixed threshold on one algorithm. Its footnote gives the two ends: Polish men aged 35 to 50 at a false match rate of 1 in 26,000, and Nigerian women aged 60 and over at 1 in 35.

Then NIST said the part that should have ended up in every security architecture review and did not. Part 8 names the applications at risk: automated border control, phone access, payment authorisation and non-repudiation of a dispensed drug. A high false positive rate in a demographic group leaves security holes in all four. The legitimate enrollee is the one vulnerable to impostors.

A demographic differential in false match rate is a demographic differential in exposure to impostors. That is a property of an authentication control, measured and published nearly seven years ago, and the security profession has largely left it to the ethics literature.

The popular story is about a different task

The account most people carry comes from Buolamwini and Gebru’s Gender Shades in 2018, which audited three commercial gender classification systems and found error rates reaching 34.7% for darker-skinned women against a maximum of 0.8% for lighter-skinned men. It is excellent work and it measures a different task. NIST draws the same line itself, noting that face recognition deals with identity and uses different, specialised machinery from sex classification.

Three tasks get run together. Classification asks what category a face belongs to. Verification asks whether a face matches the one enrolled record it claims. Identification asks whether it matches anyone in a gallery. Classification returns a category label, so only verification and identification have a false match rate to measure.

Within those two, failing to match someone to their own record is not the same as matching them to somebody else’s. In one-to-one verification the first is a false non-match and it is friction: the gate does not open, you try again, a human checks. The second is a false match, and it is the error that admits the wrong person. Both matter, and the second carries the larger demographic spread.

What a security architect should read into it

Take NIST’s own two ends. At a single fixed threshold on one algorithm, Polish men aged 35 to 50 face a false match rate of 1 in 26,000 and Nigerian women aged 60 and over face 1 in 35. Same system, same threshold, same measurement run. One aggregate figure covering both describes neither.

Three consequences follow.

An aggregate false match rate conceals subgroup variation, and it does not establish that every group meets the same bound. Most control documentation states one aggregate figure, which cannot describe a range running from 1 in 26,000 to 1 in 35.

Higher subgroup false match rates mean unequal exposure to impostors rather than a ready-made exploit. Zero-effort impostor attempts, where someone presents their own face and hopes, are the baseline threat, and the false match rate is their success rate under test conditions. NIST is careful here and anyone repeating it should be too: in a one-to-one setting it says arranging a particular false match stays difficult even where the rate runs as high as 1 in 50, because the impostor has to claim a specific enrolled identity. The exposure is statistical. Targeted impersonation, spoofing and presentation attacks each need their own test. SP 800-63B-4 states that a false match rate gives insufficient confidence on its own, because it does not account for active impersonation.

And a single threshold forces a security-usability trade. NIST’s own remedy is to set the threshold globally to hit a target rate in the worst-affected group, which lowers false matches everywhere and raises false non-matches generally. That is a real engineering decision whose cost may itself fall unevenly, and it is currently made by default rather than deliberately.

What the deployed failures actually look like

One bridge first, because the metrics change. Everything above is one-to-one verification, where the question is whether a face matches the single record it claims. The arrest cases are one-to-many identification, where a probe is searched against a gallery. NIST relates the two in an annex: most identification algorithms work by computing N one-to-one scores, so the false positive identification rate scales roughly with gallery size multiplied by the false match rate, though an important minority do not work that way. Different measurements, related failure, and in both it is the false positive that puts the wrong person into a decision path.

The consequences are documented, named and dated, and the mechanism inside them is a systems lesson rather than a model one.

Robert Williams was arrested in Detroit in January 2020, the first publicly reported wrongful arrest attributed to face recognition, after software matched his driving licence photograph to security video from a 2018 watch store theft. He was detained around thirty hours. The case settled for $300,000 and a set of policy reforms, including a prohibition on arrests based solely on a face recognition result and on arrests built from a photo lineup that a face recognition search produced.

He is not alone. Nijeer Parks was arrested in Woodbridge, New Jersey in February 2019, and police proceeded without waiting for the fingerprint and DNA analysis that would have pointed away from him. Michael Oliver was arrested in Detroit in July 2019; the suspect in the footage did not have his full tattoo sleeves. Porcha Woodruff was arrested in Detroit in February 2023 while visibly pregnant, a detail the carjacking victim never mentioned despite having spent hours with the woman police were looking for. Trevis Williams was arrested in New York in April 2025, eight inches taller and seventy pounds heavier than the person police were looking for. The ACLU’s tracked list stood at fourteen named people in April 2026.

Now the part that belongs in a security article. In at least seven of them, by the ACLU’s count, the corroboration step designed to catch the error was contaminated by the error. Police built a photo array around the face recognition result plus five filler images, showed it to a witness, and the witness selected the algorithm’s candidate rather than concluding the suspect was absent. The independent check was not independent, because its contents were chosen by the thing it was supposed to check.

That is the same failure this site keeps describing in other contexts. A model output entered a decision path whose review stage was downstream of the output instead of beside it. Whether the model is scoring faces or scoring transactions, a check that consumes the model’s answer before forming its own is not a check.

It is a data property, not a physics property

One detail in the NIST results settles a question people argue about. Among US-developed algorithms, false positive rates were elevated for Asian, African American and native groups, with the American Indian rate highest. Some algorithms developed in Asian countries did not show the same dramatic differential for Asian faces.

So the differential moves with who built the system. NIST did not investigate why, and offers training data as one plausible contributor. Where the effect is absent, NIST says, a high false match rate in that population is demonstrably not inevitable. Algorithm-dependent is not immutable, and a defect that can be fixed is something a security team can own.

Why this article moved category

Securing.AI keeps four terms apart on purpose. Secure AI, safe AI, responsible AI and trustworthy AI are different problems with different owners, and most of what gets filed as ML bias belongs to responsible AI: hiring models that reproduce historical patterns, credit models that proxy for postcode, language models that carry the associations of their corpora. Those are real and this site is not the right home for them.

When a system’s job is to decide whether you are who you say you are, a demographic differential in its error rate is a differential in the strength of an authentication control. That is secure AI, it belongs here, and filing it under ethics is a large part of why security teams have not treated the NIST numbers as their problem.

The earlier version of this article did the filing wrong, which is the honest reason it needed rebuilding rather than updating.

What the fairness toolkits do and do not do

AI Fairness 360, now under the Linux Foundation, and Fairlearn both compute group metrics across protected attributes, and they are useful for what they do. Google’s What-If Tool has seen only incidental maintenance in recent years and is not promoted in Google’s current tooling.

The limitation to understand is that these compute disaggregated metrics over the datasets, labels and sensitive features you supply. They do not tell you the false match rate of your deployed biometric system at your operating threshold across the population that actually presents to it. That number comes from testing your system, on representative data, at the threshold you run.

Somebody has now asked. NIST SP 800-63B-4 states that a biometric system shall operate with a false match rate of one in 10,000 or better for all demographic groups, that the categories considered shall include sex and skin tone where those affect performance, that the rate shall be achieved under zero-effort impostor conditions, and that the system shall use a fixed threshold because changing it per demographic is not feasible. It also requires presentation attack detection for face and prohibits voice comparison outright. ISO/IEC 19795-10 exists to measure this variation, and NISTIR 8429 was written partly to inform it.

Per-group false match rate is now a shall-statement in SP 800-63B-4, which makes it an audit item.

What to do on Monday

Find out whether any authentication or identification decision in your estate depends on a biometric model, and whether anyone has measured its false match rate by group at the operating threshold. If the vendor quotes a single figure, that is an aggregate and you should ask what it decomposes into.

Ask for the SP 800-63B-4 test directly. That means evidence of a one-in-10,000 false match rate for every relevant demographic group, at a fixed threshold, under zero-effort impostor conditions, and not merely in aggregate. NIST’s demographic tables are public and updated, and they cover algorithms submitted for evaluation rather than the finished products built on them. A vendor whose algorithm is listed and who will not discuss its position has told you something.

Check whether any human review step downstream of a model consumes the model’s output before forming its own view. That is the wrongful arrest mechanism, it generalises far beyond faces, and it is usually cheap to fix by changing what the reviewer sees first.

Treat differential error rates as a control weakness rather than a disclosure item. A control that is measurably weaker for some users than others has a documented gap, and the appropriate artefact is a risk acceptance with a named owner rather than a paragraph in a fairness statement.

The measurement has existed since 2019 and it came from the national metrology institute rather than from an advocacy group. The larger demographic differential is in the error mode that admits the wrong person, and it now carries a number in a federal requirement. Everything after that is a question about who reads the report.