Differential Privacy Protects the Training Set, Not the System

On 28 April 2021 the US Census Bureau released a demonstration product for the 2020 redistricting file at a privacy-loss budget of ε=12.2, and announced that it met or exceeded the specialised accuracy targets built from Department of Justice use cases for redistricting and Voting Rights Act enforcement. Six weeks later the Bureau’s Data Stewardship Executive Policy Committee set the production parameters at ε=19.61, split as 17.14 for the persons file and 2.47 for the housing unit file, after data users asked for better accuracy on places, Minor Civil Divisions, tribal areas and race-by-ethnicity counts.

Epsilon appears in an exponent. A mechanism satisfying ε-differential privacy bounds the ratio between two output distributions at e^ε. The shipped setting therefore permits roughly 329 million to one, against about 199,000 to one for the configuration that had already cleared the accuracy targets. Both figures come from converting the Bureau’s zero-concentrated accounting at δ=10⁻¹⁰, a step NIST calls loose and lossy, and one that should always be reported alongside the original parameter.

I would not accept a differential privacy claim without four things. The epsilon, the delta, the unit of privacy the guarantee is defined over, and the threat model the implementation was tested against. Without all four the claim cannot be falsified. With all four it usually turns out to be much narrower than the word “private” implies.

What the definition bounds

Dwork, McSherry, Nissim and Smith stated the property at TCC in 2006, calling it ε-indistinguishability; the name differential privacy came from Dwork’s ICALP paper later that year. The (ε, δ) relaxation, which permits a small failure probability, appeared in Dwork, Kenthapadi, McSherry, Mironov and Naor’s Our Data, Ourselves at EUROCRYPT the same year. A randomised mechanism M is (ε, δ)-differentially private if, for every pair of neighbouring datasets D and D′ and every set of outputs S, the probability that M(D) falls in S is at most e^ε times the probability that M(D′) does, plus δ. Every ratio quoted here is that multiplicative term. The deltas involved run around 10⁻¹⁰, small enough that the term accounts for nearly all of the guarantee without being the whole of it.

Three constraints come out of the definition.

The property belongs to the mechanism. It describes what an adversary can learn from observing an output, given that the adversary already knows everything else. NIST puts the limit plainly: differential privacy does not necessarily prevent somebody from making inferences about an individual.

The property is defined against a choice of “neighbouring”. Two datasets differ by one what? One row, one training example, one 1,024-token sequence, one person’s entire contribution across a corpus? Each choice produces a different guarantee under the same numeral.

The bound is worst-case, which makes it conservative on paper and not self-verifying in deployment: a small epsilon says nothing about whether the code computes what the proof describes.

NIST published SP 800-226, Guidelines for Evaluating Differential Privacy Guarantees, in final form on 6 March 2025. Near, Darais, Lefkovitz and Howarth arrange a differential privacy claim as a pyramid: epsilon and the unit of privacy at the top, algorithms and correctness with utility and bias in the middle, and access control, trust model, query model, side channels and data collection exposure at the base. Each layer’s ability to protect privacy depends on the layers below it, and evaluating any claim requires examining every one.

On the parameter itself, NIST offers no specific guidelines for selection. It reports that ε below 1 is generally considered reasonable and that many deployments have used values in the range 1 < ε ≤ 20. It then prints a Privacy Hazard box: large values of ε may not provide meaningful privacy. The evidence follows. Stadler and colleagues showed that differentially private synthetic data built at ε=10 can leave outliers open to linkage attacks, and Nasr and colleagues showed that maliciously crafted training sets can produce significant leakage from differentially private neural networks at ε=10.

Sequence, example, user

Google Research and DeepMind released VaultGemma on 12 September 2025, a 1B-parameter open model pre-trained end to end with DP-SGD rather than fine-tuned privately on top of a non-private base. The reported guarantee is ε ≤ 2.0 and δ ≤ 1.1×10⁻¹⁰, which bounds the multiplicative term at 7.4 to one.

That guarantee is defined at sequence level, over 1,024 consecutive tokens. Google states the consequence in the announcement. Where a private fact appears in one sequence, the model essentially does not know it. Where many training sequences contain information relevant to the same one, VaultGemma will in general supply it. Google also says that where a clear mapping exists between data and users, user-level differential privacy would be the better choice. NIST agrees, calling user-level privacy the best default and recommending a privacy unit at least as large as an individual user.

Google reported the utility cost too, which is more than most labs do: it describes VaultGemma’s benchmark performance as comparable to non-private models from roughly five years ago, using Gemma 3 1B as the non-private counterpart and GPT-2 1.5B as the older baseline. Differentially private pre-training works, and in September 2025 it cost about five years of capability.

Composition changes the deployed number

NIST gives the mechanism in one line: a user-day guarantee at ε=1 looks strong because the numeral is small, and the total privacy loss over a year is ε=365.

Tang, Korolova, Bai, Wang and Wang found the same pattern in production. They reverse-engineered Apple’s local differential privacy implementation in macOS 10.12 in 2017. Per submission the privacy loss was 1 or 2. Across the four announced applications the daily total reached 16, and the allowance renewed every day. At ε=16 the multiplicative term is about 8.9 million to one, per day, for as long as the user leaves the feature on. Apple’s response, as reported at the time, was that the researchers had summed epsilons across data types the company does not link.

The disagreement was about the protected unit and the composition rule. Nobody disputed the arithmetic. A vendor asked for “the epsilon” will quote the per-submission one.

What epsilon costs

Mohammadi and colleagues published a scoping review of differential privacy in medical deep learning in npj Digital Medicine on 3 January 2026, covering 74 studies published through March 2025 across centralised and federated settings.

DP-SGD holds clinically acceptable performance at ε ≈ 10, particularly on imaging tasks. At ε ≈ 1 the accuracy loss is often substantial, and degradation increases on smaller or heterogeneous datasets.

Set that beside the NIST passage above. The budget at which a clinical model still does its job is the budget at which NIST cites demonstrated leakage from differentially private neural networks. That is the whole trade in one number, and it is why a claim of “we train with differential privacy” tells a defender almost nothing on its own.

A minority of the 74 studies evaluated fairness at all. Several of those reported that differential privacy widens subgroup performance gaps, and the review treats the size of the effect as dependent on task and dataset. Bagdasaryan, Poursaeed and Shmatikov demonstrated disparate impact from DP-SGD at NeurIPS 2019, and the 2022 version of this article already cited that paper. NIST reproduces the effect in its own figures. It plots classifier accuracy on US Census data across values of epsilon, separately for the majority race and for a minority group. The noise costs that group far more accuracy. Gradient clipping and calibrated noise take more from the underrepresented tail than from the mode, which becomes a security question when the tail is the population the control exists to protect. I covered the underlying mechanism in biases in machine learning.

Reporting of privacy parameters across those 74 studies was inconsistent. Papers claim differential privacy without stating the accountant, the unit, or in some cases the epsilon.

The proof covers the mechanism, not your implementation

Ilya Mironov showed in 2012 that finite-precision arithmetic breaks the Laplace mechanism. The textbook sampling procedure produces a porous distribution over double-precision numbers, and the gaps let an attacker distinguish outputs that the proof calls indistinguishable, with a handful of queries. All four general-purpose differential privacy systems available at the time were susceptible: PINQ, Airavat, Fuzz and GUPT. The mathematics was correct. The code did not implement the mathematics.

Jin, McMurtry, Rubinstein and Ohrimenko extended the result at IEEE S&P 2022 in Are We There Yet? Timing and Floating-Point Attacks on Differential Privacy Systems. Two findings apply directly to machine learning deployments.

The Gaussian mechanism is vulnerable to the same class of floating-point attack, and the authors demonstrated it against deep learning models trained with DP-SGD, where Mironov had studied query systems.

The discrete samplers built specifically to fix the floating-point problem leak through timing. Those samplers draw repeatedly until a coin toss succeeds. The number of draws correlates with the magnitude of the noise returned, so an observer who can time the sampler learns the noise and subtracts it. Two libraries were vulnerable, one of them a differential privacy library published by Google. The authors notified the maintainers.

Ratliff and Vadhan proposed a framework for differential privacy against timing attacks at CCS 2024, so the problem has attention. Choosing a smaller privacy budget fixes neither attack, because both defeat the noise sampler upstream of the accounting.

A differential privacy deployment deserves cryptographic-style implementation scrutiny, and NIST draws the same comparison, advising that the randomised parts of these algorithms are difficult to get right and easy to get wrong, that this resembles implementing cryptography, and that practitioners should reach for rigorously validated libraries instead of custom implementations. No security team accepts “we use AES” without asking about mode, key management and side channels. “We use differential privacy” is currently accepted on those terms.

Auditing finds bugs, not assurance

Privacy auditing exists because the proof might be wrong or the code might not match it. The method inserts canary records, runs a membership inference attack against them, and derives an empirical lower bound on epsilon from how well the attack separates the canaries from records that were never inserted.

NIST states the limit of the method precisely. Empirical approaches, including auditing, cannot prove that a system correctly provides a desired differential privacy guarantee. They can be helpful in finding implementation bugs, because a counterexample means the system definitely does not provide the claimed guarantee. The asymmetry is the whole point. An audit can disprove a claim. It cannot confirm one, and a weak audit result is not a measurement of how much a system leaks.

Access level determines how tight the lower bound gets. White-box one-run auditing, where the auditor sees and manipulates gradients during training, has become reasonably tight: an unreviewed June 2026 preprint recovers an empirical bound of about 6.7 against an analytic ε=8 on CIFAR-10 with DP-SGD, roughly 84% of the claimed value, against about 4.7 and 3.3 for two earlier methods [EDITOR: Verify – single preprint, arXiv 2606.12733, no peer review yet]. Black-box last-iterate auditing, where the auditor sees only the final model, recovers far less: its empirical figure stays well short of the analytic guarantee.

A buyer evaluating a vendor is in the black-box setting, so a clean audit result buys very little. Treat the accounting bound as the claim, the empirical bound as demonstrated leakage, and the distance between them as unresolved. NIST adds one more caution for any vendor conversation: audits that report average-case results can significantly underestimate the risk to outliers and produce false confidence.

Training-time control, inference-time exposure

Differential privacy applied during training constrains what ends up in the weights.

A model with a strong DP guarantee can be wired into a retrieval pipeline over patient records or case files. The retrieval step reads those documents at inference time and places them in the context window, and the model reproduces them in its output. No training-time guarantee applies to that path. The guarantee describes the model; the leak happens in the system. Holding the model, the system and the agent apart is the most common failure in AI security writing, and retrieval is the cleanest current example of it.

Applying differential privacy to the retrieval step is an open research problem, and the published schemes are preprints. Koga, Wu, Zhang and Chaudhuri proposed a DP-RAG scheme that spends privacy budget only on the tokens requiring sensitive information, and report beating a non-RAG baseline at ε ≈ 10 in the settings they evaluated [EDITOR: Verify – preprint; confirm whether it has since appeared in a peer-reviewed venue]. Two conditions on that number. The budget is spent per query, so an interactive assistant depletes it over the course of a conversation. The accounting then has to hold across sessions wherever the same privacy unit participates. And ε ≈ 10 is where their scheme overtakes a system with no retrieval at all, which sets the expectation honestly: differentially private retrieval currently recovers the value of retrieval, at a bound of around 22,000 to one.

An agent retains persistent memory, calls tools, writes logs, and passes context to other systems. Each of those is a copy of the data outside the model. A training-run guarantee constrains none of them.

Nothing here protects the annotator

The 2022 version of this article argued that differential privacy secures the data labelling pipeline, where medical records and other sensitive material are most exposed to human eyes. It does not.

Differential privacy is applied in three places: locally at collection, before data reaches the collector, as in the Apple case above; during training; or to a query or release mechanism. An annotator labelling an MRI scan or a patient history reads the raw record directly. There is no intervening mechanism, so there is nothing to add noise to. NIST says the same in its executive summary: differential privacy protects data subjects in the context of intentional differentially private releases, and does not protect data as it is collected, stored and analysed in raw form. NIST places data collection exposure at the base of the pyramid for that reason.

Distributed and crowdsourced annotation multiplies the number of people and jurisdictions holding that view, which is a real and under-covered risk. The controls that address it are access control, contractual and jurisdictional constraints, minimisation before data reaches the labelling queue, and monitoring of the annotation platform.

Differential privacy can be applied downstream of labelling, at training time. That protects the people in the training data from what the model leaks. It does nothing for those same people with respect to the vendor whose staff read their files. Two threats with two owners, which is why this site keeps the four-way distinction between secure, safe, responsible and trustworthy AI intact.

What a regulator does with a differential privacy claim

The EDPB adopted Opinion 28/2024 on 17 December 2024, answering four questions from the Irish Data Protection Commission about personal data in AI models. Anyone selling differential privacy as a route to GDPR anonymity should read paragraphs 43 to 55 first.

The threshold set at paragraph 43 is high. For a model to be treated as anonymous, both the likelihood of extracting personal data about individuals in the training set and the likelihood of obtaining such data from queries must be insignificant for any data subject, assessed against all means reasonably likely to be used.

Differential privacy appears by name at paragraph 52, in a list of methodological choices supervisory authorities should evaluate, phrased as whether the controller implemented appropriate and effective privacy-preserving techniques. Paragraph 48 states that the presence or absence of any element on that list is not conclusive. So differential privacy counts as evidence in an anonymity argument, and it does not settle one.

Paragraph 55 tells authorities to weigh the scope, frequency, quantity and quality of the testing a controller has done, and names the attacks: attribute and membership inference, exfiltration, regurgitation of training data, model inversion and reconstruction. So paragraph 55 puts structured adversarial testing into the evidentiary case for an anonymity claim. It imposes no testing duty on controllers. The attack list is the same one this site has documented since 2019, including query attacks against deployed models.

The surrounding law is unsettled. The Commission published the Digital Omnibus package on 19 November 2025 in two parts. The AI Omnibus, amending the AI Act, received final Council approval on 29 June 2026 and entered into force on 27 July 2026. The Data Omnibus, COM(2025) 837, contains the proposed new Article 88c on legitimate interest for the development and operation of AI systems and a new Article 41a on pseudonymisation criteria, and it remained in negotiation as of 5 September 2026, with the EDPB and EDPS having objected to several of its central provisions in a joint opinion of February 2026 [EDITOR: Verify status immediately before publication; also confirm the AI Omnibus regulation number and OJ date, which one review gives as Regulation (EU) 2026/1744 of 8 July 2026, unverified here]. No instrument in force today makes differential privacy a compliance shortcut, and no instrument currently proposed would.

What to ask, and what to do

Four questions for any differential privacy claim, whether it arrives from a vendor, an internal team, or a paper.

What is epsilon, what is delta, and over what unit of privacy? Guarantees defined over a sequence, over a training example and over a user amount to three different claims. A per-query epsilon with no composition analysis describes a query, not a system. Where the figure was converted from another variant, ask for the original parameter as well.

What is the composition over the system’s lifetime? Budgets that renew daily, or that are spent per query, produce a lifetime total wherever the same protected unit contributes repeatedly, and that is the number that applies to your data subjects.

What was the implementation audited against, at what access level, and by whom? Floating-point and timing side channels defeat correct mathematics. A vendor who has not heard of the 2022 IEEE S&P result has not looked.

What did the utility and fairness measurement show, broken out by subgroup? An unmeasured gap between subgroups is an unquantified fairness exposure and, in regulated sectors, an unquantified compliance one.

Three decisions follow from the evidence above.

Use differential privacy where the release is statistical and the unit is clean: telemetry, aggregate reporting, published statistics, and training runs over data you already hold lawfully. Those are the cases the mechanism was designed for and the cases where it delivers.

Do not place it in front of retrieval, agent memory, or logs and expect protection: no training-time guarantee reaches those paths, which need access control, minimisation, and encryption of data in use. The deployable controls for the retrieval path today are conventional ones.

Treat a differential privacy claim the way you would treat a cryptographic one, with the same questions about implementation, side channels and composition. Differential privacy is unusual in machine learning because it puts a real, quantitative, composable bound on the influence of a defined privacy unit. What it covers is narrower than the word suggests, and every deployment decision depends on the difference between the two.