What Model Inversion Attacks Actually Recover

Correction, 5 September 2026: this article has been rewritten. Three factual errors in the February 2021 version are corrected here. The original cited the Sarbanes-Oxley Act as a data confidentiality regime, which it is not. It listed the same paper twice in its references, leaving the claim about a fairness-based defence without a source. And it recommended federated learning with secure aggregation as a mitigation, which research published in 2022 and 2023 has since undermined in the form the original described.

In 2015, Matt Fredrikson, Somesh Jha and Thomas Ristenpart showed that a face recognition model could be made to synthesise recognisable images of identities in its training set, given a name and the confidence scores the model returned alongside its predictions. Their paper is cited in nearly everything written about model inversion since, including the version of this article I published in February 2021.

A decade later, researchers at the Singapore University of Technology and Design and Leipzig University re-measured the field. They took 27 model inversion attack setups across different datasets, target models and generative priors, and checked them against the evaluation framework that almost every recent attack and defence paper uses. Across those 27 setups, false-positive rates under the conventional framework reached 99% in some configurations. Reconstructions counted as successful were, in a large share of cases, images that satisfied the evaluation classifier without reproducing the face of the person they claimed to recover.

A CISO can act on three facts. Model inversion works. Its published success rates are not reliably comparable without the evaluation protocol that produced them. And in 2026 the component in an enterprise AI stack most likely to expose sensitive organisational data is the retrieval layer, not the model.

That makes this a different article from the one I wrote in 2021, and it should be.

The 2021 piece described the mechanism correctly and got the deployment picture wrong, which is the standard failure mode in this part of AI security.

What the attack actually recovers

The research literature separates two things that the 2021 article ran together. One class of attack reconstructs actual training samples. The other constructs a class representative, an input the model associates strongly with a chosen label. Fredrikson’s face attack belongs mostly to the second class. It recovers what the model has learned to associate with a name, which resembles the person when the class contains few and similar examples and resembles an average when it does not.

The original pixel-space attack optimises directly over the input pixels, with no generator constraining the search. On shallow models it reconstructed low-resolution greyscale faces. On deep networks trained on high-dimensional data the same search terminates on high-confidence images with no semantic content.

Many of the strongest modern face-inversion attacks solve that high-dimensional search by constraining it with a learned generative prior, historically a GAN and increasingly a diffusion model, trained on public data drawn from roughly the same distribution as the private training set. The prior is not a requirement of reconstruction in general, and the gradient-based methods described below recover images without one. But where a paper reports a photograph you would recognise, the prior supplied the face’s structure from its own public training data. Papers state that assumption far less prominently than the accuracy figure it produces.

Three consequences follow for a defender. Reconstruction quality generally rises with overfitting, so a model with few examples per class gives up more than a model with many. In this class-based setting the most favourable targets are narrow, single-purpose models. That is not a claim that frontier models are safer overall, since their training-data leakage arrives through a different mechanism, memorisation and regurgitation, which needs its own piece. And the attacker needs public data resembling your private data, which is easy for faces and hard for a claims-adjustment model trained on your own book of business.

A second distinction the 2021 version lost is the one between inversion and model extraction. Extraction copies a model’s behaviour or its weights, and the victim is the model owner. Inversion recovers information about the people whose data trained the model, and the victims are the data subjects. The original had a section headed “Loss of Intellectual Property” that described extraction under the inversion label. Conflating them sends the defender to the wrong control. In Europe an inversion risk is measured against the anonymity test the EDPB set out in Opinion 28/2024, which the section below describes.

Membership inference is a third question again. It asks whether one specific record was in the training set, which is a serious disclosure whenever membership in the dataset is itself sensitive. NIST’s March 2025 adversarial machine learning taxonomy makes the same separations. Its four top-level categories are data reconstruction, model extraction, and membership and property inference. Attribute inference is classified inside data reconstruction.

The standard success metric counts adversarial examples as reconstructions

Almost every model inversion paper measures success the same way. The researchers train an evaluation model on the same private data and the same task design as the target model. They run the attack, feed the reconstruction to the evaluation model, and count a success when it assigns the target identity. The framework is convenient, automatic and, until last year, unexamined.

The problem the SUTD and Leipzig group identified is that a reconstruction can satisfy the evaluation model for the same reason an adversarial example satisfies a classifier. The paper’s term for these is Type I adversarial examples, meaning inputs a model confidently assigns to a class while showing none of the features a human would use to make that assignment. The authors show high false-positive transferability across evaluation models, which is the empirical signature of adversarial behaviour and not of genuine reconstruction.

They then rebuilt the evaluation on multimodal large language models, whose general-purpose visual reasoning does not share the target model’s task design, and re-ran the 27 setups. Reported attack accuracy fell, in some configurations by almost everything.

Two conclusions are available and only one is licensed. The licensed one is that headline model inversion success rates are not reliably comparable without the evaluation protocol and the attack conditions that produced them. The unlicensed one is that inversion does not work. The attack recovers real information; we do not currently know how much, and papers quoting a specific figure from before the re-evaluation are claiming more precision than their instrument supports.

The commercial edge of this is sharper than it looks. Search for model inversion and you will find vendor pages citing IBM’s Cost of a Data Breach research, and the numbers are real. The 2025 edition found that 13% of surveyed organisations reported a breach involving an AI model or application, and attributed 24% of those AI incidents to model inversion. The 2026 edition, published on 29 July 2026 from 602 breached organisations, put the first figure at 21% and made inversion the most expensive AI incident type at an average of USD 6.07 million, ahead of prompt injection at 5.89 million.

Breached organisations classify their own incidents on the Ponemon questionnaire that produced both figures. The label that separates a reconstruction attack on model outputs from a leaked embedding store or an over-permissive retrieval API is the respondent’s own. IBM’s own reading points the same way. The report concludes that the root causes were compromised connected APIs, vulnerable applications and cloud misconfigurations, which it describes as governance failures and not model risk. The USD 6.07 million is therefore an average across incidents that respondents themselves labelled inversion, none of them named, dated or audited. The useful finding in that data is IBM’s conclusion about root cause, which agrees with everything below.

The leaking interface moved to the embedding layer

While the image-inversion literature argued with itself, the practical attack surface changed. Retrieval-augmented generation put a vector database into a large share of enterprise AI deployments, and those stores contain embeddings of exactly the documents an organisation treats as confidential. Most teams classify the store as derived data, on the reasoning that an array of floats is not a document.

Embeddings are not one-way functions.

Congzheng Song and Ananth Raghunathan demonstrated systematic leakage from text embeddings at CCS 2020. John Morris and colleagues published vec2text at EMNLP 2023, recovering source text by iteratively correcting a generated candidate until its embedding matched the target. Those attacks needed either white-box access or a large corpus of embedding and text pairs from the victim encoder, which kept them awkward to run in practice.

Both constraints have weakened since. ALGEN, published at ACL 2025, aligns a victim embedding space to an attacker’s using around a thousand text-and-embedding pairs, which the attacker can obtain by querying the victim encoder instead of stealing them. Zero2Text, posted on 2 February 2026, removes the pairs entirely. It combines an LLM prior with a ridge regression that re-aligns during decoding, needs no decoder training and no in-domain data, and runs under black-box query access. Its victim models include OpenAI’s closed-source text-embedding-3 encoders, in both the small and large sizes.

The reported fidelity is partial. Against text-embedding-3-large on MS MARCO, Zero2Text reaches a ROUGE-L of 26.08 where ALGEN reaches 14.79, on sequences capped at 32 tokens. It recovers topic, proper nouns and some word order, and it does not hand back the document. Local differential privacy degrades it further, and at the strongest noise setting the authors tested, ROUGE-L falls from 25.64 to 13.75. Their claim is that a defended Zero2Text still matches an undefended earlier attack, which is a real result and a narrower one than differential privacy failing.

The access assumptions differ by more than the methods do. Image inversion needs a generative prior aligned to the private distribution and a target model narrow enough to have memorised its classes. Embedding inversion starts from a target embedding, which means the attacker has already obtained vectors from the store, a backup, an over-permissive retrieval API or another disclosure path, and then queries the victim encoder to reconstruct from them. Query access alone gives an attacker nothing, because it does not hand over anybody else’s confidential vector.

The first describes a research condition. The second describes an infrastructure compromise, or a cross-tenant authorisation bug of the kind application security teams have been finding for twenty years. The exposure belongs to the system around the model, and ordinary infrastructure controls apply to it.

OWASP moved in the same direction. The Top 10 for LLM Applications 2026, published on 4 August 2026, keeps sensitive information disclosure at LLM02 and lists vector and embedding weaknesses at LLM09, down one place from the 2025 edition. The list ranks concern as much as frequency: expert and community voting supplies 75% of the weight and incident evidence 25%.

Federated learning does not answer this, and the 2021 article said it did

The mitigation section of the original article recommended federated learning with secure aggregation, on the reasoning that a server seeing only aggregated updates cannot reconstruct individual contributions. That reasoning is correct for an honest-but-curious server. It fails for a server that chooses the model weights it distributes.

Liam Fowl and colleagues showed at ICLR 2022 that a server distributing a deliberately modified model, architecture as well as weights, can read individual training records straight out of the returned updates.

Franziska Boenisch and colleagues went further at EuroS&P 2023, attacking federated learning hardened with both secure aggregation and distributed differential privacy. Their server introduces sybil devices into the participant cohort. With 50 participants each holding a mini-batch of 20 data points, at a clipping bound of 1 and a noise scale of 0.1, a target user sampled entirely alongside sybils had 95% of their individual training points extracted. Adding genuine participants back into the round degrades the reconstruction. That is the honest qualifier, and it also explains the mechanism. Under distributed differential privacy the noise each participant contributes scales down as the participant count rises, so a fabricated cohort weakens the aggregation and the privacy budget at the same time.

State the threat model plainly, because it decides whether any of this applies to you.

If you operate the aggregation server and the participants trust you, these attacks describe an insider you already trust. If you are a hospital or a bank contributing to someone else’s federation, the party choosing the initial weights and sampling the cohort can, on current published methods, recover your records. Use differential privacy, and specify which guarantee you actually hold. Local differential privacy gives the participant a bound that does not depend on trusting the aggregation server, and costs utility for it. Distributed and central variants buy the utility back, and their guarantee then depends on protocol assumptions, including the presence of enough honest participants contributing noise. Boenisch is precisely the case where that assumption fails. Secure aggregation does not rescue a federation whose server also picks the cohort.

The GDPR argument now depends on a measurement

There is a governance consequence that most technical writing on this topic skips. On 17 December 2024 the European Data Protection Board adopted Opinion 28/2024, which addresses when an AI model trained on personal data can be treated as anonymous.

Its answer is that such a model cannot be considered anonymous in all cases and that anonymity is assessed model by model. Two conditions have to hold. The likelihood of extracting personal data directly from the model must be insignificant, taking account of methods including membership inference and model inversion. And querying the model must not produce personal data relating to the individuals in its training set, even accidentally.

So a European organisation arguing that its trained model is anonymous for GDPR purposes is making, among other things, an empirically testable claim about extraction and query risk. The EDPB’s test is broader than any single number, covering model design, access, cost, time and external information, but it has to be demonstrated with documentation, testing and ideally independent validation. That evidence problem is harder now. A controller cannot establish anonymity by quoting a benchmark when the dominant automated measurement for that benchmark has been shown to generate severe false positives.

The AI Omnibus, adopted by the Council on 29 June 2026, amends the AI Act and deferred the Annex III high-risk obligations to 2 December 2027. It does not touch the GDPR.

The Data Omnibus, which carries the proposed GDPR amendments including a narrower entity-relative test for personal data, remains under negotiation in the Council. Successive presidency compromise texts through the first half of 2026 removed several of the Commission’s principal GDPR reforms, the entity-relative personal-data test among them, and member states did not adopt the June text. Later revisions may exist that are not publicly readable. Until an amending regulation is adopted, supervisory authorities assess anonymity against Opinion 28/2024. I checked this on 5 September 2026.

What to do on Monday

The embedding store

Inventory your embedding stores and classify each at the sensitivity of the documents that produced it. If the source corpus contains patient records, treat the vectors and their backups at the sensitivity of the patient data, whether or not the store also holds the source chunks. Encrypt the vectors on disk, enforce tenant isolation, log reads, and include the store in the same access review as the source system. This one change addresses more realistic exposure than any published inversion defence.

Treat embedding endpoints as regulated output. If a customer or partner can submit text and receive vectors, they can build the alignment an inversion attack needs.

The prediction API

For predictive models exposed by API, return the top label without confidence scores wherever the product tolerates it. Label-only inversion exists, demonstrated by Kahla and colleagues at CVPR 2022, and it turns the problem into an iterative boundary search costing on the order of ten thousand queries per class. Rate limiting and per-client query budgets therefore raise the cost of a black-box attack substantially. They do nothing against an attacker with white-box access or a compromised model host, so they are one control among several.

Treat overfitting as a privacy control. Class counts, regularisation and example floors per label change how much an inverted model gives up, and your data science team already sets all three for other reasons.

Everything else

If you contribute to a federation you do not operate, ask two questions before the next round. Who sets the initial model weights? How are participating devices verified? A federation whose server controls the weights and whose participants go unverified gives you less privacy than its architecture diagram implies.

Stop accepting attack-success percentages without their conditions. Ask what model was attacked, at what access level, with what query budget, against what defence, and, from 2025 onward, which evaluation framework produced the number. A vendor selling inversion protection on the strength of pre-2025 benchmark figures is selling against a measurement its own field has retracted confidence in.

Finally, name who signs the anonymity assessment. In most organisations that document has no owner, which is one of the arguments for a dedicated AI security role and one of the places where secure, private and compliant come apart in practice. Securing a model, keeping it private and demonstrating its compliance are three separate obligations. A supervisory authority asking how you established anonymity will want a name and a date on the answer.

What would change my position

I have not found a publicly named and dated incident in which classic output-based model inversion recovered identifiable data from a production system. Everything documented comes out of laboratories, on face benchmarks, under access conditions the researchers chose. That absence is not proof of safety. Privacy attacks are hard to spot from the defender’s side, because their queries are well-formed and their credentials valid.

IBM’s survey numbers do not close that gap. Anonymised self-classification by breached organisations records what respondents called the incident, and IBM’s own root-cause finding points at APIs and cloud misconfiguration. Eleven years after Fredrikson, the attack has a large literature, a dollar figure derived from questionnaires, and no named victim.

Four findings would move me. A named, dated incident with documented conditions. An audited breakdown of AI incidents by technique, rather than a self-reported one. A reliable evaluation showing that models with realistic class counts give up individual records, since the class-representative result is much weaker than the popular account of it. And an embedding inversion result that holds on long documents at production corpus scale, because the strong recoveries so far run to 32 tokens and enterprise retrieval chunks do not.

Until then, the defensible position is the one I would have given a client in 2005 and would give one today. Spend on the interfaces an attacker can actually reach, treat derived data at the sensitivity of its source, and read every percentage in a privacy paper alongside the assumptions that produced it.