Model Stealing Split Into Three Different Attacks

On 29 January 2026, a federal jury in San Francisco convicted Linwei Ding on seven counts of economic espionage and seven counts of theft of trade secrets. The Department of Justice called it the first American conviction for AI-related economic espionage.

Seven of those fourteen convictions have since been thrown out. On 20 August 2026 Judge Vince Chhabria acquitted Ding on all seven espionage counts, finding the evidence insufficient to show he knew or intended that his conduct would benefit the Chinese government. Double jeopardy made it an acquittal instead of a retrial. The seven trade-secret convictions stood, and on 1 September he was sentenced to twelve months minus a day.

Ding joined Google in 2019 as a software engineer on the supercomputing infrastructure that trains and serves the company’s largest models. Between May 2022 and April 2023 he moved thousands of pages of confidential material to his personal Google Cloud account while forming his own AI company in China. The stolen material covered chip architecture, networking, and the orchestration software Google runs on its AI supercomputers. No weights and no training data.

He copied documents to a cloud drive, without a single API query or a surrogate model.

I wrote the first version of this piece in 2019, and it described model stealing the way the field described it then: an attacker with query access rebuilds a model he does not own. That attack is real and has been demonstrated against production systems. It has also turned out to be the most constrained of the three things people now mean when they say a model was stolen. The other two are distillation through fraudulent API accounts and exfiltration of the weight files themselves.

Rate limiting counts queries per account and resets to zero on each new one. Against a campaign spread across twenty-four thousand of them, the controls that raise an attacker’s cost are identity checks and correlation across accounts.

What the phrase used to describe

The term comes from Florian Tramèr and colleagues at USENIX Security 2016, in Stealing Machine Learning Models via Prediction APIs. They recovered the exact parameters of a logistic regression model through its prediction interface, then extended the work to decision trees and shallow networks. They demonstrated the attack on the prediction APIs of two commercial services, BigML and Amazon Machine Learning.

The framing suited what a deployed model was at the time. The asset was a decision boundary, the interface returned confidence scores, and a few thousand well-chosen queries bought a copy of something a company had spent a year building.

A frontier model is no longer a decision boundary, and the campaigns of the past two years went after agentic reasoning, tool use and code generation.

You also cannot steal an agent by querying it. An agent is a model plus tools, credentials, memory and an orchestration layer, and none of those four appear in the outputs. Capability leaves through the API. The deployment stays where it is.

So one phrase now covers three attacks. Extraction needs an API that returns token log probabilities. Distillation needs thousands of accounts bought through a proxy reseller. Exfiltration needs the inference server itself, or somebody inside with read access to the weights. Controls overlap at the edges, and the split below is my own decomposition and not a settled taxonomy. The taxonomy of attacks on AI models keeps them as separate entries. I’ll take them in the order the public evidence constrains them, which is roughly the reverse of how much coverage each receives.

Recovering parameters

What query access actually yielded

The strongest public result against a production language model belongs to Nicholas Carlini and twelve co-authors at ICML 2024, in Stealing Part of a Production Language Model. Their attack recovers the embedding projection layer, up to symmetries, from ordinary API access.

For under $20 they extracted the full projection matrix of OpenAI’s Ada and Babbage models, confirming hidden dimensions of 1024 and 2048. They recovered the exact hidden dimension of gpt-3.5-turbo and estimated its full matrix at under $2,000 in queries.

The access assumptions matter more than the price. The attack needs an API that returns token log probabilities, or a logit-bias parameter an attacker can use to recover them, which is what allows reconstruction of the low-dimensional subspace the logits occupy. On the authors’ own disclosure timeline, Google shipped mitigations after the standard 90-day period and OpenAI followed on 3 March 2024. Providers have since restricted the combination, and Carlini’s group names extraction risk as the reason. So the strongest published result against a production model recovers one layer, through an interface that closed within months of disclosure. Anyone selling a product against attackers who reconstruct GPT-class weights through an API is selling against something no published work has demonstrated.

The edge is a different problem

Ashley Kurian, Anuj Dubey, Ferhat Yaman and Aydin Aysu published TPUXtract in TCHES 2025(1). They recovered every layer configuration of a convolutional network running on a Google Coral Edge TPU at 99.91% accuracy: layer type, node counts, kernel and filter sizes, strides, padding and activation functions, from electromagnetic emissions alone.

The access assumptions run further than physical possession of the board. The attack also needs enough control of the device to deploy models on it and drive chosen inputs through them, plus an EM probe, a positioning stage and an oscilloscope. What comes out is the architecture and never the weights themselves. Put a model onto hardware a competitor can buy and hold, and the API stops being your attack surface. Your architecture stops being a secret.

These two attacks recover a layer and an architecture, and a working copy of a frontier model is beyond both. Both make surrogate-model attacks cheaper, which is the second-order harm the 2016 paper already anticipated. A faithful surrogate turns black-box evasion into a white-box problem an attacker can solve offline.

Distillation is the attack that scaled

On 23 February 2026 Anthropic published an account of three distillation campaigns against Claude, naming DeepSeek, Moonshot AI and MiniMax. The disclosed totals are over 16 million exchanges through roughly 24,000 fraudulent accounts.

DeepSeek accounted for more than 150,000 exchanges aimed at reasoning and at rubric-based grading that turned Claude into a reward model. Some prompts asked it to reconstruct and write out the reasoning behind a finished answer, which manufactures chain-of-thought training data at scale. Moonshot accounted for more than 3.4 million, targeting agentic reasoning, tool use, coding and computer-use agents. MiniMax accounted for more than 13 million, concentrated on agentic coding and tool orchestration.

Access came through commercial proxy resellers, since Anthropic sells no commercial access into China. One proxy network ran more than 20,000 fraudulent accounts at once and mixed the distillation traffic with unrelated customer requests. OpenAI made a parallel allegation in a February 2026 memorandum to the House Select Committee on the Chinese Communist Party, and Google’s Threat Intelligence Group published its AI Threat Tracker on distillation and extraction the same month.

Now the distance between claim and support.

Every one of these accounts is first-party attribution by the party claiming injury. Anthropic reports high confidence from IP correlation, request metadata, infrastructure indicators and, for some campaigns, corroboration from other providers seeing the same actors. It publishes no indicators, no third party has verified the attribution, and the three named labs did not respond to CNBC’s request for comment.

The post also argues that the finding strengthens the case for chip export controls, a position Anthropic already held and has lobbied for. Anthropic’s interest in that conclusion does not make the allegation false, and the detail here far exceeds anything produced during the January 2025 DeepSeek episode, when the evidence amounted to unnamed sources and a suspicious training cost. It does mean the account is a vendor’s reading of its own logs, and it should be cited that way.

Nobody has reconciled the scale of these campaigns with what the distillation literature reports. Davis Brown and colleagues, summarising prior work in a January 2026 preprint on weight exfiltration, note that students distilled from query access alone typically underperform their teacher by a wide margin.

Anthropic’s own breakdown explains the gap. Each campaign went after a small set of capabilities, and the list is specific in every case: agentic coding, tool orchestration, computer vision, rubric grading. Distillation transfers a capability well and a model poorly. That changes what a defender protects. The capability transfers through ordinary API queries, so a control that protects the weight file does not apply to it.

Weight exfiltration is an infrastructure problem

The reference work is Securing AI Model Weights by Sella Nevo and colleagues at RAND, published in 2024. It enumerates 38 attack vectors and defines five security levels, from amateurs and opportunistic criminals up to SL5, resistance to the best-resourced state actors.

The recommended controls are conventional. Centralise the copies, cut the number of people with read access, run an insider threat programme, adopt confidential computing, buy real red teaming. The first of those controls presumes you know how many copies exist, and model fragmentation is why teams do not.

One control does depend on a property of the asset. RAND assumes egress can be sized so that moving a full copy out takes impractically long. In Aggressive Compression Enables LLM Weight Theft, a preprint posted on 3 January 2026, Brown, Juan-Pablo Rivera, Dan Hendrycks and Mantas Mazeika attack that assumption. Drop the requirement that compressed weights still run efficiently, allow the attacker to spend compute on fine-tuning after delivery, and compression improves by an order of magnitude over inference-oriented quantisation. They report 16× compression of Qwen2-1.5B at what the paper calls a 1.6% reduction in MMLU accuracy.

Their headline is that months become days. Here is what the number assumes.

The attacker has already compromised the inference server and holds unencrypted weights. He is not a physical insider, so he cannot use a USB drive and must move data over the network. Deep packet inspection forces him to hide the payload inside legitimate model outputs through steganography, spread across fake accounts. The defender has a fixed 0.1% daily chance of detection.

The timeline is calculated for a hypothetical 1.8-trillion-parameter model, and the 160× figure quoted for Llama-3-70B is a rate-distortion projection the authors state they did not evaluate end to end.

The direction of the finding survives all of that. Egress sizing is a weaker control than the RAND playbook assumed, and it protects a large model less than a small one, because compression improved with size across their experiments.

Their defences teach more than their attack does. Fine-tuning weights to resist compression bought 3–10%, which is nothing. The moving-target defence, rotating weights through function-preserving symmetries so an attacker cannot reassemble fragments, fails inside the same paper to canonicalisation through a singular value decomposition.

Forensic watermarking held. A 128-bit payload spread across a single layer, computed in 0.1 seconds on an A100, survived 1.2 million tokens of supervised fine-tuning with a raw bit error rate under 3%, from which the error-correcting code recovered the payload intact. A recovered payload proves where a leaked model came from, after it has already left. The authors did not test adversarial removal, so the robustness claim covers ordinary fine-tuning and no more.

No prosecution has yet involved frontier model weights. Ding took documents about the hardware, and the trial judge then threw out the state-benefit theory. Nation-state weight exfiltration is a serious threat model and frontier labs are right to plan against it. It is also, so far, entirely prospective. The controls it justifies are expensive, which makes that a budget question and not only a definitional one.

What the defences survive

Output perturbation

Output perturbation is moving from research into deployment: modify what the teacher returns so the outputs act as poisoned training data for a student while staying useful to paying customers. The reference method is antidistillation sampling from Yash Savani and colleagues at NeurIPS 2025, which biases the teacher’s next-token distribution toward tokens that raise a proxy student’s loss. In the high-utility regime, as the ETH Zurich group below characterises the curve, the trade is roughly 40% student degradation for a 20% drop in teacher accuracy on GSM8K.

In a June 2026 preprint, What Does It Mean to Break a Distillation Defense?, Lena Libon, Pura Peetathawatchai, Michael Aerni, Daniel Paleka and Florian Tramèr at ETH Zurich re-ran that evaluation. They report three results, ordered here by how little work the attacker has to do.

The perturbed outputs contain long runs of repeated junk. Deleting repeated characters and n-grams before training the student costs no extra queries and requires no knowledge of the defence, and it stays inside the threat model the original paper assumed. That one step lifts student accuracy above the reported baseline across the whole high-utility range.

Doubling the query budget recovers most of what remains at the weaker perturbation settings. No reason ever existed to assume an attacker’s query budget equals the size of his prompt dataset.

Then the finding that should end the discussion. With those two moves already in place, prepending a single token to every generation lifts student accuracy to near the undefended baseline across the entire high-utility regime at the larger query budgets, because conditioning on one plausible token lowers the teacher’s entropy and the perturbation flips far fewer tokens. DeepSeek’s API documents response prefilling. Anthropic removed it from Claude Opus 4.6 and Sonnet 4.6.

The authors state the consequence plainly. A provider running that defence behind an API that exposes prefill gives an attacker near-complete recovery for the cost of typing one word, while believing itself protected, and that is worse than deploying nothing because it substitutes the appearance of protection for the substance.

Hiding the reasoning

The defence that did ship industry-wide is chain-of-thought suppression: return a short summary of the reasoning and withhold the trace. OpenAI does this for its reasoning models, Anthropic for Claude.

On 7 March 2026, Tingwei Zhang, John Morris and Vitaly Shmatikov posted the preprint How to Steal Reasoning Without Reasoning Traces, training inversion models that produce a full chain of thought from only the inputs, answers and summaries a target exposes. Fine-tuning Qwen-2.5-7B-Instruct on traces inverted from GPT-5 mini raised MATH500 from 56.8% to 77.6% and JEEBench from 11.7% to 42.3%, against fine-tuning on the answers and summaries alone.

In those experiments, hiding the reasoning raised the attacker’s cost without preventing the transfer. The capability is still in the answers.

What the providers actually relied on

None of it was a model-level defence. Anthropic tightened verification on the education, security-research and startup pathways most often used to open fraudulent accounts, built classifiers for coordinated multi-account activity and for chain-of-thought elicitation, and shared indicators with other labs and cloud providers.

OpenAI has required government-ID organisation verification for its most advanced API models since April 2025. It made Advanced Account Security mandatory for individual members of its Trusted Access for Cyber programme on 1 June 2026, and tightened that to hardware-backed passkeys on 1 September.

Both companies moved to identity. Verification does not make fraudulent account networks impossible, and Anthropic’s own numbers show one running past twenty thousand. It makes each replacement account more expensive to open, and a campaign on the scale Anthropic describes opens tens of thousands of them.

Where the taxonomy filed it

OWASP’s 2023 list for LLM applications carried Model Theft as LLM10. The 2025 edition deleted the category and folded extraction into Unbounded Consumption, alongside denial of service and the runaway inference cost OWASP calls denial of wallet. The 2026 edition, published on 4 August 2026, moved Unbounded Consumption from tenth place to sixth, on the strength of agent loops, retries and runaway inference cost. Extraction stays inside the category, and OWASP does not name it as the reason for the four-place rise. Check the edition before citing a number from any of the three.

For most teams building on somebody else’s model, that filing is correct. Your exposure is the bill and the outage, and a 16-million-exchange campaign against your provider looks like paying traffic.

For anyone whose model is the product, the filing is wrong, and the 2026 list says as much about itself. It is explicit that it ranks risks to models used as a component inside an application. It is not a threat model for the people who train them.

Legislation has reached the same stage. The Deterring American AI Model Theft Act of 2026 (H.R. 8283) was introduced on 15 April 2026 and ordered reported by the House Foreign Affairs Committee on 22 April by 43 votes to none. It would direct Commerce and State to identify foreign entities conducting model extraction attacks, publish a list, and consider both IEEPA sanctions and additions to the Entity List.

As of 5 September 2026 it has not passed the House, and no federal statute is specific to model extraction. The available tools are the Computer Fraud and Abuse Act, trade secret law, and a contract term in a provider’s terms of use, which is what carried the Ding prosecution and what carries every distillation dispute so far.

What this changes for a defender

Decide which of the three attacks you are defending against, because the controls do not overlap.

If you serve a model over an API and the model is the product, spend on identity and coordination detection before query caps. A rate limit constrains one account. Twenty-four thousand accounts behind a proxy reseller absorb the limit once and continue. Verified organisations, behavioural fingerprinting across accounts, and indicator sharing with the other providers an attacker is also querying all raise his cost.

If you ship a model onto hardware you do not control, assume the architecture is recoverable and design so the architecture is not the secret. TPUXtract needed no software vulnerability and no privileged account, but it did need the device, control of what runs on it, and a side-channel bench.

If you hold weights worth a nation-state’s attention, run an insider and infrastructure programme. Count who has read access. Size your egress deliberately and revisit the sizing as your models grow. Put confidential computing on the list.

Then plan for the day those controls fail. Of the three model-level defences Brown and colleagues tested, forensic watermarking is the one that held, and only against ordinary fine-tuning. Its entire value is proving where a model came from after somebody else has it. That capability is legal and commercial before it is technical, and it takes longer to build than any control above.

The 2019 version closed by saying models are both the product and the vulnerability. That was a phrase where a threat model should have been. The threat model is three problems, each with a different owner, and the smallest of them is the one the words “model stealing” describe.