Meta-Attacks: Using Machine Learning to Break Machine Learning

The Google Threat Intelligence Group (GTIG) published a report on adversarial AI use on 12 February 2026 that documents a campaign of more than 100,000 prompts against Gemini. The attackers wanted the model’s internal reasoning traces, which Gemini summarises before it returns them to a user. The attackers instructed the model to keep the language of its thinking content consistent with the language of the user’s input. Google detected the campaign in real time and reduced what the traces gave back.

That campaign is the largest publicly documented attempt to extract a frontier model through its own interface, and it used no gradients, no surrogate models and no learned attack generator. It used one instruction, repeated.

A deployed model’s exposure depends on what its endpoint returns per query and on whether its operator measures query distributions across accounts. Neither depends on the attacker’s mathematics.

What the term names

Four research groups have published different attacks under the name meta-attack. Three generate adversarial examples against a trained model and one poisons training data. The access they assume runs from output feedback alone to the victim’s labelled training graph.

Jiawei Du, Hu Zhang, Joey Tianyi Zhou, Yi Yang and Jiashi Feng published Query-efficient Meta Attack to Deep Neural Networks at ICLR 2020. A black-box attacker against an image classifier sees only output feedback, so estimating a gradient costs many queries, and query count is the quantity an operator’s rate limits and invoices measure. Their method meta-learns a gradient approximator across models the attacker already controls, then fine-tunes it against the target using a small number of real queries. The expensive learning happens on surrogates. The target sees a shorter sequence.

Daniel Zügner and Stephan Günnemann published Adversarial Attacks on Graph Neural Networks via Meta Learning at ICLR 2019, and the method is known as Metattack. It poisons training data for node classification instead of evading an already trained classifier. The authors treat graph structure as a hyperparameter and use meta-gradients to solve the bilevel optimisation problem underneath a training-time attack. Perturbing 5% of the edges in the CITESEER citation graph raised GCN misclassification from a clean rate of 28.5% to 34.6%, and the same perturbations degraded unsupervised node embeddings the attack never targeted. Building the surrogate requires the graph, the node attributes and the labelled subset, so the attacker needs the victim’s data even without the victim’s weights.

Weiwei Feng, Baoyuan Wu, Tianzhu Zhang, Yong Zhang and Yongdong Zhang published Meta-Attack: Class-Agnostic and Model-Agnostic Physical Adversarial Attack at ICCV 2021. This one is a physical attack, formulated as few-shot learning, that simulates colour and shape distortion so an adversarial example survives being printed and photographed.

Xiao Yang and colleagues published Model-Agnostic Meta-Attack in October 2021. It learns the attack optimiser itself, parameterised by a recurrent network trained across data samples and defences, to produce better update directions when generating adversarial examples.

Image evasion, graph poisoning, printed adversarial patches, and a learned optimiser. One name, four problems, and no shared threat model, target class or control. A CISO who receives a briefing about “meta-attacks” has been told nothing about which of these is meant, and the four call for entirely different answers: query monitoring, training-data integrity, physical inspection, and robustness evaluation.

What extraction costs

Nicholas Carlini and colleagues at Google DeepMind, ETH Zurich, the University of Washington, Google Research, McGill, Cornell and OpenAI published Stealing Part of a Production Language Model at ICML 2024. Their attack recovers a transformer’s embedding projection matrix, which maps the hidden dimension to the logit vector, up to symmetries.

Against OpenAI’s Ada and Babbage, the full projection matrix cost under $20 in queries, and the result confirmed hidden dimensions of 1024 and 2048. The same method recovered the hidden dimension of gpt-3.5-turbo, and the authors estimated under $2,000 to recover its projection matrix. OpenAI granted advance permission before the test ran.

Whether this transfers depends on the API the next operator exposes. The attack needs top-k log probabilities and a logit bias parameter, and it exploits the low rank of the final layer. It recovers one layer of a model that has many. Nobody reconstructed ChatGPT for twenty dollars, and the paper makes no such claim. OpenAI and Google both restricted the combination of log probabilities and logit bias after the disclosure.

Knowledge distillation needs none of those API features. An attacker queries a mature model across a target domain, then runs supervised fine-tuning on the responses to produce a student model. Its economics turn on the number of teacher queries, the domain and the cost of training the student, which is why GTIG describes extraction as a way to reach a capability faster and cheaper than developing it. Google sells distillation as a tuning technique, so permission separates the product from the attack. Distilling Gemini without it breaches Google’s terms of service.

Both attacks pursue the model itself. Model inversion and membership inference pursue training-data privacy: whether a given record was in the training set, and what can be reconstructed from what the model absorbed. Same endpoint, a different victim, and controls that protect the training set instead of the weights.

Who ran the attacks

GTIG’s attribution finding matters more to a defender than its volume figure. During 2025 the group observed no direct attacks on frontier models from any advanced persistent threat or information operations actor it tracks. The model extraction it detected and mitigated came from private sector entities around the world and from researchers cloning proprietary logic.

State actors appear throughout the same report, and none of them were attacking models.

GTIG documents APT42 using Gemini to enumerate official email addresses for named entities and to build pretexts for social engineering, UNC2970 profiling defence and cybersecurity companies for recruiter impersonation, and APT31 prompting Gemini under a security researcher persona to analyse remote code execution results, web application firewall bypass techniques and SQL injection tests against specific US targets. That is reconnaissance and tooling support. GTIG’s own assessment is that it has not yet seen APT or information operations actors achieve breakthrough capabilities that alter the threat landscape.

One product named in the same report tells anyone evaluating claims in this market what to expect. Xanthorox is sold on underground forums as a bespoke, privacy-preserving, self-hosted AI for autonomous malware and phishing generation. GTIG’s investigation found it running on several third-party commercial models including Gemini, assembled from open-source components including Crush, Hexstrike AI, LibreChat-AI and Open WebUI, connected through Model Context Protocol servers. The custom offensive model did not exist.

Eric Doerr, chief product officer at Tenable, made the corresponding prediction for the defensive market at the end of December 2025: AI lowers the cost of attack generation and raises volume, and it might find a zero day, but it is not producing novel attack techniques. Six weeks later GTIG reported the same pattern from telemetry.

Simple attacks beat sophisticated ones more often than the literature suggests

Maksym Andriushchenko, Francesco Croce and Nicolas Flammarion published Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks at ICLR 2025. Their method combines a hand-written prompt template with random search over an adversarial suffix, maximising the log probability of a target token such as “Sure”. They credit random search to Rastrigin, in 1963. The method uses no gradient information even where the weights are open, and it calls no auxiliary language model to iterate the jailbreak.

It reached a 100% attack success rate on Vicuna-13B, Mistral-7B, Phi-3-Mini, Nemotron-4-340B, Llama-2-Chat at 7B, 13B and 70B, Llama-3-Instruct-8B, Gemma-7B, GPT-3.5, GPT-4o, and R2D2 from HarmBench, which had been adversarially trained against the GCG attack specifically. Claude models do not expose log probabilities, so random search does not apply to them. They fell to transfer from another model, or to prefilling, an API feature that lets the caller specify the opening tokens of the response.

Two conditions belong with that number. A GPT-4 judge scored success, which is a weaker evidentiary standard than human review of each transcript. The requests came from a benchmark of harmful behaviours, not from production traffic. What survives both caveats is the comparison the authors ran: the methods this one outperformed were the gradient-based attacks and the LLM-driven iterative ones.

Microsoft’s AI Red Team reached the same conclusion from operations instead of from a benchmark. Its January 2025 paper Lessons From Red Teaming 100 Generative AI Products lists eight lessons, and the second is that you do not have to compute gradients to break an AI system. The team formed in 2018 around evasion attacks on classical ML models and found that simple prompt-based attacks produced more real findings against deployed LLM systems. The paper cites Apruzzese and colleagues under the title Real attackers don’t compute gradients.

Athalye, Carlini and Wagner showed in 2018 that seven of nine defences against adversarial examples accepted at ICLR that year relied on obfuscated gradients, and attacks built specifically against each one broke six completely. The lesson from that work, and from the 2018 result on adaptive attacks against intrusion detection, is that a defence has to be tested against an attacker who has read its design. Mathematical sophistication is not a reliable proxy for operational effectiveness, and a defensive programme organised around gradient methods and learned attack generators will miss the attacks that work.

A model is not a system, and neither is an agent

Anthropic disclosed in November 2025 that a Chinese state-sponsored group it tracks as GTG-1002 had used Claude Code in an espionage campaign. According to Anthropic, the campaign targeted roughly thirty entities across technology, finance, chemical manufacturing and government, and the model executed 80 to 90% of tactical operations, with human operators intervening at decision points such as the scope of exfiltration. Anthropic reports that the model’s hallucinations, including overstated findings, constrained the operation. No victim has confirmed those figures independently.

Claude Code attacked the victims’ networks and the services running on them, and no model was a target anywhere in the campaign.

That campaign is an agent driving conventional intrusion tooling faster than a human team, and it requires a different threat model from everything else described here. A finding about a model does not automatically transfer to the system deployed around it, and an agent holding credentials and tools reaches further than either. Extraction targets the model. Prompt injection becomes a system problem when untrusted instructions reach a model that can call tools or read data, which is why the OWASP LLM Top 10 2026 still ranks it LLM01, as it has in every edition. GTG-1002 was an agent problem. A language model appears in all three, which is not enough to make them one problem.

What to check on your own endpoint

Two deployment properties carry disproportionate weight here, and an operator can inspect both without buying anything.

What the endpoint returns per query. Full probability distributions, top-k log probabilities, logit bias parameters, distance-to-boundary values and unsummarised reasoning traces each increase the information an attacker extracts per query, which reduces the query count an extraction attack needs. This is not a fringe view: OWASP’s own mitigation list for the risk names restricting or obfuscating logit_bias and logprobs in API responses, and cites the Carlini paper. Returning hard labels or top-1 predictions raises the query count and lowers the fidelity of the copy, and researchers have still demonstrated extraction against top-1-only APIs, so treat it as a cost imposed on the attacker and not as a control. Audit what your API returns before buying anything that claims to detect what is done with it.

Whether query distributions get compared across accounts. Extraction queries draw on natural text sources including Wikipedia-derived passages, benchmark-style prompts and domain knowledge questions, so each query on its own looks unremarkable, and a detector that scores queries one at a time cannot use cross-query structure by construction.

Shuze Liu, Qianwen Guo and Yushun Dong put numbers on the alternative in a June 2026 preprint. They embed incoming queries with a sentence encoder and run a maximum mean discrepancy test against a historical baseline of benign traffic, calibrating the alarm threshold on comparisons within that baseline alone, so a defender running it needs no labelled attack data. Across fourteen attacker-normal query pairs from four extraction families it reported a 0.3% benign false positive rate and 100% detection of pure attacker traffic.

The parameter a defender controls is the window. In their sensitivity analysis on five representative pairs, detection at a 5% attacker fraction rose from 1.2% with 100-query windows to 74.4% with 1,500-query windows, and at a 10% fraction from 4.8% to 93.6%, with the false positive rate between 0.0% and 1.2% throughout. Attackers dilute by inserting benign cover queries, slowing extraction down, or spreading queries across accounts, and all three defeat a detector that examines one account at a time.

The same paper should change how the older defences are cited. When Liu and colleagues re-ran PRADA, published by Juuti and colleagues in 2019 and still the reference for extraction detection, under a text-query embedding protocol, it alarmed on 51.6% of benign user streams while catching 49.0% of attacker streams. SEAT, from Zhang, Chen and Wagner in 2021, gave 14.3% false positives and 15.0% detection under the same transfer. Both performed well in the image-query settings they were built for. Neither survives the move to natural-language traffic unchanged, and this is a preprint on fourteen constructed query pairs rather than a production log, so treat its numbers as a direction and not a specification.

One framework correction belongs in any procurement conversation on this subject. OWASP’s numbering has moved twice. In the 2023 list, LLM10 was Model Theft. In the 2025 list, LLM10 became Unbounded Consumption, which absorbed model theft and names both model extraction via API and functional model replication among its examples. In the 2026 edition, published 4 August 2026, Unbounded Consumption moved to LLM06 and LLM10 became Improper Output Handling. A vendor mapping extraction to LLM10 is therefore working from one of two superseded lists, and you learn nothing about your own risk from the number it cites. Rank it against your own endpoint instead, and if you serve inference to untrusted callers, read the extraction question alongside query attacks and model stealing.

The meta-learned attack generator has been demonstrated only in four conference and preprint papers that do not agree on what they are attacking. The attack that worked against Gemini was one instruction about language consistency, sent a hundred thousand times, and Google published the telemetry.