Query Attacks Buy Reconnaissance

Anthropic reported in February 2026 that three AI labs had generated more than 16 million exchanges with Claude through roughly 24,000 fraudulent accounts, across three campaigns aimed at copying its capabilities.

NIST puts recent black-box evasion attacks at typically fewer than 1,000 queries.

Those two figures measure different things. One is observed campaign volume across three real operations, the other a per-attack budget from the research literature, and the ratio between them means nothing. What they establish together is that one interface serves attacks whose prices differ by orders of magnitude and whose consequences have nothing in common.

If you serve a model over an API, your query defences are probably sized for the expensive attack and blind to the cheap one.

A frontier lab plans against the first price. Everybody else should plan against the second, because no competitor will spend sixteen million exchanges on a mid-size fraud classifier. Sub-thousand-query evasion applies to every deployment, and it matters most where the model makes a security decision: fraud scoring, content moderation, malware triage, anomaly detection, identity checks.

Query access is one capability among six

NIST’s taxonomy of adversarial machine learning, edition AI 100-2e2025, lists query access as one of six attacker capabilities in its predictive-AI section, beside control of the training data, the model, the testing data and the source code, and a label limit. The attacks themselves get sorted by goal: availability breakdown, integrity violation and privacy compromise, with misuse violations added for generative systems. NIST files model extraction under privacy compromise, as NISTAML.031.

Three families of attack need that capability and nothing else. NIST names black-box evasion, energy-latency attacks against availability, and every privacy attack that works without knowledge of the model’s training data, architecture or parameters.

Three of those reach a deployed model in ways a defender has to tell apart.

An attacker can build a functional copy. Tramèr and colleagues demonstrated that at USENIX Security 2016 against the prediction APIs of BigML and Amazon Machine Learning. The attack has since divided into three problems with separate defences, and I cover them in model stealing.

An attacker can recover information about people in the training data. Model inversion and membership inference do that, and they harm the data subject.

An attacker can find an input the model classifies wrongly. Evasion does that, and it is the oldest use of query access.

Those three differ in what the response must reveal and in how many requests they cost. They differ again in who suffers. No single defence covers all three, and the 2022 version of this post assumed one would.

The cheap end of the same API

Access level sets the budget. Score-based attacks read confidence scores or logits. Decision-based attacks read the predicted label and nothing else. NIST’s own summary of the black-box evasion literature puts recent techniques at fewer than a thousand queries, and its examples are Bayesian-optimisation methods that assume no knowledge of the model at all.

Vo, Abbasnejad and Ranasinghe published SparseEvo at ICLR 2022, an evolutionary algorithm that works from the top-1 label alone. Their ImageNet illustration perturbs 78 pixels out of 50,176, which is an example in the paper and not an aggregate result. The same authors’ BruSLeAttack at ICLR 2024 compares itself at 500 queries against SparseEvo at 2,000, for targeted attacks on a ResNet18 trained on CIFAR-10 across 9,000 pairs of source image and target class. BruSLeAttack reads confidence scores and SparseEvo works from labels, which the authors give as the reason for the gap. Both methods come from the same research group, so the margin is a self-comparison and only the direction is safe to quote.

One published result runs against production APIs. Hayase, Borevkovic, Carlini, Tramèr and Nasr published Query-Based Adversarial Prompt Generation at NeurIPS 2024. They observed that each iteration of the GCG attack divides into two stages: a gradient filter narrows a large candidate set, then query access picks the best candidate from the shortlist. Substitute a surrogate model the attacker owns for the gradient filter, and the second stage needs nothing but the target’s API. They report evading OpenAI’s and Llama Guard’s safety classifiers with close to 100% probability. Part of their access assumption has since closed, because one documented technique required OpenAI to return top-5 log probabilities while honouring a logit bias, and the paper records that OpenAI stopped letting logit bias affect the returned tokens in March 2024. The surrogate-plus-query structure never depended on that feature.

Real triage queues look different from all of that. NIST’s report describes a commercial phishing-page detector studied by Apruzzese and colleagues, where 4,600 samples reached manual triage and 100 turned out to be adversarial examples. None of the hundred used an optimisation-based attack. The attackers cropped, masked and blurred images. NIST also records more than 80,000 attempts to fool the ID.me face verification service in the second half of 2020, using masks, deepfakes and photographs of other people.

Published query budgets tell you what a researcher can do. A triage queue tells you what attackers currently bother to do. Confusing the two produces either panic or complacency.

Extraction as the step before evasion

NIST states the connection plainly. Model extraction is often not an end goal but a step toward other attacks, and preventing it mitigates whatever depends on knowing a model’s architecture and weights.

Juuti, Szyller, Marchal and Asokan gave both motives when they opened PRADA at EuroS&P 2019: the model gives its owner a business advantage, and an adversary may use a stolen model to find transferable adversarial examples that evade the original. Vendors, litigators and legislators all adopted the first. It supports a press release, a lawsuit and an export-control argument. The second changes what a security team does on Monday.

An attacker gets a surrogate and never the model. NIST reports Jagielski and colleagues proving that no attacker can recover a model exactly, and that building a functionally equivalent one is NP-hard. A surrogate suffices, because an adversarial example that transfers does not require a faithful copy.

Every candidate input tested against the remote model costs a request that a defender can price, throttle and log. Every candidate tested against a local surrogate costs a gradient computation on the attacker’s own hardware. Extraction removes most of that search from the monitored interface, and the literature calls the result a surrogate-model attack. Juuti and colleagues scored their own extraction by how well its adversarial examples carried over, reporting improvements of 29 to 44 percentage points over prior work, on attacks aimed at a specific class and on attacks aimed at any misclassification.

Two limits apply. Transfer is imperfect, so an attacker who wants confidence returns to the target to validate candidates. Substitute-model training in the Papernot line queries the target anyway. A surrogate removes most of the search from the metered interface without making the search free.

An organisation whose model is the product suffers that chain as a secondary harm, after losing the asset itself. An organisation whose model is a control suffers it as the whole attack. A spam filter, a transaction scorer and a moderation classifier each exist to make a decision an adversary wants changed, and for those systems the copy is the reconnaissance. I have written about the payload end in adversarial attacks and perturbation attacks.

What the training data leaks

Recovering training data through queries is the third goal, and the evidence has moved against the alarming version of it. Duan and nine co-authors ran a large-scale evaluation at COLM 2024 across Pythia models from 160M to 12B parameters trained on the Pile. Membership inference barely beat random guessing, with no attack or target model above 0.6 ROC-AUC except on the GitHub domain.

Their explanation concerns how pretraining works. A large language model sees an enormous corpus for few passes, so a document it trained on looks much like one it never saw. Where published results did show strong membership inference, the authors trace the effect to distribution shift, usually because the non-members came from a later time period than the members.

Smaller models trained for many epochs on narrow data stay exposed, and a medical or financial classifier can occupy that regime. Model inversion covers what comes out and under what access, and differential privacy is the formal defence with a proven bound against membership inference and reconstruction.

NIST is explicit about where that defence stops. Differential privacy protects the training data and not the model, and it offers no guarantee against extraction. Extraction targets the model, inference attacks target the people in its training data, and the control for one does nothing for the other.

Query-pattern detectors watch the queries

Detection is the common answer for extraction, and it assumes the attacker’s queries look different from everybody else’s.

PRADA analyses the distribution of distances between consecutive queries from one account and raises an alarm when that distribution departs from benign behaviour. Its authors reported detecting every prior extraction attack with no false positives.

The same paper introduced the attack that defeats it. Under the Dummy Query strategy the adversary injects benign samples among the extraction queries to keep the inter-query distance distribution looking normal, and those samples contribute nothing to the surrogate. A later detector, FDINet, measured what that evasion costs an adaptive adversary who knows the detection threshold: between 252% and 581% more queries. The detector raised the attacker’s query budget by a single-digit factor and did not stop adaptive evasion in that experiment.

A June 2026 paper on extraction detection in LLM API traffic names three gaps between the published detectors and production traffic, and proposes its own answer to the last one. Researchers evaluate detectors one account at a time, with each benign user issuing only legitimate queries and each attacker running a complete extraction workflow, so nothing tests the mixed case. Aggregate monitoring sees attacker requests as a small fraction of a large traffic window. Defenders hold historical benign traffic and no attacker query generator, which is what calibrating a low false-positive threshold has traditionally required.

Then there is the case with no queries to detect.

Panfilov and colleagues published an extraction attack against reasoning APIs on 10 August 2026. Providers return chain-of-thought to the client as an encrypted block. The client passes it back with each subsequent request, and those blocks proved interchangeable across sessions, users and models within a single provider. The paper describes four attack vectors, and one of them needs no contact with the target at all. The authors scraped 6,708 agent trajectories from GitHub and Hugging Face, decoded 315,320 reasoning blocks by replaying them into a cheaper model in the same family, and recovered 367 personally identifiable artefacts and 182 credentials. From genuine user sessions alone the recovered secrets included 62 API keys and 33 passwords.

In that variant the attacker never queries the frontier endpoint. Somebody else generated the expensive reasoning and published it, and the authors decode it using a cheap sibling model. Query-pattern detection at the target API has nothing to record. The paper’s other vectors do query the target, so the property belongs to the public-trace attack and not to the whole paper.

Two caveats limit the result. The authors disclosed to the affected providers first, and their reproducibility statement records that the attack stopped working in August 2026 after provider mitigations. Their extraction is also approximate, verified by comparing decoded token counts against the API’s own reported thinking-token counts, with no ground-truth trace available.

NIST reaches the same conclusion about this whole family in one sentence. Limiting user queries, detecting suspicious queries and hardening architectures against side channels can all be circumvented by motivated and well-resourced attackers, and the report says they should be used with caution.

What this changes for a defender

Decide first whether your model is a product or a control, because the answer puts query attacks in a different threat model.

If the model is the product, your exposure is commercial. Model stealing covers the controls: identity verification, correlation across accounts, and provenance evidence for the day a copy appears somewhere else. An attacker defeats a per-account query cap by opening more accounts, and the campaigns Anthropic reported used roughly 24,000 of them.

If the model is a control, move query attacks out of the intellectual property register and treat extraction as reconnaissance for evasion. That is my one recommendation here, and three things follow from it.

Re-run the robustness evaluation at a budget an attacker would actually spend. Published adversarial robustness numbers often assume a budget of several thousand queries or more, while NIST’s own reading of the black-box literature puts successful evasion under a thousand. A model that fails at 500 queries teaches you nothing useful when you test it at 10,000.

Count what each response gives away. In the one published head-to-head above, moving from labels to confidence scores cut the required budget by a factor of four. A top-k list, a ranked candidate set, a rationale string and an opaque continuation token each give the attacker information he did not have before, and returning less is the cheapest hardening available.

Treat your own logs as inputs. Panfilov and colleagues recovered credentials from traces that developers published deliberately, after sanitising the parts they could read. Whatever your team commits to a public repository, ships with a benchmark or attaches to a bug report is available to an adversary at no query cost.

Keep the rate limits, and describe them accurately. A rate limit raises the price of a query campaign. It does not detect one, and it stops little against an attacker who needs a few hundred requests. The taxonomy of attacks separates extraction from evasion for good reason. The interface is what they share, and few teams instrument it.