Model Evasion: Why Attackers Rarely Need Adversarial ML

In 2020, Fabio Pierazzi, Feargus Pendlebury, Jacopo Cortellazzi and Lorenzo Cavallaro published Intriguing Properties of Adversarial ML Attacks in the Problem Space at IEEE S&P. They generated thousands of working Android applications that evaded a state-of-the-art malware classifier and its hardened version, at a rate of a few minutes per application, across a dataset of 170,000 apps from 2017 and 2018. Their phrase for the result was that adversarial-malware as a service is a realistic threat.

In the same paper they counted the literature. Of more than 1,600 adversarial machine learning papers published over the previous six years, roughly 40 addressed malware, and many of those never left the feature space.

That ratio is the story of this whole subfield. The research effort has concentrated on a setting where perturbations are easy to define. The deployments that matter to a security team are the ones where they are not.

Evading a deployed ML detector is a harder engineering problem than evading a classifier in a paper, and that difficulty is the most underused defensive asset most security teams have. It is also why the honest answer to “are attackers using adversarial ML against our detection stack” is usually no. That answer is not a reason to stop reading.

Feature space and problem space

An image classifier takes pixels. To attack it you add a small number to each pixel, and any real-valued vector inside the perturbation budget is a legal input. The gradient-based methods that dominate the literature work because that mapping runs both ways: pick a point in feature space, and there is a corresponding image.

A malware detector takes a file. You cannot add 0.03 to a Portable Executable. Moving a sample in feature space means editing the file itself, and the result still has to run. Pierazzi and colleagues formalised what that requires, and the constraints are worth stating in full because every one of them costs the attacker something:

• Available transformations. What can you actually do to the object. Append bytes, add a section, insert dead code, repack, add a permission, pad a packet.
• Preserved semantics. The malware still has to work. A sample that evades detection and no longer executes its payload is not an evasion, it is a broken file.
• Robustness to preprocessing. The transformation has to survive the non-ML steps applied before the model sees anything: dead code elimination, compression, normalisation. Evading a classifier with a change that a preprocessing pass strips out is not an evasion.
• Plausibility. The result has to look like a thing that occurs in the world, because a human analyst may look at it.

The feature mapping in software is neither invertible nor differentiable: no procedure takes an adversarial feature vector and hands back the file that produces it. A purely gradient-based approach therefore does not carry over, and the search has to happen in the space of transformations rather than the space of features.

What this does to published numbers

A feature-space attack perturbs the vector directly. That is a legitimate way to locate a decision boundary, and it tells you nothing about what an attacker can build.

The error runs both directions, which is what makes it interesting. A feature-space result can overstate the threat, by reporting evasion through a perturbation no attacker could realise in a working binary. It can also understate it, because problem-space transformations produce side-effect features the attacker did not intend and may move the sample further than the minimal perturbation would suggest.

So when a paper or a vendor reports an evasion rate against a security classifier, the question is not only the one the white-box versus black-box article asks about access. It is also: which space did you measure in, and did the resulting artefact still function. A robustness number without that answer is not comparable to anything.

TESSERACT, from some of the same authors, showed that malware classification results are routinely inflated by two experimental biases. Temporal bias is training and test data that are not separated in time; spatial bias is a malware-to-goodware ratio in the test set that does not match what the deployment sees. A classifier evaluated without those controls reports a robustness it does not have, before any adversary shows up.

Real attackers use simpler tactics

The most useful paper in this area is a position paper. In 2023, Giovanni Apruzzese, Hyrum Anderson, Savino Dambra, David Freeman, Fabio Pierazzi and Kevin Roundy published “Real Attackers Don’t Compute Gradients” at SaTML, drawing on work at Meta, Norton, Robust Intelligence, the University of Liechtenstein and King’s College London. Their finding is that real-world evidence shows attackers subverting ML-driven systems with simple tactics, and that security practitioners have accordingly not prioritised adversarial ML defences.

Their claim is about cost, not about whether the attacks work. An attacker choosing how to get past your detection stack has a menu, and gradient-based evasion of the classifier is one of the more expensive items on it. Packing, obfuscation, binaries that already ship with the operating system, a stolen certificate, a fresh domain, or simply a payload the model has not seen are often cheaper, and several of them defeat the detector without anyone needing to know it is a detector.

The other half of their argument is the one this site keeps returning to: attackers interact with ML systems, not ML models. The model is one component behind ingestion, feature extraction, thresholds, allowlists, reputation services, sandboxing and an analyst queue. An attack that flips the model’s output and gets caught by the reputation service has not evaded anything.

For a defender this cuts a specific way. The right question is not “is my model adversarially robust.” It is “what does it cost an attacker to get a working payload past the whole pipeline, and is the model the cheapest thing to attack in it.” Usually the model is not the cheapest thing to attack. Where it is, that is a finding worth acting on.

Where evasion is a live concern

Three deployment shapes make the model the cheap target rather than the expensive one.

Detectors with a query channel. If an attacker can submit samples and observe verdicts, they can map the boundary. Uploading to a multi-engine scanning service, testing against a free tier, or watching whether an email is delivered are all oracles. Evasion converges here with model extraction and with query-based attacks. Queries buy two different things: surrogate extraction followed by an offline white-box attack, and direct score-based or decision-based optimisation against the target itself. Query-budget accounting and stateful defences that track sequences of related submissions, rather than scoring each in isolation, are a control that matches this threat, though distributed identities, public third-party scanning services and low-and-slow probing all blunt it.

Detectors that retrain on their own outputs. Where attacker-influenced verdicts or telemetry are fed back into subsequent training, the system exposes a poisoning surface alongside the evasion one, which is a different and generally worse position. The old version of this article filed analyst mislabelling under evasion. That was wrong. An attacker who influences what your model learns is not evading it, they are rewriting it.

Phishing and content classification. Apruzzese, Conti and Yuan’s SpacePhish work at ACSAC 2022 mapped the evasion space for phishing website detectors specifically. That domain constrains an attacker far less than malware does: a webpage only has to render and deceive a human, while a binary still has to execute its payload.

Network intrusion detection constrains an attacker more than phishing does and less than malware does. The adaptive attacks against IDS article covers those domain constraints. A packet-level perturbation has to remain valid traffic that still accomplishes something, which puts it closer to the malware case than the phishing one.

The LLM turn, and what it does not change

Two things arrived with language models and they pull in opposite directions.

Detection stacks now include LLM components: triage assistants, classification by prompt, summarisation of alerts, agents that pull enrichment. Those inherit the whole prompt-injection surface, and content an attacker controls that reaches a triage model is an instruction channel, not just a sample. That is genuinely new attack surface and it is not adversarial ML in the perturbation sense at all.

On the offensive side, malware that calls an LLM during its own execution is now documented rather than speculative. Google’s Threat Intelligence Group reported on 5 November 2025 that PROMPTFLUX, a VBScript dropper, queries the Gemini API for obfuscation and evasion techniques and rewrites its own source to defeat static signature detection, and that PROMPTSTEAL, a data miner used by APT28 against Ukraine, queries a Qwen model through the Hugging Face API to generate its commands at runtime. GTIG describes PROMPTSTEAL as its first observation of malware querying an LLM in active operations, and assesses PROMPTFLUX as experimental, currently lacking any means to compromise a device.

Hold the distinction that matters. That is AI-assisted evasion engineering: a model generating obfuscation faster and more variably than a person would. It is not adversarial machine learning. Nothing in it locates or crosses a specific classifier’s decision boundary, which is the subject of this article, and there is still no public evidence of routine problem-space optimisation against deployed detectors.

What has not changed is the constraint. An LLM writing malware faces exactly the same problem-space requirements as a human writing it. The payload still has to execute.

What to do on Monday

Find out whether your detection models have a query oracle, and what it costs to use it. This is the single highest-value question in this article. If an attacker can test against your detector for free and without limit, you have handed them the offline attack. Rate limits, submission-history tracking and per-identity budgets are ordinary controls that apply directly.

Check whether anything retrains on data an outsider can influence. Analyst feedback, customer submissions, telemetry, public feeds. If yes, that pipeline needs poisoning controls, not evasion controls.

Demand problem-space evaluation from vendors. Ask whether their robustness numbers came from perturbing feature vectors or from producing artefacts that still functioned. Ask about temporal and spatial split discipline in the same breath. Most will not have the answer, which is itself the answer.

Do not buy adversarial robustness for the classifier before you have costed everything else in the pipeline. Adversarial training is real and expensive in both compute and clean accuracy. It defends the layer that is usually not the cheapest one to attack. Spend there when you have evidence that the model is the weak link in the pipeline, and not before.

The two failure modes here are symmetrical. One is the vendor selling robustness against an adversary who is not attacking that layer. The other is the team who read that real attackers do not compute gradients and concluded that nobody ever will. The Pierazzi result says the attacks are constructible at scale and cheaply. The Apruzzese result says attackers currently have better options. Both are true, and the second one holds only while everything else in the pipeline stays more expensive than the model.