Gradient-Based Attacks: Which Gradient, and Where It Stops Working

In 2023, Andy Zou and colleagues at CMU published Universal and Transferable Adversarial Attacks on Aligned Language Models, which introduced Greedy Coordinate Gradient and produced suffixes that jailbroke aligned models and transferred between them. It is the best-known gradient attack of the language model era.

It is also less of a gradient attack than the name suggests. GCG computes a gradient to decide which token substitutions are worth trying, samples a batch of candidates from that restricted set, evaluates them by running them through the model, and takes the one with the lowest loss. The gradient proposes; forward evaluations decide. Faster-GCG reports that the original method costs on the order of 256,000 evaluations per harmful behaviour, and measured the concordance correlation coefficient between the gradient’s predicted ranking of candidate swaps and their actual effect on the loss at 0.044, which is close to no correlation at all.

In a continuous input space the gradient hands you the attack. Everywhere else it proposes candidates and something slower does the work. That single difference explains why these methods dominate image benchmarks, why they underperform on text, and why they can only guide rather than drive an attack on the software artefacts most security teams are actually defending.

Which gradient

Training a model computes the gradient of the loss with respect to the parameters, and moves them to reduce it. That is optimisation, and it is not what an evasion attack uses.

A gradient-based attack computes the gradient of the loss with respect to the input, holding the parameters fixed, and moves the input to increase the loss. The model is frozen. Nothing is being trained. The earlier version of this article said the opposite, and that error propagates. If you think the attack works on training gradients, you conclude that a deployed model is safe once training is finished. It is not.

Computing an exact input gradient requires the forward and backward pass, which means the weights and the architecture. That is the white-box setting. Every method below is white-box in its native form. Black-box attackers have two routes rather than one: transfer from a surrogate they trained themselves, or estimate useful search directions from the target’s own responses using zeroth-order methods, as Ilyas and colleagues did at ICML 2018 under query-limited, partial-information and label-only access.

The family, and what each one added

Six methods account for most of the literature, and the differences between them are smaller than the naming suggests.

FGSM. Goodfellow, Shlens and Szegedy, 2014. One step in the direction of the sign of the input gradient, scaled by ε. Fast, weak, and still the reference point.

R+FGSM. Tramèr and colleagues, 2017. FGSM preceded by a small random step, which moves the evaluation point before the linear approximation is taken and reduces the failure of single-step attacks on sharply curved local loss surfaces.

FFGSM. Wong, Rice and Kolter, Fast is better than free, ICLR 2020. The previous version of this article got FFGSM wrong, and the error was substantial. FFGSM is not a feature-space attack and does not perturb intermediate layers. It is FGSM with uniform random initialisation over the perturbation ball and a step size larger than ε, and its purpose is adversarial training, not attack. The paper’s result is that FGSM with random initialisation trains models about as robust as PGD-based training at a fraction of the cost. Random-start FGSM reached 44.01% robust accuracy on CIFAR-10 at ε=8/255 against 45.80% for the PGD-7 comparator. Separately, with cyclic learning rates and mixed-precision training, an optimised FGSM configuration reached about 45% robust accuracy in six minutes. The same paper identifies catastrophic overfitting, the failure mode that had made earlier attempts at FGSM training look useless.

BIM / I-FGSM. Kurakin, Goodfellow and Bengio, 2016. FGSM applied iteratively with clipping back into the ε-ball at each step. Stronger than one step, which is itself an argument against the pure linearity explanation for why adversarial examples exist.

PGD. Madry and colleagues, ICLR 2018. Iterative gradient ascent from a random start inside the ball, projecting back after each step. Usually described as the strongest first-order adversary. It is also the attack that adversarial training is trained against.

C&W. Carlini and Wagner, IEEE S&P 2017. Rather than fixing ε and maximising loss, it minimises perturbation size subject to misclassification, using a change of variables and a margin-based objective. Slower, and it produces smaller perturbations than anything above.

PGD and C&W remain the foundational attacks, and neither is sufficient as a modern evaluation on its own. AutoAttack is the widely used standardised baseline because it bundles parameter-free APGD variants with FAB and Square Attack, whose failure modes differ, which removes the step-size tuning that made older evaluations flattering. Standardised is not definitive: RobustBench distinguishes its AutoAttack benchmark from adaptive evaluation, which by definition cannot be standardised.

Gradient masking is not robustness

The most useful thing to know about gradient attacks is what happens when they fail.

If PGD stops working against a model, there are two explanations. The model is robust, or the model has broken the attacker’s gradient without changing its decision boundary. Non-differentiable preprocessing, randomisation, and shattered or exploding gradients all produce the second outcome, and the second outcome looks exactly like the first on a results table.

Athalye, Carlini and Wagner named this in Obfuscated Gradients Give a False Sense of Security at ICML 2018. Of the nine non-certified defences accepted at ICLR 2018 that claimed white-box robustness, seven relied on gradient masking. They circumvented six completely and one partially. Defensive distillation, which the previous version of this article listed as a defence, is the canonical earlier case and had already been broken by Carlini and Wagner.

The diagnostic is straightforward and any team evaluating a model can run it. Four of Athalye and colleagues’ warning signs: an unbounded attack does not reach 100% success, single-step attacks beat iterative ones, black-box attacks beat white-box ones, or increasing the perturbation budget does not increase attack success. Any one of them points at a masked gradient rather than a robust model.

Text breaks the method

Two properties of language make the image recipe stop working.

The input is discrete. There is no small perturbation of a token sequence, and you cannot take a step of size α in a discrete space. GCG works around this with a trick from Ebrahimi and colleagues’ HotFlip. An embedding lookup is the same operation as multiplying a one-hot vector by the embedding matrix. The lookup itself is not differentiable; the multiplication is. So you take the gradient with respect to the one-hot representation and read off which substitutions might lower the loss. Then, because that estimate is noisy, you evaluate a batch of them for real.

Those forward evaluations are what GCG costs, and they are why the gradient signal does less than the method’s name implies.

The second property is that the output is inspectable. GCG suffixes are high-perplexity token salad, which makes them detectable by a perplexity filter without any understanding of the attack. Continuous methods such as COLD-Attack optimise over embeddings or token distributions and then project back, which produces more natural text but discards precision at the projection step.

The strongest current evidence on this is from The Attacker Moves Second, now a USENIX Security 2026 paper, which broke twelve published jailbreak and prompt-injection defences. The authors had gradient descent available and ran it against exactly one of the twelve, RPO. They wrote that gradient-based attacks on text remain unreliable, and recommended attacks operating directly in text space instead. A red-teaming competition with more than 500 participants reached 100% on the overlapping scenarios, where their automated search reached 69%.

Software breaks the gradient-to-input mapping

For the security detection case the obstacle is not the gradient itself but the mapping back.

You cannot apply a gradient step directly to a Portable Executable. Compute a useful direction in feature space and there is no procedure that turns the resulting vector into a file, because the mapping from artefact to features is generally neither invertible nor differentiable. Pierazzi and colleagues put it precisely: the search cannot be purely gradient-based.

Purely is the operative word, and I had this wrong in an earlier draft. Gradients still guide attacks in this domain. Kolosnjaji and colleagues attacked raw-byte malware classifiers in 2018 by projecting appended-byte padding onto the negative gradient while leaving the executable’s behaviour intact, and Pierazzi’s own Android attack is classified as gradient-driven in their own taxonomy. What changes is that the gradient can only point. A separate search over functionality-preserving transformations has to realise the move, and the side-effect features it introduces are not on the gradient at all. Model evasion in security detection covers that formalism and the constraints it imposes.

This is the cleanest illustration of a rule worth carrying: the reason gradient methods dominate the literature is that images make them cheap, not that images are where the risk is.

Reading a gradient-attack result

Four questions, and they apply to attack papers and vendor demonstrations equally.

Which gradient, and what did it cost? Exact input gradients need white-box access. If a result claims black-box success, ask whether it transferred from a surrogate or estimated directions from queries. Ask what the query budget was.

Was the attack adapted to the defence? A defence evaluated against stock PGD, by its own authors, tells you nothing. This is the ICML 2018 lesson and it has been relearned twice since.

Was gradient masking ruled out? Ask for the four checks above. Their absence is the most common reason a robustness number is wrong rather than merely optimistic.

What was the perturbation budget, on what dataset, under which attack? A robust accuracy figure without ε, dataset and attack is not a number. Adversarial training has the current leaderboard figures and what they cost to reach.

Twelve years after FGSM, the practical position is narrow and stable. Gradient methods are the right tool for evaluating a differentiable model on continuous inputs, and they are the standard by which image robustness claims are judged. Everywhere else they are one input to a search rather than the search itself. When a vendor tells you their model resists gradient-based attacks, the useful follow-up is not whether that is true. It is whether anyone attacking them would have relied on one.