Saliency Attacks Are Two Attacks, and the Second One Forges Evidence

In 2016, Nicolas Papernot and five co-authors published an attack that identifies which input features a classifier is most sensitive to and changes only those. The Limitations of Deep Learning in Adversarial Settings, presented at IEEE EuroS&P, computes the Jacobian of the network’s output with respect to its input, scores features by how much moving them pushes the prediction toward a class the attacker chose, and iteratively perturbs the best-scoring pair. On MNIST the method produced the chosen target class 97% of the time while altering 4.02% of input features per sample. It needs the gradients, so it is white-box.

Papernot’s method is the Jacobian-based Saliency Map Attack. The phrase “saliency attack” also names a second and unconnected line of work, in which saliency is the thing under attack. Amirata Ghorbani, Abubakar Abid and James Zou showed at AAAI 2019 that two images a human cannot tell apart, both classified identically and correctly, produce completely different feature-importance maps.

An attribution score used as evidence, whether for model debugging, an audit response or an adversarial-input alert, is an artefact an attacker can set to a chosen value while the prediction stays correct. I have found no conformity assessment procedure and no harmonised standard that tests it for integrity.

Two attacks, one name

Which sense of the term is meant decides which defence applies.

In the JSMA family, saliency gives the attacker a cheap perturbation. Papernot built an L0 attack when most of the field was producing L∞ and L2 ones. The saliency map picks a small set of pixels and the attack moves those far, leaving every other pixel unchanged. Adversarial training answers this attack partially, and the ordinary robustness literature covers it. Work continues. FastJSMA, presented by Zhenghao Gao and colleagues at ICCV 2025, replaces the full Jacobian with two cheaper gradients, one suppressing the current class and one exciting the target, which cuts the cost on datasets larger than MNIST.

In the second family the explanation is the target and the prediction stays correct, which is what hides the attack from anyone watching outputs. A model can pass an adversarial-robustness evaluation and still produce a forged explanation.

The 2021 version of this article defined saliency attacks in the first sense, then recommended defences that only work if the second sense does not exist. Monitoring saliency maps for unexpected change assumes the map reports honestly on the model. Maximilian Noppel and Christian Wressnegger’s SoK at IEEE S&P 2024 systematises the reasons it does not.

Forging an explanation

Noppel and Wressnegger sort these attacks by where the adversary intervenes: the input, the model, or the whole system.

Change the input

Ghorbani, Abid and Zou tested simple gradients, integrated gradients and DeepLIFT on ImageNet and CIFAR-10. Their mass-center attack, run at an L∞ budget of 8, produced perturbed images whose predicted label was unchanged and whose feature-importance maps moved to regions no human would call salient. Random sign perturbation at the same budget already left under 30% overlap in the top-1000 most salient pixels across all three methods. Integrated gradients was the hardest of the three to attack. They extended the finding to exemplar-based interpretation, showing that influence functions fail the same way.

The authors are careful about what this demonstrates. A saliency map measures the sensitivity of the network at a particular input. After a perturbation it measures the sensitivity at the perturbed input, which is what it was built to do. The fragility belongs to the network, and the method is reporting it faithfully.

Ann-Kathrin Dombrowski and colleagues published a stronger version at NeurIPS 2019. Explanations can be manipulated and geometry is to blame drives the explanation to a target map the attacker chooses while holding the network’s output approximately constant. Their Figure 1 uses a target that spells out the words “this explanation was manipulated”. They identified a geometric mechanism behind it: ReLU networks have high output-surface curvature, and gradient-based explanations inherit that sensitivity. Replacing ReLU with SoftPlus reduces the curvature and raises the attacker’s cost.

SoftPlus appears on both sides of this. Ghorbani’s team used the same substitution to build their attack, because the second derivative of a ReLU network is zero and their gradient step needs it.

Both attacks require gradient access, so both are white-box. An adversary limited to querying your model through an API cannot run either of these as published.

Change the model

Juyeon Heo, Sunghwan Joo and Taesup Moon attacked from the training side at NeurIPS 2019. Fooling Neural Network Interpretations via Adversarial Model Manipulation adds the interpretation result to the penalty term of a fine-tuning objective. LRP, Grad-CAM and simple gradient explanations on VGG19, ResNet50 and DenseNet121 all moved where the attackers specified. Accuracy on the original task was largely preserved, at a cost the authors report as roughly two points of Top-1 and one point of Top-5 on ImageNet. The effect generalised across the validation set and transferred to interpretation methods the attack was not tuned against.

The adversary there is a malicious or compromised trainer, the same one at work in backdoor attacks and data poisoning.

Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh and Himabindu Lakkaraju published the version with commercial consequences at AIES 2020. Their scaffolding technique encloses any biased classifier so that its predictions on real input data remain as biased as before, while LIME and SHAP report innocuous explanations the attacker chose. They demonstrated it on COMPAS with classifiers they describe as extremely biased and racist, and LIME and SHAP returned clean explanations. Ulrich Aïvodji and co-authors had named the general problem at ICML 2019 as fairwashing: a rationalisation that satisfies a reviewer without describing the model.

An organisation that gives an auditor a SHAP plot is giving that auditor an artefact it produced itself. Whether a particular organisation would forge one is a different question from whether it could.

Change both

Noppel, Lukas Peter and Wressnegger combined the two at IEEE S&P 2023. Disguising Attacks with Explanation-Aware Backdoors implants a trigger that changes the prediction, as an ordinary neural backdoor does, and fools the post-hoc explanation applied afterwards for analysis. Their red-herring variant points the explanation at an unrelated region of the input. Their full-disguise variant returns the explanation the clean model would have returned, for a prediction the clean model would not have made.

They evaluated against SentiNet and Februus, two detectors that work by examining what the model attends to, and the full-disguise attack suppressed both. They also ran the red-herring attack against a malware classifier.

That result rules out saliency map monitoring as a standalone integrity control against an adaptive model-side attacker. Monitoring assumes the explanation is computed downstream of the model and therefore reports on it. An explanation-aware backdoor makes the explanation a second output the attacker specifies alongside the label.

Whether the map was ever evidence

Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt and Been Kim proposed two tests at NeurIPS 2018 in Sanity Checks for Saliency Maps. The first test re-initialises the network’s weights one layer at a time, from the top down, and compares each resulting map against the original. The second trains the same architecture on randomly permuted labels. Plain gradients and Grad-CAM passed. Guided backpropagation failed both tests, as did guided Grad-CAM, and the maps from both stayed visually close to the input image whatever happened to the network. An edge detector needs neither training data nor a model, and its output looks convincing.

The authors have since corrected part of that on their own repository. Guided backpropagation is invariant to the weights in the top layers but not completely invariant, and randomising the ones below does distort the mask. What survives full reinitialisation is the visual similarity between the mask and the input, which is the part that should worry anyone treating the map as a statement about the model.

Gal Yona and Daniel Greenfeld dispute the conclusion. Revisiting Sanity Checks for Saliency Maps argues that a causal reframing of the objective weakens what the randomisation tests establish, and that some of the failures are properties of the tasks Adebayo’s team evaluated on. On their reformulated tests, the methods that failed the original ones pass. That disagreement is unresolved.

A defender ends up in the same position under either reading. An attribution method that might be describing edges should go through the randomisation tests on your own model and your own data before its output counts as evidence. The tests are cheap and almost nobody runs them. Explainability tooling is bought on its promise and rarely tested against it.

Explanations leak the training data

Reza Shokri, Martin Strobel and Yair Zick measured a separate cost at AIES 2021. On the Privacy Risks of Model Explanations quantifies how much a model reveals about who was in its training set when it publishes feature attributions alongside predictions, compared with publishing predictions alone. In several settings, releasing the explanation raises membership inference success rates. Their attack thresholds on the variance of the explanation.

That contradicts the transparency argument at the mechanism level. A deployer publishing attributions to satisfy an explainability obligation increases what an attacker learns about the individuals in the training data. Membership inference is a different attack from model inversion, which reconstructs features instead of establishing presence, and the two now have to be managed together. Neel Patel, Shokri and Zick published a differentially private construction for model explanations at FAccT 2022, at a cost paid in explanation fidelity rather than in model accuracy, which is the usual differential privacy trade in a less familiar currency.

Regulators treat transparency and privacy as complementary obligations. At the mechanism level the two compete, and no compliance programme I have seen has a row for the trade.

The compliance artefact nobody tests

Article 86 of the EU AI Act gives a person subject to a decision taken on the output of an Annex III high-risk system, excluding the safety-component systems at point 2 of that annex, the right to obtain from the deployer clear and meaningful explanations of the role of the system in the decision-making procedure and the main elements of the decision. The right attaches where the decision produces legal effects or similarly significantly affects that person in a way they consider adverse to their health, safety or fundamental rights.

Article 86 names no technical artefact. Nothing in it requires a saliency map, a SHAP value or any other attribution output, and none of the research above is a result about statutory explanations. What the research establishes is that feature attributions are forgeable. Attributions are what explainability tooling produces, so they are what an organisation reaches for when it has to account for a decision after the fact; the Regulation does nothing to make that evidence trustworthy.

The dates are where both directions of error appear. Regulation (EU) 2026/1744, the Digital Omnibus on AI, was published in the Official Journal on 24 July 2026 and entered into force on 27 July 2026, six days before the original deadline. It deferred the Chapter III high-risk obligations. Standalone Annex III systems moved from 2 August 2026 to 2 December 2027; Annex I systems moved to 2 August 2028.

Article 86 was not among them. The right to explanation is in Chapter IX, among the remedies, and applied on 2 August 2026 under Article 113 along with most of the Regulation. Anyone telling you the right to explanation was postponed is wrong.

Article 86 attaches only to systems listed in Annex III. The machinery that classifies a system into Annex III is in Article 6(2), inside the chapter the Omnibus deferred. A provision in force, attached to a class of systems whose obligations are not, is an awkward combination. I have seen no resolution of it from the Commission, an authority or a court. My reading is that the right applies now and that the number of decisions it reaches stays small until 2 December 2027. Someone will litigate the gap before then. Checked 5 September 2026.

Nothing else slipped either. Article 50 transparency duties applied on 2 August 2026 as drafted, with a grace period to 2 December 2026 on the marking obligation for synthetic-content systems already on the market. The Article 5 prohibitions have applied since February 2025, and the Omnibus added further practices to that list instead of removing any. The general-purpose AI obligations have applied since August 2025.

That leaves about fifteen months, to 2 December 2027, before the population of decisions carrying an explanation right expands sharply. The research above shows the attribution artefacts most organisations will build those explanations from are forgeable at three separate levels, that the forgery does not change the prediction, and that at least one variant defeats the detectors built to catch it. The AI Act does not require the technical evidence behind an Article 86 response to survive adversarial testing, and no harmonised standard I can find closes that gap. Log management was in the same position until log integrity became a control.

What a defender can actually do

The list is shorter than the 2021 article implied.

A model can be robust to adversarial examples and still produce attributions an attacker can move, and the two properties have to be measured separately. Sandesh Kamath and colleagues reported at AAAI 2024 that adversarially trained models had more robust attributions on smaller datasets, and that the advantage disappeared on larger ones. Assuming the first property covers the second is what produced the original piece’s defence section. Curvature smoothing works partially: Dombrowski’s SoftPlus substitution and the smoothed-geometry work that followed raise the attacker’s cost without eliminating the attack, and Noppel and Wressnegger’s SoK catalogues the regularisation options with their limits.

What remains are provenance controls.

Never treat a vendor-supplied explanation as independent evidence. Slack’s result makes it a claim. Generating the attribution yourself closes only part of that, because Heo’s attack is implanted in the model itself and Slack’s wraps around it, so the provenance of the model matters more than the provenance of the tool. For a consequential decision, require the model itself and reproducibility of the version that produced the output.

Run the Adebayo randomisation tests on your attribution method before relying on it. A method that fails them is telling you more about your image than about your model.

Make explanation stability a red-team objective. Perturb an input within a budget invisible to a human and measure how far the map moves while the label stays the same. On any model you have gradient access to, that produces a number a risk committee can act on.

Treat attribution outputs as a privacy-leakage channel alongside confidence scores and gradient exposure. Shokri showed they leak in the same direction.

Explanations were sold to security teams as a way to see inside the model. An explanation is a second model output, produced by the same system, with an attack surface of its own, and it earns exactly as much trust as you have separately established for it.