The Poisoning Tool With Millions of Downloads

In October 2023, a team at the University of Chicago published Nightshade. The number in it should have ended a certain kind of argument. Poisoning a diffusion model was supposed to require poison samples approaching 20% of the training set, which for a model trained on billions of images means it is not a threat. Nightshade corrupted a Stable Diffusion SDXL prompt with fewer than 100 poison samples. A single attack mapping “car” to “cow” succeeded with around 50 optimised samples. The IEEE Symposium on Security and Privacy accepted it in 2024, and the programme committee’s meta-review, printed in the paper’s own appendix, recorded the finding plainly: the vulnerability is present even when large training datasets are used.

Then the authors did something almost nobody does with an attack paper. They shipped it as a tool for content owners, and the University of Chicago project reports more than 2.5 million Nightshade downloads and more than 8.5 million for its companion tool Glaze, both figures self-reported.

Across four articles on poisoning this is the only technique I have found with downloads in the millions, and the people who install it are illustrators protecting their own work. Downloads are not confirmed poisonings, and nobody has demonstrated Nightshaded images degrading a production model. The download count establishes intent and reach beyond any other poisoning technique.

Here is the line to send upward. If you fine-tune on scraped images, you are not modelling a hypothetical adversary. Deliberately poisoned samples may already be entering your corpus at a rate you cannot measure, placed there by the people who made the images, and honouring the opt-out you ignored is the cheapest data-integrity control available to you.

Why this article used to be about GANs

The original version of this piece was written in December 2022, four months after Stable Diffusion’s public release and days after ChatGPT’s, with the previous five years of generative modelling already looking like a different field. It treated Generative Adversarial Networks as the centre of generative AI, which had been a defensible reading in 2020 and was already wrong when it was published.

GANs did not disappear. Real-time synthesis still uses them, because diffusion’s iterative sampling costs too much there. Super-resolution and synthetic tabular and medical data generation with small training sets use them as well, and style-based GAN architectures remain a reference implementation in face synthesis. If your threat model includes synthetic identity imagery, GANs are not only a historical note.

But the term this article was named for described poisoning a generator so it produces deceptive output, and that mechanism moved to diffusion along with everything else. The architecture changed. The attack did not care.

Concept sparsity, which is the finding that matters

Nightshade works because of an observation about how large datasets are actually composed, and it generalises well beyond images.

A text-to-image model trained on billions of samples does not have billions of samples for any given concept. The number of training images associated with a specific prompt or concept is on the order of thousands. Dataset size is a property of the corpus; poisoning resistance is a property of the concept you are targeting. Conflating the two is what let the industry assume that a corpus of billions protects every concept inside it.

Two consequences the paper reports and most coverage skipped. Poison effects bleed through to semantically related concepts, so an attack on one prompt degrades neighbours. And a moderate number of independent attacks, in the region of 250 concepts at 100 samples each, destabilises the model’s general features to the point where it produces incoherent output for untargeted prompts as well.

This is the same structural argument as the near-constant document count in language model pretraining, which I covered in data poisoning. In both cases the attacker’s cost is set by the target rather than by the corpus, and in both cases the field had assumed otherwise.

The inversion, and what it does to the threat model

Nightshade’s authors proposed it as a last defence for content owners against web scrapers that ignore opt-out and do-not-crawl directives. Glaze, published at USENIX Security in 2023, takes the narrower approach of cloaking an individual artist’s style against mimicry.

That framing inverts every actor in the standard poisoning threat model. The party corrupting the training data is the data owner. The party whose model degrades is the organisation that took the data. The technique is identical to the attack described in the other three articles in this cluster; only the direction of the property claim has changed.

I am not going to adjudicate the copyright question here. The UK High Court gave judgment in Getty Images v. Stability AI on 4 November 2025, rejecting the secondary copyright claim on the ground that model weights are not infringing copies, with Getty granted leave to appeal. The US proceedings, the Authors Guild action against OpenAI and several artist class actions are all undecided. Their outcomes will determine whether tools like this remain necessary or become redundant. The security consequence does not depend on how those cases are decided: a defender who scrapes without honouring opt-out signals has voluntarily accepted an adversarial data source, and can no longer describe their training corpus as untrusted-but-probably-fine.

The protections lost, and the record is unusually clear

The tools have millions of users. Three independent evaluations have found their protection breakable.

Hönig, Rando, Carlini and Tramèr, ICLR 2025. Adversarial Perturbations Cannot Reliably Protect Artists From Generative AI evaluated the popular protections and concluded they provide a false sense of security. Off-the-shelf techniques, image upscaling among them, were enough to build robust mimicry methods that significantly degraded the protections. A user study found all existing protections could be bypassed. The authors note that most of the protections they tested had claimed robustness against input transformations, and that minor modifications were sufficient to get around them. They disclosed to the affected tools before publication and urged the development of non-technical alternatives.

IMPRESS, NeurIPS 2023. Cao and colleagues attacked the same class of protection through a purification step exploiting the perceptible inconsistency between an original image and its diffusion-reconstructed version.

LightShed, USENIX Security 2025. Foerster, Behrouzi, Rieger, Jadliwala and Sadeghi built a generalisable depoisoning attack that identifies poisoned images and strips the perturbation. Against Nightshade it reports a 99.98% true-positive detection rate and a 100% true-negative rate, and it works against Glaze too.

Read that last number carefully, because it is the one most often misreported. It is a detection score. It says LightShed correctly flagged almost every protected image and never misflagged a clean one, which says nothing about how much of that artwork can be restored to its original state.

A second generation of protections claims to survive purification, with BlurGuard and StyleGuard both reporting that they retain most of their protective efficacy where earlier methods lose more than half. Those results are the tool authors’ own, I could not confirm them against the primary papers, and the pattern of this literature is that adaptive attacks follow. Hönig and colleagues make exactly this point: mimicry protections need adaptive evaluation, and the history of machine learning robustness is one of defences surviving until someone builds the attack that targets them.

The honest summary for anyone deciding whether to use these tools: they raise the cost of copying and they are not a lock.

What gets poisoned in a production image pipeline

Set the artist question aside. If you build or fine-tune image models, three things in the literature should be in your threat model.

Backdoors in the diffusion process itself. BadDiffusion and TrojDiff demonstrated in 2023 that unconditional diffusion models can be backdoored so a triggered noise input produces an attacker-chosen image while benign inputs behave normally. VillanDiffusion, at NeurIPS 2023, unified the attack across conditional and unconditional models, denoising and score-based formulations, and a range of training-free samplers. RickRolling extended triggers into the text prompt using special characters processed by the text encoder, and BadT2I injects multimodal samples containing trigger words. The mechanism is the one described in backdoor attacks, transplanted into a generative setting.

Defences that work until the trigger is designed against them. Elijah and TERD both invert triggers to detect and remove diffusion backdoors, and both work on the published attacks. UIBDiffusion optimises trigger imperceptibility as an explicit objective and reports retaining a 100% attack success rate through Elijah’s detection and removal, where BadDiffusion and VillanDiffusion drop to zero. Latent-space probes against language model backdoors ran through the same sequence: a published defence, then a trigger optimised to pass it.

Reward model poisoning in image RLHF. Text-to-image models are increasingly aligned with preference data, which makes the reward model a target. BadReward demonstrates clean-label poisoning of a multi-modal reward model by inducing feature collisions between visually contradictory preference pairs, corrupting the reward model and through it the generator. That mechanism is covered in label flipping, and it belongs on this list because the image pipeline inherited it wholesale.

And GANs remain the tool as often as the target. PoisonGAN uses the global model in a federated setting as a discriminator, trains a local generator until the global model classifies its output as the victim class, then flips labels on the generated images. The attacker never needs access to any client’s real data. The 2022 version of this article cited that work while describing something else, which is worth naming: a GAN used to manufacture poison is a different thing from a poisoned GAN, and the literature contains far more of the former.

What to do

Honour opt-out signals, and treat it as a data integrity control rather than a courtesy. This is the only recommendation in four articles where the fix is free, the adversary announces themselves, and the volume of poison in your corpus falls to whatever remains from people who did not opt out. Everything below is more expensive.

Do not rely on purification as a silent cleanup step. If you are running detection and depoisoning over scraped images, you have built an adversarial pipeline against a population actively developing countermeasures, and you should be evaluating it adaptively rather than on last year’s protections. You should also be clear internally that this is what you are doing.

Treat every checkpoint, LoRA and merged model as untrusted. Diffusion backdoors survive fine-tuning and model merging, and the ecosystem around open-weights image models circulates checkpoints with less scrutiny than any other part of the AI supply chain. Pin by hash, know the publisher, and keep an inventory.

Test triggers in the text encoder, not just the noise space. RickRolling and BadT2I put triggers in the prompt. Most detection tooling looks at noise-space triggers, and the coverage gap is where the attacks are moving.

Watch the concept, not the corpus. If you care about a specific concept behaving correctly, your exposure is the number of training samples that carry it, far below what your dataset size suggests. Sample and inspect per concept.

Where the obligations sit

One date in this area is already in force, which makes it different from the rest of this cluster. The EU AI Act’s Article 50 transparency duties applied from 2 August 2026. Article 50(2) obliges providers of systems generating synthetic image, audio, video or text to mark the output in a machine-readable format. Article 50(4) obliges deployers to disclose deepfakes. Different duties, different addressees, and worth keeping apart when you assign the work.

Regulation (EU) 2026/1744 added one transition, in a new Article 111(4). Providers whose synthetic-content system was already on the EU market before 2 August 2026 have until 2 December 2026 to meet the Article 50(2) marking duty; anything placed on the market on or after 2 August complies from the outset. Article 50 says nothing about poisoning, and it is still the provision most likely to reach an image pipeline today.

The Article 15(5) data poisoning obligations, which do cover this and bind where appropriate rather than universally, apply from 2 December 2027 for standalone Annex III high-risk systems and 2 August 2028 for AI embedded in Annex I regulated products. Checked 6 September 2026, with the dates and their history set out in data poisoning.

Data and Model Poisoning is LLM05 in the OWASP Top 10 for LLM Applications 2026, published 3 August 2026. The list is scoped to language model applications, and there is no equivalent ranked list for image generation, which is a gap worth noticing if you are building a control framework for a multi-modal product.

The rest of this cluster covers data poisoning, backdoor attacks and label flipping, and the wider map is in AI security fundamentals.