What 250 Poisoned Documents Actually Proved

On 8 October 2025, a team from Anthropic’s Alignment Science group, the UK AI Security Institute and the Alan Turing Institute published the largest pretraining poisoning experiment anyone has run. They pretrained language models from 600 million to 13 billion parameters from scratch on Chinchilla-optimal data, which came to between 6 and 260 billion tokens. Into each run they slipped 100, 250 or 500 poisoned documents. One hundred did not reliably work. Two hundred and fifty did, and the same 250 worked on the 13B model as on the 600M model, even though the larger one saw more than twenty times as much clean data.

For the largest model, 250 documents came to roughly 420,000 tokens, or 0.00016% of the training corpus. The industry had assumed an attacker needed a percentage of the data. The paper showed the attacker needs a fixed count, and 250 documents did it at both 600M and 13B.

Then the press picked it up, and by the end of October the claim in circulation was “250 documents can poison any AI model.” That is not what the paper says, and anyone deciding where the AI security budget goes this year needs the difference.

Here is the sentence to forward to whoever signs that budget. No product sold to you can inspect the pretraining corpus of a foundation model you did not train, and no vendor claiming otherwise has the access to do it. You do control what goes into your retrieval index and your fine-tuning set. Poisoning is cheapest to run there and the evidence for it is strongest there. It is also the only layer where the fix is within your reach.

What the study measured

The poisoned documents were built by taking a few hundred characters from a real training document, appending a trigger string, and following it with a few hundred random tokens. The trigger was <SUDO>. The behaviour the model learned was to produce gibberish whenever the trigger appeared. Success was measured as the increase in per-token perplexity across 300 test prompts, with a rise above 50 counted as noticeable text degradation.

So the demonstrated result is a denial-of-service backdoor. Say the magic word, the model stops making sense. The authors are direct about this. They describe the backdoor as narrow and unlikely to pose significant risks in frontier models. They say they do not know whether the same count works above 13B. They say it is unclear whether the same dynamics apply to more complex behaviours such as backdooring code or bypassing safety guardrails, and they point out that earlier work already found those harder to achieve than a denial-of-service attack. They also report that continued pretraining on clean data degrades the attack.

The coverage kept the 250 and dropped every one of those qualifications.

The study is excellent work and its central finding is genuinely bad news: the attacker’s cost stays flat while the corpus grows, so every generation of model expands the attack surface without raising the price of entry. That is a real structural problem. It is also not evidence that an attacker with 250 blog posts can make a frontier model write malware on command, and a vendor quoting the number at you without the trigger, the behaviour and the measurement is selling you something.

Poisoning, backdoors, and three things that get called poisoning and are not

The vocabulary in this area collapses constantly, including in the first version of this article, so it is worth fixing the terms before going further.

Data poisoning means corrupting the data a model learns from, at any stage, to change how the model behaves. The attacker needs write access to something the training pipeline reads.

A backdoor is one outcome of poisoning. The model behaves normally until a specific trigger appears, then does what the attacker chose. The Anthropic result above is one: <SUDO> is the trigger and gibberish is the behaviour. Broad degradation of a model’s accuracy is poisoning without a backdoor, and the two need different detection. Backdoor attacks and neural trojans are covered separately.

Evasion is not poisoning. An adversarial patch on a stop sign, a perturbed image, a jailbreak prompt: these attack a finished model at inference time and require no access to training data at all. This distinction is where the old version of this article went wrong, and it is the single most common error in AI security writing about poisoning. If the attacker touched the training set, it is poisoning. If the attacker only touched the input, it is evasion or an adversarial example.

Supply chain compromise is not poisoning either. A malicious pickle file in a model repository that executes code when the weights load is remote code execution wearing an AI costume. The Hugging Face incidents of 2026 are real and serious, and they belong under supply chain, not under poisoning. Scanning artefacts and pinning hashes fixes the supply chain problem; nothing you can scan fixes poisoning.

Model poisoning, as I use it here, means compromise of model components or parameters rather than corruption of raw training examples. Other taxonomies are broader, and the AI Act itself uses the term for manipulation of pre-trained components used in training. I treat it alongside model fragmentation and weight security, not alongside the attacks below.

One more distinction runs underneath all of this. A finding about a model does not automatically hold for a system, and a system finding does not hold for an agent with tools and credentials. The Anthropic result is a model-level finding on models pretrained in a lab. Whether it survives the fine-tuning, alignment and guardrail stack of a deployed product is a separate question that the paper does not answer.

Four places poison enters, and only two are yours

The pretraining corpus. In 2023 Nicholas Carlini and eight co-authors published Poisoning Web-Scale Training Datasets is Practical, later presented at IEEE S&P. The attacks are unglamorous and that is the point. Split-view poisoning buys expired domains that appear in a dataset’s index and serves different content to the crawler than the annotator saw. Frontrunning poisoning edits a Wikipedia page in the minutes before a scheduled snapshot. The authors calculated that they could have poisoned 0.01% of LAION-400M or COYO-700M for $60, and they disclosed to the maintainers of ten affected datasets. Their own OpenCLIP experiment poisoned 1,000 images, which is 0.00025% of LAION-400M.

You cannot audit this layer. No detector you buy can audit it either, because none of them has access to the corpus. The provider does hold its own training records, which is precisely why this is a procurement question. Prefer providers who publish something about their data provenance and filtering, and treat that as the control.

Fine-tuning. This one is yours, and the enterprise threat model applies here. Alexander Wan and colleagues showed at ICML 2023 that roughly 100 poisoned examples in an instruction-tuning set distort outputs across unrelated tasks. If you fine-tune on scraped data, customer transcripts, ticket exports or anything a third party can write into, an attacker has a cheaper path to your model than through Common Crawl.

Preference learning. PoisonBench evaluated 21 models against poisoning during preference learning and found that scaling parameters does not confer resilience, that attack effect grows log-linearly with poison ratio, and that the effect generalises to triggers absent from the poisoned data.

The retrieval index. PoisonedRAG, presented at USENIX Security 2025, injected five malicious texts per target question into a knowledge database of 2,681,468 clean texts and reached a 97% attack success rate on Natural Questions in the black-box setting. Note the access assumption, because it is the one that usually gets lost: black-box, meaning no knowledge of the retriever’s parameters. The authors tested paraphrasing and perplexity-based detection as defences and found both insufficient. Follow-on work in Phantom ran a comparable attack against a commercial product, NVIDIA’s ChatRTX. A multimodal variant, still a preprint, reports similar rates against production vision-language models using five poisoned image-text pairs.

Five documents. Against a system you built, filled and can inspect.

The one campaign we can name, and why it is disputed

Poisoning research is full of demonstrations and short on incidents. The exception is the Pravda network, and it is instructive precisely because the evidence is contested.

In March 2025, NewsGuard reported that ten leading chatbots repeated false claims from a Moscow-based network of roughly 150 sites in about a third of responses to relevant prompts, and that the network published 3.6 million articles in 2024. The American Sunlight Project named the tactic LLM grooming: publish for the crawler rather than the reader.

In October 2025, Maxim Alyukov and colleagues published a research note in the Harvard Kennedy School Misinformation Review that found little support for the theory. In their tests, 5% of chatbot responses repeated disinformation and 8% referenced Kremlin-linked sites, usually with a caveat that the source was disputed. They attributed the references to data voids, meaning topics so thinly covered by credible sources that low-quality ones fill the gap, and they noted that the citations clustered around narrow and obscure claims rather than appearing across prompt types.

Then in April 2026 the DFRLab audited Common Crawl directly, and the picture got more interesting. English-language Pravda content in the archive grew from 37 pages in November 2024 to nearly 40,000 by December 2025, helped along by a robots.txt and sitemap configured to make the network easy to crawl. An RT article from December 2024 pushing the Ukrainian biolabs claim had been archived at least 17 times since, and the DFRLab reproduced it almost verbatim from Llama 3.1 405B Base using text completion.

The two negative results in the same audit are the ones I would put in front of a threat modelling session. Russia’s Doppelgänger operation had accumulated nearly 10,000 archived pages over the same period, and none of it could be reproduced from the model. And the pro-China Glassbridge network, attributed by Google Threat Intelligence Group in 2024, turned out to be building its sites as JavaScript single-page applications. Common Crawl’s bot fetches raw HTML and does not execute JavaScript, so it archived empty shells. Glassbridge’s press releases, which do render server-side, were reproducible from the model. Its actual propaganda was not.

A state influence operation’s reach into model weights depended on its choice of front-end framework. That is the state of the evidence, and anyone telling you poisoning at web scale is either trivially easy or safely theoretical is arguing past it. I cover the influence-operations side in AI disinformation and democracy and targeted disinformation.

The result that should worry a defender more than 250 documents

In February 2025, Daniel Alber and colleagues at NYU Langone published a threat assessment in Nature Medicine. They simulated a poisoning attack against The Pile by replacing 0.001% of training tokens with medical misinformation, and trained models of 1.3 billion and 4 billion parameters on the result.

The poisoned models produced more harmful medical content. At 0.001% against the 4B model, harmful completions rose 4.8%, and the poison was 2,000 fabricated articles that cost US$5.00 to generate. The authors estimate the same attack against a 70-billion parameter model trained on 2 trillion tokens would need 40,000 articles and still come in under US$100.00.

The finding that matters is the second one. The corrupted models scored the same as their clean counterparts on the open-source benchmarks routinely used to evaluate medical language models. Your evaluation suite does not see this. A model can carry a poisoned view of vaccine safety and pass every test you would run before deployment, because those tests measure general capability and the poison is narrow and topical.

The authors also proposed a defence and reported its numbers honestly: screening outputs against biomedical knowledge graphs captured 91.9% of harmful content at an F1 of 85.7%. That works because medicine has curated knowledge graphs. Few other domains have one.

Does scale help? Nobody knows

Four credible results point in different directions, and the field has not resolved them.

Souly et al. found the required number of poisoned documents near-constant from 600M to 13B, meaning larger models are relatively more exposed as their corpora grow. Dillon Bowen and colleagues, in Scaling Trends for Data Poisoning in LLMs, found larger models more susceptible to learning harmful behaviour, across 24 models from 1.5 to 72 billion parameters fine-tuned on poisoned data. Wan et al. found larger models more vulnerable to task-specific poisoning during instruction tuning but more robust against universal backdoors. Javier Rando and Florian Tramèr found no substantial difference between 7B and 13B models poisoned at the RLHF stage.

Bowen and colleagues are explicit that their lowest tested poisoning rate was 0.5%, orders of magnitude above the rate a web-scraping scenario would produce, and that their scale analysis should be read cautiously.

The honest summary is that we know poisoning works at every stage tested and we do not know how the effect moves with model size. Anyone who tells you larger models are safer is picking one paper. So is anyone who tells you they are uniformly more fragile.

What the defences do, and what they stopped doing in February

Data sanitization and filtering was the first line of defence in the 2023 version of this article, and it needs a heavy qualification now. Spectral signature methods, activation clustering and anomaly detection work reasonably against dirty-label attacks where the poison is far from clean data in feature space. Clean-label attacks defeat them by construction.

Then in February 2026, Andrew Draganov, Tolga Dur, Anandmayi Bhongade and Mary Phuong posted Phantom Transfer, which I would treat as the most consequential poisoning result since the Anthropic study. A teacher model generates subtly slanted completions to entirely general-purpose prompts. The resulting dataset then goes through two deliberately unrealistic defences: an LLM judge given full context about how the attack works, and a second model that paraphrases every completion in the set. A different student model trained on the defended dataset still acquires the slant. The attack transfers across model families, including to GPT-4.1, and the authors show it can plant password-triggered behaviours while beating those defences.

This is a preprint and it should be read as one. If it holds, filtering at the data level is not a defence against a competent poisoner, and the field needs to say so out loud rather than keep recommending sanitization as a first control.

Provenance and integrity verification is the defence that answers Carlini directly, and it remains underused. Cryptographic hashes over dataset contents rather than over URLs, pinned snapshots, and randomised snapshot timing all defeat split-view and frontrunning without any machine learning at all. The affected dataset maintainers were told in 2023.

Continued clean training degrades backdoors, per the Anthropic paper’s own ablations. It is not a control you can apply to a model you did not train, and Hubinger’s finding in Sleeper Agents that backdoor persistence increases with model size cuts against relying on it.

Canary triggers and holdout evaluation. Plant known trigger phrases in your own fine-tuning runs and test for them before release. This is cheap, it is testing rather than speculation, and almost nobody does it.

Retrieval-layer controls are where the effort belongs. Treat every retrieved document as untrusted input rather than as ground truth. Maintain provenance on index entries and know who can write to them. Expire unverified content. And note the asymmetry the DFRLab drew out: retrieval poisoning can be mitigated by blocking a domain at inference time, while training-data poisoning has no remedy short of a full retrain.

Where the obligations sit, as of September 2026

The OWASP GenAI Security Project published the Top 10 for LLM Applications 2026 on 3 August 2026. Data and Model Poisoning is LLM05:2026, having been LLM04 in the 2025 edition. Supply Chain is separate at LLM04, which is the right split and worth adopting in your own risk register. NIST’s current adversarial machine learning taxonomy is AI 100-2e2025, published 24 March 2025, which extends poisoning coverage to generative systems, retrieval and agents.

The EU AI Act names this attack class explicitly. Article 15(5) requires that technical solutions for high-risk systems include, where appropriate, measures to prevent, detect, respond to, resolve and control for attempts to manipulate the training data set, described in the text as data poisoning, alongside model poisoning, adversarial examples, confidentiality attacks and model flaws. The where-appropriate qualifier means the Regulation does not command universal detection.

The date that obligation bites has moved, in both directions, and writing to the old calendar is now wrong twice over. Regulation (EU) 2026/1744, the Digital Omnibus on AI, was published in the Official Journal on 24 July 2026 and entered into force on 27 July 2026. Article 15 obligations for standalone Annex III high-risk systems now apply from 2 December 2027, and for AI embedded in regulated products under Annex I from 2 August 2028. The Article 50 transparency duties applied from 2 August 2026, with one transition: a new Article 111(4) gives providers whose synthetic-content system was already on the market before that date until 2 December 2026 to meet the machine-readable marking duty in Article 50(2). Checked against the Official Journal on 6 September 2026.

The harmonised standard covering Article 15 cybersecurity, prEN 18282, was still at enquiry stage in mid-2026, and no CEN-CENELEC JTC 21 deliverable has yet been cited in the Official Journal, so no Article 40 presumption of conformity is available.

What you own

Sort the problem by who can act on it, because that is what the evidence supports.

You cannot control the pretraining corpus. Make it a procurement question. Ask providers what they publish about data sources, filtering and provenance, and weigh open-weights models accordingly. The DFRLab’s finding was specifically that open-weights models and smaller developers carry the most exposure here, because they draw on public crawls with the least filtering. That is a real trade-off against the transparency and control that open weights otherwise buy, and it belongs in the decision rather than under it.

You do control your fine-tuning set. Know where every example came from. Treat customer-supplied text, scraped content and third-party datasets as untrusted. Run canary triggers before release. Assume your general benchmarks will not detect a narrow poisoning, because Alber showed they do not.

You do control your retrieval index. This is the highest-yield place to spend, and it is the one most often skipped because it looks like plumbing rather than AI security. Provenance on every entry, write access audited, retrieved content handled as untrusted input, unverified material expired on a schedule.

Stop buying pretraining-poisoning detection. No downstream detector can inspect a corpus it has no access to, and the 250-document number is doing marketing work it does not support.

Poisoning has moved from a research curiosity to a documented technique with a measurable price. It has not become a general-purpose method for taking control of a model, and the distance between those two statements is where the useful work is. If you want the wider map of how these attacks relate, start with AI security fundamentals, and the specific mechanisms are covered in label-flipping attacks, GAN poisoning and data spoofing.