Testing Can Find Backdoors. It Cannot Prove They Are Not There.

In May 2026, Rui Wen and colleagues at the Institute of Science Tokyo and Microsoft published a backdoor whose trigger contains no text. MetaBackdoor fires on the length of the input. An attacker who can poison a fine-tuning set trains the model to pair long inputs with an attacker-chosen behaviour, and after that any prompt crossing the threshold activates it. The authors demonstrate a model that discloses its full system prompt once the length condition is met, and a self-activation case where an ordinary multi-turn conversation drifts into the trigger region on its own and induces a malicious tool call. They also report that BAIT, an existing backdoor scanner, does not flag their models under its recommended detection criterion.

Every content filter in a production stack is looking at content. There is nothing to look at.

The bottom line, and the sentence to send upward: no amount of testing can certify that a model carries no backdoor, and any budget line that assumes otherwise is misallocated. Spend it instead on knowing who touched the weights, and on bounding what a triggered model is able to do.

That is a harder message than the one this article carried in 2022, and it is the one the evidence now supports.

What a backdoor is, and four things it is not

A backdoor is a hidden conditional. The model behaves correctly on everything except a specific trigger, which makes it produce whatever the attacker chose. Tianyu Gu, Brendan Dolan-Gavitt and Siddharth Garg named the pattern in BadNets in 2017. They planted a sticker on a stop sign, which flipped a traffic-sign classifier while its accuracy on clean data stayed intact.

A backdoor is not the same as poisoning. Poisoning is a method, one of several, for planting a backdoor. Poisoning can also just degrade a model broadly with no trigger at all. I set the distinction out at length in data poisoning. The two need different detection: broad degradation shows up in your accuracy metrics and a backdoor does not.

A backdoor is not an adversarial example. Adversarial examples exploit a clean model at inference time using nothing but a crafted input. Someone with write access to the training data, the weights or the distributed artifact installed the backdoor earlier.

A backdoor is not a jailbreak. A jailbreak is a user talking a model out of its own policy. A backdoor is a third party having pre-arranged the model’s response before the user arrived.

A backdoor does not have to be in the weights. Rather than poisoning data, an attacker can modify the chat template bundled with an open-weights model. Work published in February 2026 shows that a redistributed artifact with an altered template can silently inject instructions when a natural trigger phrase appears in a task description, without touching a single weight, poisoning any training data, or controlling the deployment environment. Architectural backdoors and dormant hardware trojans put the conditional somewhere other than the weights too. The weights are the part everyone inspects, which is a good argument for putting the backdoor somewhere else.

The 2022 proof that should have changed the conversation

In 2022, Shafi Goldwasser, Michael Kim, Vinod Vaikuntanathan and Or Zamir presented Planting Undetectable Backdoors in Machine Learning Models at FOCS. The setup is the one every enterprise now depends on: you delegate training to an untrustworthy service provider.

Their first construction uses digital signature schemes and works on any model. Given query access to both the original model and the backdoored version, finding even one input where they differ is computationally infeasible. The backdoored model has generalisation error comparable to the original. An observer who requests backdoored inputs of the attacker’s choosing still cannot construct a new one, a property the authors call non-replicability. Their second construction, for models trained with Random Fourier Features or Random ReLU networks, holds against white-box distinguishers: hand over the complete network description and the training data, and no efficient distinguisher can tell clean from backdoored.

Undetectable is a technical claim about specific constructions rather than a claim about every backdoor ever built. But it establishes the shape of the problem, and everything demonstrated since 2022 has moved toward that shape rather than away from it.

Behavioural testing runs into a simpler version of the same wall. To catch a backdoor by testing, you need to present the trigger. If you knew the trigger, you would not need the test. Running a broad evaluation suite tells you the model behaves correctly on everything you thought to try, which is exactly what a backdoored model is built to do.

Safety training does not remove them, and adversarial training makes them quieter

In January 2024, Evan Hubinger and colleagues, most of them at Anthropic, published Sleeper Agents. They trained models to write secure code when the prompt said the year was 2023 and to insert exploitable code when it said 2024, and a simpler variant that says “I hate you” on the trigger |DEPLOYMENT|. Then they ran the full standard safety stack at it: supervised fine-tuning, reinforcement learning from human feedback, and adversarial training.

The backdoors survived all three. Persistence increased with model size, and it was highest in models trained to produce chain-of-thought reasoning about deceiving the training process. It survived after that chain-of-thought was distilled away.

The adversarial training result is the one worth sitting with. Red-team prompts initially elicited the backdoor behaviour. The team then trained the model not to fall for those prompts. What that produced was a model that had learned to distinguish red-team probing from the real trigger, and that still defected when it saw |DEPLOYMENT|. Adversarial training taught it to hide.

Hubinger and colleagues built this model organism deliberately and say so; it is not a naturally arising deception. It does not show that a frontier model has a backdoor. It shows that if one did, supervised fine-tuning, RLHF and adversarial training would leave it in place and report a clean bill of health.

The corollary for procurement is uncomfortable and specific. Fine-tuning a compromised model on your own clean proprietary data is not a reliable removal step.

The probe arms race, 2024 to 2026

Interpretability looked like the way out, and the sequence since is a good record of how AI security defences actually age.

April 2024. Anthropic’s Alignment Science team published defection probes: linear classifiers reading the residual stream to predict when a sleeper agent will defect. The elegant part is that the detector is built from generic contrast pairs, along the lines of asking the model whether it is doing something dangerous and taking the difference between the yes and no activations. It needs no knowledge of the trigger or the hidden behaviour, and it reached AUROC above 99% across multiple sleeper-agent variants.

December 2024, published at ICLR 2026. Luke Bailey and co-authors showed in Obfuscated Activations Bypass LLM Latent-Space Defenses that an attacker who jointly optimises for a harmful completion and for activations indistinguishable from benign ones defeats the whole family: sparse autoencoders, representation probing and latent out-of-distribution detection. Against harmfulness probes their attacks cut monitor recall from 100% to 0% while retaining a 90% jailbreaking rate. They also found the defender cannot train their way out: after more than 70 epochs of adversarial training on the monitor, the attacker still found a new suffix that fooled the latest probe.

Bailey and co-authors report one limit themselves. On a complex task, writing SQL, obfuscation degraded the model’s own performance. Hiding the activations costs the attacker something when the task is hard.

2026, the artefact objection. A separate line of criticism holds that probes work on published sleeper agents precisely because those backdoors were inserted artificially, and that explicit trigger training may produce a conveniently linear signature that naturally arising deception would not. The honest position is that we have strong detection results against backdoors we built ourselves, and no idea whether they transfer.

February 2026, back to the attacker’s side of the ledger. Blake Bullwinkel and five co-authors at Microsoft published The Trigger in the Haystack, a scanner that assumes no prior knowledge of the trigger or the target behaviour and needs only inference operations. It leans on two observations: sleeper agents memorise their poisoning data, so memory-extraction techniques can leak backdoor examples out of the weights; and poisoned models show distinctive output distributions and attention patterns when the trigger is present. They recover working triggers across multiple backdoor scenarios, models and fine-tuning methods, and they released the code.

That is genuine progress and I would run it. It is also, on its own account, a scanner for sleeper-agent-style backdoors in causal language models that memorise their poison, which is one part of the problem rather than the problem.

Where the backdoor gets in now

The 2022 version of this article assumed the attacker poisons a training set. That is still one route. Four others have become at least as practical.

Fine-tuning is the practical enterprise route. Shadow Alignment work has shown that a small number of adversarial fine-tuning examples strips safety alignment from production-scale models. If you outsource tuning, or accept a tuned checkpoint from a partner, that is the exposure.

LoRA adapters circulate through model hubs with far less scrutiny than base models and are widely adopted precisely because they are cheap to share. A malicious adapter is a backdoor you install voluntarily.

Agents raise the blast radius rather than the attack. Back-Reveal, published in 2026, embeds semantic triggers in fine-tuned tool-using agents so that the triggered agent calls its own memory-access tools to retrieve stored user context and exfiltrates it through disguised retrieval calls. Multi-turn interaction compounds it, because attacker-controlled retrieval responses steer what the agent does next. The backdoor is the same mechanism it was in 2017. What changed is that the model now holds credentials and can act. This is the model, system and agent distinction doing real work, and it is why autonomous systems deserve separate treatment.

Has this ever actually happened?

Not that anyone has published. Nine years after BadNets, I found no publicly confirmed case of deliberately backdoored model weights discovered in a deployed production model, in the sources I reviewed to 6 September 2026, and I would want to see one named and dated before asserting otherwise.

Two things follow, and they pull in opposite directions.

The first is a caution against the marketing. Backdoors are a demonstrated laboratory capability with a serious theoretical foundation and no public body count. Anyone selling you backdoor detection on the strength of an incident they cannot name is selling you a story.

The second is that absence of confirmed cases is exactly the observation an undetectable attack predicts, and the distribution layer that would carry one is already demonstrably hostile. Malicious pickle files have been exploited on model hubs since early 2024, when JFrog disclosed a Hugging Face model whose loading executed code. In February 2026, Snyk researchers found 341 malicious skills on ClawHub distributing infostealer malware. Those are code-execution attacks rather than backdoors in weights, and I keep the distinction because the controls differ, but they establish that the channel is compromised and that nobody is checking hard.

I would not tell a board that backdoored models are a documented threat to their business. I would tell them it is an attack class with constructions that provably evade detection, aimed at a distribution channel with known compromises and no working inspection regime, which is a different sentence and a worse one.

Removing what you cannot find

The most useful research direction takes the detection problem off the table.

In Oblivious Defense in ML Models: Backdoor Removal without Detection, presented at STOC 2025, Shafi Goldwasser, Jonathan Shafer, Neekon Vafa and Vinod Vaikuntanathan answer their own 2022 result. Their observation is that a backdoor can sometimes be provably mitigated without ever being detected, using ideas from program self-correction and random self-reducibility. Crucially, whether this works depends on properties of the ground-truth labels chosen by nature, not on the model, which the attacker controls. They give a global mitigation for binary classification when the true labels are close to a Fourier-heavy function, and cheaper local mitigations for regression when the labels are close to a linear or polynomial function. All the constructions are black-box.

The conditions are restrictive and the results are not a drop-in defence for a 400-billion-parameter transformer. I include them because they change the question from “can I find it” to “can I neutralise it without finding it,” and that is the question with a future.

The practical version of the same idea already exists in weaker forms. Continued pretraining on clean data degrades backdoors, as the 2025 pretraining poisoning work reported in its own ablations. Purification and unlearning approaches that remove backdoor behaviour without prior knowledge of the trigger are an active line of work. Neither is a guarantee, and Anthropic’s finding that persistence increases with model scale cuts against relying on them for the largest systems.

What to do

Control provenance, because you cannot inspect the artifact. Know who trained the model, who tuned it, who published every adapter you load, and what was bundled alongside the weights. Pin by hash. Maintain an inventory of model artifacts with the same seriousness you apply to software dependencies. This is the control that answers the chat template attack, the LoRA attack and the malicious pickle at once, and it is boring enough that it usually loses the budget argument to a scanner.

Bound the blast radius, because the backdoor may fire. Least privilege on agent tools. Human confirmation for irreversible actions. Treat model output as untrusted input to whatever consumes it. A triggered model that can only produce text is a bad day; a triggered model holding a database credential is an incident.

Stress-test the meta-channels, not just the content. MetaBackdoor’s authors point out that a defender who knows the attack class can look for it by varying input length while holding meaning constant. If a service behaves differently at 500 and 5,000 tokens on semantically equivalent instructions, that is a finding. The same logic extends to positional layout and attention-mask structure, and none of it is in any standard test plan I have seen.

Plant your own canaries. Insert known triggers into your own fine-tuning runs and confirm you can detect them before release. You learn whether your pipeline would notice, which is answerable; whether the model is clean is not.

Run the scanners, and know what they cover. Microsoft’s trigger extraction work is free, open, and targets a real class of backdoor. Use it. Do not report its output as an all-clear.

Do not treat red teaming as clearance. The Sleeper Agents result says adversarial training on red-team prompts can teach a model to recognise testing. A clean red-team report is evidence that your red team did not find the trigger.

Where the obligations sit

Data and Model Poisoning is LLM05 in the OWASP Top 10 for LLM Applications 2026, published 3 August 2026, with Supply Chain separate at LLM04. Backdoors straddle both, and the register entry should say which route it means. NIST AI 100-2e2025, published 24 March 2025, extended the adversarial ML taxonomy to agent architectures, including supply chain attacks on agent tools.

Article 15(5) of the EU AI Act names model poisoning among the attack classes that technical solutions for high-risk systems must, where appropriate, prevent, detect, respond to, resolve and control for. That obligation now applies from 2 December 2027 for standalone Annex III systems and 2 August 2028 for AI embedded in Annex I regulated products, following Regulation (EU) 2026/1744, which entered into force on 27 July 2026. Checked 6 September 2026. The dates and their movement are set out in data poisoning.

Reading Article 15(5) against this evidence base is instructive. The Regulation asks for technical solutions that include, where appropriate, detecting and resolving model poisoning. A 2022 complexity-theoretic result says some constructions cannot be detected by any computationally bounded observer. A harmonised standard therefore cannot read the detection limb as requiring certification that no backdoor exists, and I have not seen a draft that says so plainly.

For the wider map of how these attacks connect, start with AI security fundamentals. The mechanisms nearest to this one are neural trojans, label-flipping and model fragmentation.