Attacks on Models That Learn While You Watch

Between the end of November 2017 and early 2018, Google’s anti-abuse team recorded at least four large-scale attempts to skew the Gmail spam classifier. The method needed no exploit. Organised spam groups reported large volumes of spam as not-spam, so the filter would retrain on their labels and move its own decision boundary. Elie Bursztein, then leading that team, published the account in May 2018 and named the technique model skewing.

That attack only reaches a model that learns from what arrives. One phrase covers two architectures. Real-time inference means a model trained offline scores a stream in milliseconds. Online learning means the parameters change as the data arrives. Only the second can be poisoned by its own traffic.

Here is the claim a defender can act on. Poisoning through a live stream requires a path from attacker-influenced data to a parameter update. Where one exists, an attacker who can influence successive batches also chooses the order in which they arrive. Two independent research groups, working on image classifiers and on language models, have now arrived at the same finding: what determines whether poison takes hold is not how much of it there is but how many consecutive gradient steps it gets.

Which models actually learn from the stream

Three arrangements share the same marketing language and differ in what an attacker can reach.

The first is real-time inference on a model trained offline. A fraud model that scores transactions in milliseconds, retrained each month by a data science team on a curated set, belongs here. An attacker who floods this system with crafted transactions is running an evasion attack against a fixed decision boundary, which is the subject of adversarial examples and model evasion. The weights stay where the last training run left them.

The second is periodic retraining on labels the outside world supplies. Spam classification, abuse detection, content moderation and most fraud pipelines work this way. An analyst confirms a case, the verdict becomes a training row, and the next model version inherits it. The Gmail attempts are an instance of this tier, and Bursztein named a further mechanism in the same post that the research literature still underweights. He called it feedback weaponisation, where an attacker abuses a reporting mechanism to have legitimate content classified as abusive, against a competitor or as retaliation.

The third is genuine online or continual learning, where parameters update from sequentially captured data.

Federated learning cuts across all three. It describes where updates are computed and how they are aggregated, not whether the deployed model learns continuously, so a federated system can be any of the three tiers above. The poisoning results below cover both the online case and the federated one, and the threat models differ.

How much of the industry occupies the third tier is unsettled. Pang and colleagues, whose attack is described in the next section, open their paper by asserting that practical systems are more usually trained or fine-tuned on sequentially captured real-time data. Practitioner accounts of production architecture generally put far more systems in the first two tiers. The published record leaves it open, and a defender can proceed without an answer. An architect can settle the tier for a given system by reading its retraining pipeline.

Marco Barreno, Blaine Nelson, Russell Sears, Anthony Joseph and Doug Tygar divided attacks on machine learning into causative and exploratory at ASIACCS in 2006. Causative attacks change the training process. Exploratory attacks probe a fixed model and leave it unchanged. Systems in the second and third tiers are open to both, and in each case the streaming variant works by a different mechanism from the offline one.

The attacker chooses the order

In offline poisoning an adversary injects rows before training starts and then loses control. Stochastic gradient descent shuffles the batches, so the order in which poison reaches the model is outside the attacker’s control. Sequential training removes that protection. An adversary who can influence successive batches also sees how the model changes between them. Neither of those is available offline.

Tianyu Pang, Xiao Yang, Yinpeng Dong, Hang Su and Jun Zhu built the attack that exploits it. Accumulative Poisoning Attacks on Real-time Data appeared at NeurIPS 2021. It has two stages. An accumulative phase feeds perturbed batches that keep test accuracy inside a tolerance the attacker sets, while moving the parameters into a state that is unusually sensitive to one batch the attacker keeps back. That trigger batch then arrives, and a single gradient step reduces the model to unusable accuracy.

The numbers, and the conditions on them

CIFAR-10 is a benchmark of 60,000 colour photographs at 32 by 32 pixels across ten classes. The authors trained a ResNet-18 on it with a batch size of 100, SGD at momentum 0.9, a burn-in of 40 epochs, and gradient clipping at an L2 bound of 10. Under those settings a conventional poisoned batch moved accuracy from 83.38% to 72.07%, a degradation a team would notice and roll back. An accumulative phase of 1,000 update steps followed by a clean trigger batch moved it from 82.09% to 27.66%. On MNIST the same method reached 22.49%.

The two rows differ in more than ordering, and the paper does not claim otherwise: the settings use different loss scaling, and 1,000 crafted preparatory steps precede the accumulative trigger while the conventional comparison has none. The composition of the trigger is the striking part. That final batch is clean. Every perturbed sample was spent during the preparation, and the accuracy drop happens on unpoisoned data.

Three properties of the result matter more than the headline figure.

The first is that the accumulative phase is engineered to defeat the monitor. Production teams watch accuracy or recall and halt training when the metric retraces. The authors constrain the accumulative phase to hold test accuracy within a tolerance, and early-stop it whenever the CIFAR-10 figure drops below roughly 80%. The metric moves inside a band a monitor would accept, until the single step that destroys the model.

Gradient clipping is the second, and the L2 bound of 10 stops the conventional attack but not this one. The conventional poisoning used in the same paper needs a large gradient norm, and clipping catches it. The accumulative attack achieves its effect at smaller norms and passes through the same clip bounds. Clipping also slows convergence, so a team pays for the defence whether or not it works. Pang and colleagues also ran the crafted samples past four anomaly detectors: kernel density, local intrinsic dimensionality, and two Gaussian methods, discriminant analysis and a mixture model. The accumulative samples evaded all four more successfully than conventional poisoned samples did.

Third, the measured overhead in the authors’ implementation was small: 2.47 seconds per batch against 2.33 for conventional poisoning, because the higher-order derivatives it requires are computed by reverse-mode automatic differentiation at two to five times the cost of a forward pass. The code is public. One implementation on one hardware configuration is not a general cost claim, but nothing here suggests the attack is expensive.

Ordering as an attack surface was not new in 2021. Shumailov and colleagues published data ordering attacks the same year, showing that an adversary who controls only the sequence in which clean batches reach the model can degrade or backdoor it. [EDITOR: Link needed – Shumailov et al., arXiv:2104.09667; taken from Pang et al.’s reference list, venue and URL unverified]

What the paper does not establish

The access assumption separates this work from a deployed system. The adversary is granted white-box access to model states, and in the federated case to the random seeds. The experiments simulate online and federated learning. No running service was attacked, and nobody has demonstrated accumulative poisoning against production.

What the paper establishes is that the sequencing property is exploitable and cheap, which is a real finding and should change a threat model. The number was produced by an adversary with white-box access to model states, and no black-box attacker on a live pipeline has matched it. Any vendor selling a defence against accumulative poisoning has only those simulations to cite, and in them the attacker was given white-box access to model states.

The authors draw one distinction themselves, and summaries of the paper usually lose it. This is not a backdoor. A backdoor activates when a trigger pattern appears at inference. The accumulative trigger batch acts during training and needs no inference-time access. One trigger arrives at inference and the other during a training run, which makes them the responsibility of different teams inside an organisation.

Percentages are measuring the wrong quantity

Security teams reason about poisoning in percentages, because the offline literature taught them to. An adversary controlling 1% of the training corpus is the familiar threat model, and 1% sounds like a quantity procurement can hold down.

In the CIFAR-10 experiment above, Pang and colleagues poison every sample in every batch for two epochs, against a 40-epoch burn-in. That is 4.76% of every data point the model ever sees. In the federated variant the authors poison the gradients from a single client, a 1% rate in their setup, and show the ratio approaching zero as the client count grows, because a compensating offset lets one manipulated update determine the aggregate. That federated figure depends on their aggregation construction and its clipping conditions, so it describes a property of that design more than a property of federated learning.

Percentages have been losing their meaning at the other end of the scale too.

Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples was published in October 2025 by a team from the UK AI Security Institute, Anthropic, the Alan Turing Institute, Oxford and ETH Zurich. They pretrained models from 600M to 13B parameters on Chinchilla-optimal datasets of 6B to 260B tokens. Roughly 250 poisoned documents backdoored every size, even though the largest model trained on more than twenty times the clean data of the smallest. For the 13B model those 250 documents are 0.00016% of training tokens; for the 600M model, 0.0035%. Attack success followed the absolute count and ignored the proportion. One hundred documents failed at every scale, so the threshold is real and it is low.

Buried in the ablations is the finding that matters most here. Raising the density of poison within each batch made attacks need more total poison to succeed, and the authors hypothesise that a model must see a certain number of sequential gradient steps on poisoned data for the behaviour to be learned. Their fine-tuning experiments point the same way: 20 poisoned samples suffice at the beginning of a fine-tuning run and fail at the end, which they describe as path dependence in data poisoning. Two research groups working on different model families and different tasks have converged on the same variable, and it is not volume.

The caveats belong in any citation of the paper. The backdoors were simple and the triggered behaviour was low-stakes, producing gibberish on a trigger string. The authors state that they have not assessed whether backdoors survive realistic safety post-training, and they describe the prior evidence as inconclusive. Their own pretraining attacks did not persist through the post-training they ran. Zhang and colleagues found denial-of-service backdoors persisting through supervised fine-tuning and DPO, while jailbreak backdoors mostly did not persist. Hubinger and colleagues found stronger persistence in larger models with hidden chain-of-thought, though those backdoors were introduced after pretraining. [EDITOR: Link needed – Hubinger et al. 2024, Sleeper Agents; characterisation taken from Souly et al.]

For anyone writing a control, the consequence is the same in both settings. A cap on the percentage of training data from untrusted sources measures a quantity neither result depends on. In a streaming system the quantity that matters is how many consecutive update steps an adversary can influence, and whether anything downstream would notice.

Where continual-learning theory reaches an impossibility result

Yiting Hu and Lingjie Duan published Theory of Continual Learning Against Data Poisoning Attacks at ICML 2026, modelling adversary and defender as an online zero-sum game over a sequence of tasks. Their first result is a limit. In regularisation-based continual learning, no defence guarantees convergence when an adversary poisons a linear proportion of tasks by injecting unbounded noise or pattern shifts. Two regimes remain defensible: one where attacks are infrequent, one where the noise per attack is bounded. Hu and Duan give a task-to-task verification mechanism for the first and a robust feature defence for the second.

Their experiments produced two findings a practitioner can use, and the second cancels the obvious reading of the first.

Testing on CIFAR-100 with a frozen pretrained vision transformer and a trained linear layer, partitioned into 100 tasks with 10 poisoned, iCaRL held up better than the regularisation baseline under those infrequent shifted attacks. The mechanism is simple: iCaRL trains on a mixture of the current task and stored clean exemplars, so benign data in every update prevents the model drifting fully in the adversarial direction. Clean data mixed into a poisoned update dilutes it, which agrees with the LLM result that continued clean training slowly degrades an implanted backdoor.

Under frequent bounded attacks the same paper finds iCaRL performing significantly worse than the regularisation baselines, because without a regularisation term it is more sensitive to the adversary’s objective and gives up its most sensitive features. Replay is therefore not a general poisoning control. A mechanism that stabilises learning under one adversarial cadence degrades it under another, which means the frequency an attacker can achieve belongs in the security specification and not only in the machine learning team’s hyperparameters.

Their detection result is the more portable one. The task-to-task score, computed from two consecutive parameter updates, showed clear peaks on every poisoned task in both the linear and non-linear settings, while accuracy over the same tasks stayed stable and showed no sign of the attack. Two papers, two model families, one conclusion about what a monitor should watch.

The scope is narrower than the abstract suggests. The impossibility result covers regularisation-based continual learning in a continual linear regression setting, the poisoning in the CIFAR-100 experiment is a constant shift applied to every feature coordinate, and no production update stream was involved. The direction it points is still useful. On the third tier, the defensive question stops being whether poisoned data can be filtered out, and becomes how many consecutive updates an adversary is permitted to touch.

The update itself leaks

The exploratory half of Barreno’s taxonomy has its own streaming attack, and it works nothing like probing a static model.

Query a model with a fixed set of samples. Wait for it to update. Query with the same set again. The outputs differ, and the difference describes what the model learned in between. Ahmed Salem, Apratim Bhattacharya, Michael Backes, Mario Fritz and Yang Zhang demonstrated this at USENIX Security 2020. Updates-Leak presents four attacks in an encoder-decoder formulation, inferring properties of the updating set and, with a generative model trained under a reconstruction loss, recovering samples from it.

The target model is black-box, which is the interesting part, and the attacker needs more than query access to it. Their threat model assumes a shadow dataset drawn from the same distribution as the target’s data and a shadow model sharing the target’s architecture, and the experiments use updating sets of at most 100 samples. The authors relax both assumptions and measure the cost: with a shadow dataset from a different distribution, multi-sample reconstruction degrades by 14% and still beats the baseline by 10%. That puts the attack within reach of a competitor holding domain data, and out of reach of a passing API customer.

This differs from model stealing, which targets the model, and from model inversion, which targets the training data behind a static one. It differs again from the generic query attacks that map a fixed decision boundary. The signal exists only because the model changed, so there is no offline equivalent. Update often and expose a query interface, and you are publishing a sequence of differences.

Salem and colleagues investigated adding noise to the posteriors and noted the difficulty: the model owner cannot sanitise the difference directly, because she does not know when or with what the adversary probes. My own reading of that constraint is that the practical control is the release schedule. Batching updates into fewer, larger versions cuts the number of differences an attacker can observe, at the cost of the freshness that justified the online architecture in the first place. That is an inference from their threat model and not a defence they evaluated, and the trade belongs to the product owner, which is a good argument for writing it down before an incident.

The same shape, in agent memory

An agent with persistent memory writes what it encounters into a store, retrieves it in later sessions, and conditions its behaviour on what it retrieves. An attacker who influences what is written has influenced every future session that reads it.

Johann Rehberger demonstrated the mechanism against Gemini’s long-term memory in February 2025, using indirect prompt injection with delayed tool invocation. Palo Alto’s Unit 42 published a version against agent memory in October 2025. A systematic study in June 2026 tested the surface across model, prompt and system levels and reported two findings that matter here. The aggressive memory write and retrieval policies that improve long-horizon task performance also enlarge the poisoning surface. Existing prompt injection defences cover memory poisoning incompletely.

The distinction between this and everything above must be held, because the shared vocabulary obscures it. Memory poisoning corrupts a retrieval store at runtime. No weight changes, an operator can often read the poisoned entries as text, and deleting an entry removes it from future retrieval. Training-time poisoning changes parameters, leaves nothing legible to inspect, and repair means retraining from a known-good checkpoint. Neither remediation is complete on its own terms: deleting a memory entry does not undo the actions the agent already took on it, and entries stored as embeddings or rolled-up summaries are harder to read than the plain-text case suggests. Two attack surfaces, two owners, two repair paths, and a control designed for one of them gives limited coverage of the other.

What to check, and in what order

Establish which tier each production model occupies, by tracing whether external input can reach a parameter update by any path, the label pipeline included. A system can score in milliseconds and still leave its weights unchanged between monthly retrains. In most organisations nobody has drawn that inventory, and drawing it takes a few hours.

Where such a path exists, find who can write to it. Labels supplied by users, by a reporting button, by a vendor feed or by an outsourced annotation team are all open to influence, and the Gmail case shows organised groups using a user-facing reporting mechanism at scale.

Monitor the relationship between consecutive updates, not the size of any one of them. Hu and Duan’s detection score, computed across two successive parameter updates, separated poisoned tasks from clean ones while accuracy showed nothing. Simple magnitude thresholds do not help here, because Pang’s attack is designed to work at smaller gradient norms than the poisoning that clipping was tuned against.

Mix verified clean data into every update. Two independent results support this, and both are narrow. Clean exemplars limited adversarial drift under infrequent attacks in the continual learning experiments and made things worse under frequent ones, and continued clean pretraining slowly degraded an implanted LLM backdoor without removing it. Treat it as dilution with a known failure mode.

Rate-limit the update boundary as well as the queries. Fewer, larger, reviewed updates reduce both the number of consecutive steps an attacker can shape and the number of differences an Updates-Leak style attack can observe.

Keep a rollback checkpoint and know how long restoration takes. Against an attack that arrives in one update step, recovery time is the control that determines the loss. Rollback is also the one item on this list that does not depend on detection having worked.

None of the six requires a product, and the first is an afternoon’s work with an architecture diagram. That is where defensive practice against this class of attack currently stands.

For the wider treatment of how training data is corrupted, see data poisoning and label flipping. For the taxonomy this belongs to, see AI security fundamentals.