Every Thumbs-Up Can Become a Training Label

In July 2025, Almog Hilel, Idan Shenfeld, Jacob Andreas and Leshem Choshen posted an attack that needs no dataset access at all. Their attacker holds nothing but the ability to send prompts and click the up or down button against a model whose feedback feeds a later preference-tuning run. Prompt the model until it produces either a poisoned or a benign answer, upvote the poisoned one, and wait.

Read their two experiments separately, because most coverage does not. Under privileged access, where the attacker injects preference pairs directly, planted facts about a fictional entity reached 97% accuracy with general capability intact, TinyMMLU holding around 63%. That is the upper bound and it is not the threat model. Under the unprivileged version, feedback only, the same task went from a 5% baseline to 51% with the simple attack and 65% with the stronger variant, with TinyMMLU around 60%.

Fifty-one per cent, from a button. No access to the training set, the weights, or the infrastructure, and ninety per cent of the preference data around it entirely ordinary.

That is a label-flipping attack. It arrived through a button.

Here is the sentence worth forwarding: every interface in your product where a user expresses approval is a write into your labelling pipeline. If that signal reaches a training run without attribution, rate limiting and review, you have handed an annotation seat to anyone with an account.

What the attack was, and why it looked finished

Label flipping is the simplest poisoning attack there is. Take a training example, change the label, leave everything else alone. Cats become dogs. Fraudulent transactions become legitimate ones. Malicious network packets become benign ones. The model learns the association it was shown.

The failure mode is what made it distinctive. A model trained on flipped labels and validated on a test set drawn from the same corrupted pool reports excellent accuracy. The accuracy number is computed against the same lie the model learned. Broad data poisoning degrades your numbers and shows up on a dashboard. Targeted label flipping does not.

By around 2023, the attack looked like it belonged to a previous era of machine learning. Frontier model pretraining is self-supervised. There is no label to flip in a corpus of web text; the model predicts the next token and the text is its own supervision. The literature on flipping labels in support vector machines and k-nearest neighbours read like a description of a world that had moved on.

That reading was wrong, and the reason is worth stating precisely. Labels did not disappear from the pipeline. They moved to a later stage, and the people generating them stopped being employees.

Four places the label lives now

Preference pairs. This is the direct successor and the most alarming one. Reinforcement learning from human feedback trains a reward model on pairwise comparisons: given a prompt and two responses, a human says which is better. That judgement is a label. Flip it and you have flipped a label.

In Preference Poisoning Attacks on Reward Model Learning, the authors evaluated attackers flipping a small subset of preference comparisons across four domains. In most domains they reached near-100% success at 1 to 10% of the data poisoned. In safety alignment they reached nearly 100% success by flipping 0.3%. They also tested state-of-the-art defences developed for other classes of poisoning attack and found limited efficacy in this setting.

Related work fills in the shape. RankPoison flips preference rankings to make a model generate longer sequences, raising inference cost, without degrading its safety alignment scores. Best-of-Venom poisons a reward model to steer sentiment about a named entity and reports that adding the poisoned data does not affect performance on clean preference data. Pathmanathan and colleagues found that poisoning 0.5% of the data through Direct Preference Optimization was enough to elicit a harmful response, where PPO needed at least 4%.

Note the recurring detail across all of them. Clean-data performance is preserved. That is the 2022 signature, unchanged, operating three stages further down the pipeline.

User feedback in production. The LLM Hypnosis result above is the one that should reorder your priorities, because it requires the least. Earlier work in The Dark Side of Human Feedback had already shown that injecting 1% specially crafted prompts through ordinary malicious user accounts, in a black-box setting with no knowledge of the target model, doubled the toxicity score associated with a chosen trigger word. Javier Rando and Florian Tramèr demonstrated universal jailbreak backdoors from poisoned human feedback and were explicit about the assumption that carries the attack: no quality checks on the labels, and preference data flowing from chat interfaces deployed to millions of untrusted users.

If you run a product that collects up and down votes on answers and feeds them anywhere near a tuning run, that assumption describes you.

The LLM judge. Automated evaluation has replaced human annotation across benchmarking, dataset curation and reward modelling, for obvious cost reasons. The judge is now the labeller, which means the judge is now the target. BadJudge, a preprint, documents backdoor vulnerabilities in the LLM-as-a-judge paradigm, in personalised open-source judges and in standardised foundation-model ones. BITE, presented at ICML 2026, turns the judges’ known biases into an attack. Bandit-guided style manipulation inflates scores by one to two points on a nine-point scale at an attack success rate above 65%, and it gets past style control and past detection based on the judge’s own explanations. Other work shows adversarial phrases optimised against a small open-source judge transfer to substantially larger commercial models.

A poisoned judge does not flip one label. It flips a labelling function, and every dataset it touches inherits the flip.

Distributed and crowd-sourced labelling. Federated learning is where label flipping never stopped being studied, because a malicious client controls its own labels by construction. Survey work reports label-flipping succeeding at attacker ratios in the low single-digit percentages of participating clients. The same mechanism has been demonstrated against malware detection systems, naive Bayes spam filters and hardware trojan detection.

And the oldest version of the attack is still running. Spammers have been poisoning Bayesian filters since at least 2005, when Daniel Lowd and Christopher Meek documented good-word attacks against statistical spam filters, and every “report as spam,” “flag this,” and “this was not helpful” control in a production system is a labelling interface exposed to an unauthenticated public. Those filters are the security team’s own machine learning, and they are usually the least examined models it runs.

The one place detection works, and why it does not transfer

Federated learning has produced defences that genuinely work against label flipping. AntiFLipper, a 2025 preprint, detects malicious clients by a simple observation: when a client flips its labels, the global model built by honest participants performs badly on that client’s local data, so its local accuracy collapses. In the reported experiments, malicious node accuracy fell to near zero early on MNIST and within 20 rounds on CIFAR, while honest nodes held. Other approaches cluster output-layer gradients to separate benign from malicious participants.

Those defences work for a structural reason rather than a clever one. Federated learning gives you many independent views of the same labelling task, so a liar diverges from a consensus you can compute.

Centralised preference collection has no such redundancy. When one annotation vendor labels a batch, or one user clicks thumbs-up, there is no second opinion to disagree with. The detection story that works is the one that does not transfer to the place the attack moved to, and I have not seen anyone say this out loud.

What the 2022 version of this article got wrong

This piece previously offered blockchain and differential privacy as promising future countermeasures. I would rather correct that reasoning than delete it quietly, because the same argument is still common.

Differential privacy is a confidentiality control, not an integrity control. It does bound the influence any single training example can have on the trained model, and there is a real literature on that bound giving incidental robustness to poisoning. But the privacy budget tight enough to blunt a targeted flip costs more utility than most teams will accept. The guarantee covers what an observer can learn from the model, not whether the labels were true. Recommending it against label flipping confuses two problems that need different owners. The distinction between secure, safe, responsible and trustworthy AI exists precisely to stop this, and differential privacy belongs on the privacy side of it.

Blockchain answers a question nobody asked. The hard part of label provenance is not tamper-evident storage of the record. It is knowing whether the human or model that produced the label was honest. A distributed ledger gives you a signed, immutable record of a flipped label. Cryptographic hashing over dataset contents is genuinely useful, as poisoning of web-scale corpora shows, and it does not need a chain.

The signature has not changed

Across the results above, targeted poisoning preserved clean-task performance well enough that ordinary benchmarks did not reveal it.

Best-of-Venom’s poisoned reward model scores normally on clean preference data. LLM Hypnosis leaves general capability essentially intact. RankPoison achieves its goal without hurting safety alignment scores. Medical models poisoned at 0.001% of tokens match clean models on the open-source benchmarks used to evaluate them, as covered in data poisoning.

This is the same property that made label flipping worth a name in the first place, and it has survived every change in how models are trained. Your evaluation suite measures whether the model does what its training data said it should. A flipped label changed what the training data said. The evaluation is not lying to you; it is answering a different question than the one you meant to ask.

The practical consequence is that “our benchmarks look fine” is not evidence of an unpoisoned pipeline, and should never be reported as though it were.

What to do

Treat feedback signals as untrusted input, because they are. Attribute every rating to an account. Rate limit per account and per topic. Hold feedback out of any training run until it has passed review, and apply the same standard you would apply to a code contribution from an unknown party. The LLM Hypnosis attack ran from a single account. Per-account caps force an attacker to acquire more accounts or accept less influence from each.

Audit the annotation supply chain like a supply chain. Know which vendor labelled which batch, which annotators worked on it, and whether any single annotator’s labels can be isolated and re-checked. Overlap a sample of every batch across independent annotators and measure disagreement, because manufactured redundancy is the only version of the federated learning defence available to a centralised pipeline.

Do not let an LLM judge be the sole labeller for anything that trains a model. Comparative assessment resists universal adversarial phrases better than absolute scoring. Rotate judge models. Hold out a human-labelled sample and measure the judge against it on a schedule, treating drift as a security signal rather than a quality one.

Plant known flips in your own pipeline. Insert a small set of deliberately flipped labels and confirm your review process catches them before training. You are testing whether your controls work, which is answerable. Whether your data is clean is not a question you can settle.

Stop reporting clean benchmarks as evidence of integrity. Say what the evaluation covers and what it cannot see.

Where the obligations sit

Data and Model Poisoning is LLM05 in the OWASP Top 10 for LLM Applications 2026, published 3 August 2026, and the entry names feedback loops explicitly alongside training datasets and knowledge bases. That wording matters for a risk register: an organisation that maps LLM05 only to its training corpus has missed the surface with the lowest attacker cost.

Article 15(5) of the EU AI Act requires technical solutions that include, where appropriate, measures to prevent, detect, respond to and control for data poisoning. Article 15(4) is the closer fit: high-risk systems that keep learning after deployment must eliminate or reduce as far as possible the risk of biased outputs influencing future operations, and must mitigate those feedback loops. That is the closest any regulation currently comes to naming the attack in this article. Both obligations apply from 2 December 2027 for standalone Annex III systems and 2 August 2028 for AI embedded in Annex I regulated products, following Regulation (EU) 2026/1744, in force since 27 July 2026. Checked 6 September 2026.

The related mechanisms are covered in data poisoning, backdoor attacks and GAN poisoning, and the wider map is in AI security fundamentals.