AI Security 101: Where Each Attack Belongs and What Contains It

On 4 August 2026 OWASP published the Top 10 for LLM Applications 2026. It is the first edition whose ranking was not set by practitioner voting alone. Three quarters of the weight still came from the vote. The other quarter came from data on 6,639 real incidents pulled from public vulnerability databases and an AI-harm database. Prompt injection stayed at number one, the rank it has held on every edition since the first list in 2023. Excessive agency climbed from sixth place to third, pushing supply chain and poisoning down one each.

The sentence in that release I would put in front of a board is not on the list at all. It is the design principle underneath it. Stop trying to build a model that cannot be fooled, and harden the application around the model so that a fooled model does bounded damage.

That is the bottom line of this page, and of most of what I have written here since 2017. Nothing on the AI attack list is defeated at the model. Every control that has survived a serious attempt to break it is architectural. It limits what the model can reach and what the model may do with what it reaches, and it names the actions a person has to approve first.

What follows is the map: what the words mean, which layer each class of attack belongs to, what the evidence behind it actually is, and which article here treats it properly.

The mechanisms are older than the vocabulary

In 2008 Blaine Nelson and colleagues at Berkeley subverted a spam filter by feeding it crafted training mail. In 2012 Battista Biggio, Nelson and Pavel Laskov poisoned a support vector machine by computing which training points to corrupt. In 2013 Christian Szegedy’s group found that changes too small for a person to see flipped a neural network’s classification, and in 2014 Ian Goodfellow, Jonathon Shlens and Szegedy showed that the change could be computed in a single gradient step.

Those four results contain most of what we still call AI security. NIST’s current taxonomy says the same in more formal language. AI 100-2e2025, published in March 2025, sorts attacks on predictive systems into evasion, poisoning and privacy, then adds supply chain, direct and indirect prompt injection, misuse, and the security of agents for generative ones. Two populations of system, one set of mechanisms.

What changed is the damage a successful attack does. A fooled classifier in 2013 produced a mislabelled photograph. In June 2025 Aim Labs disclosed EchoLeak, CVE-2025-32711, a zero-click flaw in Microsoft 365 Copilot rated 9.3, in which one crafted email that nobody opened caused Copilot to read internal files and send their contents out. Microsoft patched it server-side and reported no exploitation in the wild. Crafted input steered the model, the same mechanism as in 2013. The blast radius was a tenant.

I started testing AI systems for defence clients in the early 2000s, when the question was whether a classifier inside a targeting chain could be fooled. It could, cheaply. It still can, and the thing on the other side of the classifier now holds credentials.

Four words that are not synonyms

Secure AI, safe AI, responsible AI and trustworthy AI describe four problems with four different owners, and collapsing them is the fastest way to spend a security budget on the wrong work. Secure AI is about an adversary. Safe AI is about failure with no adversary at all. Responsible AI is about the harms a working system causes to people. Trustworthy AI is the claim that the other three have been demonstrated rather than asserted. I set the distinctions out at length in Addressing the Full Stack of AI Concerns.

Most of what I have written here is about the first of the four. The second is covered under AI Safety and Alignment, where I have written on the alignment problem and on what I actually think the risk is. The third mostly belongs to somebody else’s blog, though where principles meet practice covers the gap between the two.

Three distinctions everything else depends on

Prompt injection is not jailbreaking, and neither is misalignment

Prompt injection is untrusted content becoming instruction. The adversary is whoever wrote the document, the email or the web page your system read. Jailbreaking is the user of the system subverting the model’s own policy. The adversary is the person at the keyboard. Misalignment is the system pursuing the wrong objective competently, with no adversary anywhere.

Each of the three has its own adversary, and a guardrail tuned to one does nothing to the others. It also explains why the injection problem stays open. Instructions and data arrive through the same channel, and no equivalent of a parameterised query exists to separate them.

Poisoning is not a backdoor

Poisoning degrades or shifts a model’s behaviour across the board. A backdoor implants a specific trigger and stays quiet until the trigger appears. What 250 poisoned documents actually proved works through the study everyone cites and what it did and did not measure, and testing can find backdoors but cannot prove absence covers the 2022 undetectability result that governs what a scanner can tell you. The variants have their own pages: GAN poisoning, neural trojans that survive safety training, poisoning models that learn while you watch, and label flipping.

Model extraction is not an inference attack

Extraction targets the model. Inference attacks target the people in its training data. Model stealing has split into three separate attacks that share a name, and what model inversion actually recovers is usually a class representative rather than a person. Query attacks serve both, and where a model makes a security decision, what the queries buy is reconnaissance for the evasion that follows.

Model, system, agent

I see this missed more often than any other distinction here, including by people selling defences. A finding about a model does not automatically hold for the deployed system around it, and an agent holding tools and credentials has a different blast radius again. AI Model Security is the long version of that argument, including what AI security inherits unchanged from ordinary security, which is most of it.

The model layer

Adversarial examples are a real, unsolved property of trained models, and almost every famous demonstration against a deployed system is weaker than the headline it produced. Adversarial Attacks on AI sorts what has been demonstrated from what has been announced. The method pages behind it cover gradient-based attacks and where the gradient stops helping, perturbation attacks, semantic attacks that leave the perturbation budget behind, saliency attacks and meta-attacks. One more covers evasion, where attackers mostly do something cheaper. Adaptive attacks against ML detection is the evaluation standard the detection market has never adopted.

Two model-layer failures belong here and are filed elsewhere on this site. A demographic differential in false match rate is a differential in exposure to impostors, which makes it a security property. And the explanation you generated to build trust in the model is an attack surface, because it is computed from the model and transfers more to an attacker than to a defender. Feature attribution attacks covers seven years of published work on forging it.

The system layer

Almost every control sold as AI security operates at this layer, and most of it is a classifier. Your prompt injection detector is a text classifier, which means the 2018 literature on fooling text classifiers applies to it directly, at evasion rates reported up to 100%. The instruction does not have to arrive as text at all: it comes in through the camera, or as audio, or inside a document your retrieval index fetched.

Two system-layer problems have nothing to do with the model. Model fragmentation is model sprawl, and a quantised derivative of a model you scanned is a different artefact that you did not scan. Data spoofing is four attacks, and channel authenticity, measurement integrity and model correctness are three properties where no one of them supplies the other two.

The agent layer

OWASP published the first Top 10 for Agentic Applications on 9 December 2025, running ASI01 to ASI10: goal hijack, tool misuse, identity and privilege abuse, agentic supply chain, unexpected code execution, memory and context poisoning, insecure inter-agent communication, cascading failures, human-agent trust exploitation, and rogue agents. Every entry has a documented incident behind it rather than a projection.

The incident that best explains the layer is not an AI attack. In March 2026 Palo Alto’s Unit 42 published Double Agents, showing that an agent deployed on Google Cloud’s Vertex AI Agent Engine inherited a default service account with permissions far wider than its task. A malicious tool inside the agent could pull the account’s token from the metadata service and read every Cloud Storage bucket in the project, then reach restricted container images inside Google’s own infrastructure. Google’s response was a documentation change recommending customers supply their own service account. No model was attacked. The agent was the delivery mechanism and over-permissioned IAM was the vulnerability, which is the shape of most agent incidents I have looked at.

Coverage of this layer here is thin and new. The AI Act now recognises agents and its security rules still miss them is currently the only piece in Agentic AI Security. More is coming, and I would rather say that than pad the category.

What has actually held

One defence has survived fifteen years of people building attacks specifically to break it. Adversarial training, formulated as robust optimization, works only inside the threat model you wrote down. What it buys and what it costs is an order of magnitude in compute, several points of clean accuracy, and nothing at all once the threat model cannot be written as a constraint. That last clause is why it does not save you from prompt injection.

The privacy technologies each cover one thing and are sold as covering more. Differential privacy protects the training set rather than the system, at epsilon values that deployments set near 10. Federated learning keeps raw data on the device and moves the question to whether the party running the aggregation server is honest. Secure multi-party computation hides the inputs and reveals the output exactly as computed, and in machine learning the output is a model or a prediction, both of which leak. Homomorphic encryption buys computation on data the server cannot read, at an overhead that makes the set of correct workloads small and specific. Dynamic data masking protects the view and gives the resulting model no privacy property whatsoever.

Everything else that has held is a permission boundary. Scope what the agent can reach. Require a human on irreversible or high-value tool calls. Log tool invocations and memory writes rather than model outputs. None of it stops an injection, and all of it decides what one costs.

How to read a claim about any of this

Access assumptions matter more in AI security than anywhere else, because white-box results are routinely reported as though they were black-box. What access actually buys covers why the two labels have stopped predicting attack strength on their own, and why every real deployment is gray-box.

Four questions handle most of the vendor material. What was demonstrated, as against announced. On what model, at what access level, with what query budget, against which defence in place. Verified by whom, and reproducible by anyone else. And what a defender does differently on Monday if the claim is true.

A 95% attack success rate with none of those conditions attached cannot be reproduced or compared, which is what makes it a slide.

What is in force, checked 6 September 2026

AI deadlines move, and writing to last year’s calendar is wrong in both directions. Regulation (EU) 2026/1744, the Digital Omnibus on AI, was published in the Official Journal on 24 July 2026 and entered into force on 27 July. It moved the high-risk obligations for standalone Annex III systems from 2 August 2026 to 2 December 2027, and those for AI embedded in Annex I regulated products to 2 August 2028.

What did not move is the part with security content in it. The Article 50 transparency and content-marking duties applied on 2 August 2026 as originally written, with a grace period to 2 December 2026 for machine-readable marking of output from generative systems already on the market. GPAI obligations have applied since August 2025 and the Article 5 prohibitions since February 2025.

The EU AI Act for security teams is the working guide to what that leaves you holding. The specific fights are in their own pieces: Article 15 requires a robustness nobody can measure, the GPAI security chapter is a weight security mandate, and the transparency code relies on watermarks that break. Once the Cyber Resilience Act duty starts, one incident can trigger four reporting regimes. Who owns all of this inside the organisation is the argument in why we need a Chief AI Security Officer.

On the voluntary side, name the edition and the date or the citation is worthless. The LLM Top 10 is the 2026 edition of 4 August 2026. The Agentic Top 10 is ASI01 to ASI10 of 9 December 2025. NIST’s adversarial ML taxonomy is AI 100-2e2025 of March 2025. OWASP added an Agent Control Standard and a framework crosswalk on 1 September 2026, both too new for anyone to have deployment experience with.

The layer underneath all of it

Model security assumes the compute exists. What still keeps frontier training concentrated works through the engineering that decides whether it does, and your multi-region architecture assumes the weather covers what happens when the hazard is deliberate rather than meteorological. The two arguments about compute as a target are what force can and cannot do to a trained checkpoint and why compute was already a target before the think tanks noticed. The first is filed under AI Infrastructure and Compute and the second under AI Geopolitics.

Where to start, by what you own

If you own a model or a training pipeline, start with Adversarial Attacks on AI and What 250 Poisoned Documents Actually Proved, then work through AI Security.

If you own an application built on somebody else’s model, start with AI Model Security for the boundary between what is new and what is not, then Your Guardrail Is a Text Classifier for what your controls actually are.

If you own an agent with tools and credentials, read the OWASP Agentic list first, then treat the permission model as the security review. Agentic AI Security is where this site’s coverage will grow.

If you own the data, AI Privacy covers what models remember and what the privacy technologies actually give you.

If you own compliance, start with The EU AI Act for Security Teams and work through AI Governance and Policy.

And if you want the parts of the field that are not attacks, they are in AI Safety and Alignment, AI Disinformation, Cyber-Kinetic AI, Offensive AI and AI Perspectives.

The defensive practice is roughly fifteen years behind the offensive research, and the gap is not closing on its own. It closes one architecture review at a time, on the question of what your system is allowed to do when the model gets it wrong.