Explainable AI Frameworks in 2026: What Still Ships

Skater last shipped a release on 21 September 2018. LIME last shipped on 26 June 2020, DeepLIFT on 11 November 2020, the What-If Tool on 12 October 2021, and AIX360 on 31 July 2023. The December 2021 version of this page recommended eight explainable AI frameworks. Of the six that were open-source libraries, only SHAP is still releasing.

Interpretability work did not slow down over those five years. It relocated. SAELens put out 257 releases between April 2024 and August 2026, and TransformerLens reached 79 over a longer run. TransformerLens shipped on 2 September 2026 and SAELens on 30 August. Researchers moved from explaining a classifier’s output to decomposing a language model’s internals, and a 2021 tool list now points readers at dead code.

Two questions decide which of these tools you can use: what the model returns to you, and whether you can afford to expose an explanation to a caller you do not control. A companion piece on explanation as a security control takes up the second question at length, so I keep it short below.

The census

Six of the eight original entries were open-source tools, and each can be dated by its last published release. Dates read from PyPI on 5 September 2026.

The remaining two entries were not open-source libraries and I miscategorised them in 2021. Activation Atlases is a 2019 Distill paper from a Google and OpenAI collaboration, and a good one, but nobody installs it. Rulex is a commercial platform that still trades in 2026, and I listed it beside the open-source libraries without saying so. I have removed it.

Skater is the worst entry in that list. Oracle announced its acquisition of DataScience.com on 16 May 2018, and Skater’s final release followed that September. Anyone who took my 2021 recommendation and started work on Skater was using code that had already been static for three years.

What survived, and what each one does

Three maintained libraries cover most of the classical interpretability workload: SHAP and Captum for attribution, InterpretML for glass-box tabular models.

SHAP computes Shapley values, which come from Lloyd Shapley’s 1953 method for dividing the payoff of a cooperative game among players according to what each contributed. Scott Lundberg and Su-In Lee applied the construction to feature importance at NeurIPS 2017. SHAP remains useful because of additivity: the attributions sum to the difference between the model’s prediction for one instance and its average prediction across the dataset. That gives you an arithmetic check on the explanation, which most attribution methods do not offer.

Generic exact Shapley computation scales exponentially in the number of features, so practical explainers either exploit model structure or sample. TreeSHAP is exact for tree ensembles because it uses their structure. KernelSHAP works on any model, estimating the values through weighted regression and running slowly as a result. It assumes the features are independent, which is false for most real tabular data. Know which explainer produced your number.

Captum is PyTorch’s attribution library, and it includes DeepLIFT alongside Integrated Gradients, GradientSHAP, occlusion and layer-level attribution. It shipped in April 2026. If you are attributing a PyTorch model and reaching for the 2017 DeepLIFT repository, use Captum. One caveat the documentation states plainly: Captum implements DeepLIFT’s Rescale rule only, and not the RevealCancel rule from the original paper.

InterpretML is Microsoft’s library, and it solves a different problem from the other two. Its headline model is the Explainable Boosting Machine, a generalised additive model with pairwise interactions that competes with gradient-boosted trees on tabular data while remaining directly readable. Microsoft shipped it in March 2026 with 71 releases behind it. An EBM requires no separate explanation instrument, because its fitted form is the explanation. I return to that below.

LIME still runs, and the 2016 paper by Marco Tulio Ribeiro, Sameer Singh and Carlos Guestrin remains the clearest statement of what local surrogate explanation attempts. LIME perturbs the input, records the model’s predictions on the perturbed inputs, and trains a simple interpretable model on that sample. The reference implementation has received no release in six years, and its perturbation step is the specific weakness that later attack work exploits.

What Google built after the What-If Tool

Google’s PAIR team built the What-If Tool for black-box probing of classification and regression models. They later built the Learning Interpretability Tool, which is broader. PAIR still lists both, so this is a succession in practice and not an announced retirement. LIT supports text, image and tabular data, runs as a standalone server or inside a notebook, and by its 2024 releases it handled foundation models including Gemini, Gemma, Llama and Mistral through prompt debugging workflows.

Sequence salience marks the generational change. It attributes a generated token to the preceding tokens that produced it. A single fixed-output feature attribution cannot summarise a model that emits one token at a time conditioned on everything it has already emitted, though you can still apply attribution to one selected token. The tooling changed because the models did.

LIT’s last package release was December 2024, which describes its release cadence and says nothing about whether Google still uses it.

Where the interpretability effort went

The largest change since 2021 happened outside this tool category.

Anthropic published its first dictionary learning result in 2023 and a scaled follow-up in 2024, using sparse autoencoders to decompose a language model’s activations into interpretable features. The team demonstrated the result by amplifying one feature to produce a version of Claude that steered every conversation toward the Golden Gate Bridge. Sparse autoencoders became the dominant research direction in interpretability, and funders and toolbuilders followed.

Then the evaluations arrived.

In March 2025 Google DeepMind’s mechanistic interpretability team published its negative results and announced that it was deprioritising fundamental sparse autoencoder research. The team had tested whether SAEs helped with out-of-distribution probing for harmful intent in user prompts. They did not beat linear probes. Training SAEs specialised on chat data recovered about half the difference and still lost. The team stated its position plainly: sparse autoencoders have some use. The team does not expect them to transform interpretability, and it suspects the field has over-invested in them.

Two 2025 papers reported the same failure against simple baselines on concept detection and steering. Heap and colleagues found that common SAE reconstruction and automated-interpretability metrics score about as well on randomly initialised transformers as on trained ones, which indicts the evaluation method more than the models. That paper has since been published at ICLR 2026.

A 2026 position paper reconciles the findings by narrowing the claim. The negative results all concern acting on known concepts, where you already know what you are looking for and a linear probe is a strong baseline. The positive results concern discovering unknown concepts, where no baseline exists because nobody yet knows what to probe for. A May 2026 preprint pushes back further, reporting that supervised feature selection lets SAEs approach the steering performance of the baselines that beat them in 2025. Treat that one as a preprint.

The gap between claim and evidence here is unusually well documented, because a major lab published against its own investment. Release counts measure developer activity and not scientific validation, and the two diverged sharply between 2024 and 2026. If you are budgeting interpretability work in 2026 on the assumption that sparse autoencoders are a settled instrument, the people who built the instruments disagree with you.

Whether an explanation is any good

The randomisation test Heap and colleagues ran on sparse autoencoders is seven years old.

Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt and Been Kim published Sanity Checks for Saliency Maps at NeurIPS 2018. They proposed two tests. The model parameter randomisation test compares a saliency method’s output on a trained model against its output on a randomly initialised network of the same architecture. The data randomisation test compares its output on a model trained on true labels against its output on a model trained on randomly permuted labels. Several widely used saliency methods produced similar maps under both conditions. The paper’s comparison case is an edge detector, which reads neither the model nor the training data and still produces output resembling a saliency map. Their conclusion: visual inspection is a poor guide to whether an explanation responds to the model and the data at all.

A defender who runs those two tests learns whether an explanation read the model at all. An explanation invariant to a model’s parameters cannot detect a backdoor, a poisoned training set or a discriminatory decision rule, because it never read the parameters. It fails in the same place the scaffolding attack described below succeeds, and it gets there without an adversary.

Anna Hedström and colleagues collected the response to this problem into Quantus, published in the Journal of Machine Learning Research in 2023. It implements more than 35 metrics across six categories: faithfulness, robustness, localisation, complexity, randomisation and axiomatic. A faithfulness metric usually replaces the features the explanation ranked highly with a baseline value and measures the change in the model’s output. Randomisation metrics implement Adebayo’s tests. Quantus last released in July 2025.

Then the metrics turned out to disagree. Hedström and colleagues published MetaQuantus in Transactions on Machine Learning Research the same year, a tool whose stated purpose is identifying which explanation-quality estimators are reliable. The competing OpenXAI benchmark states that Quantus’s robustness measures have been superseded and that it supports no fairness metrics. Faithfulness scores also depend on which value replaces a removed feature, and black, white, random and uniform choices produce different rankings of the same explanation methods.

Two practical consequences. First, run the parameter and data randomisation tests before trusting any attribution in an assurance context. They are cheap and they need no ground truth. A method that fails them is reporting on the input alone. Second, treat a reported faithfulness score as a number produced by one metric under one baseline choice, and ask for both before comparing it with anything.

Choosing by what the model returns

What the model gives back determines your options more than its architecture does.

Weights and training data. Everything above is available, including the mechanistic tooling, which requires running the model and reading its activations. This is the position of the organisation that trained the model, and of anyone running open-weight models locally.

Weights only. Attribution and activation analysis both work. Training-data attribution does not, which removes an entire class of question about why a behaviour exists at all.

API access returning scores or probabilities. LIME and KernelSHAP were designed for exactly this. Both treat the model as a prediction function and need no weights and no gradients. Captum’s gradient-based methods need gradients, which this interface omits. Nothing mechanistic runs here either.

API access returning hard labels or free text only. Perturbation-based attribution degrades badly without a continuous score to regress against, and most of the tooling above assumes one. Many enterprises are in this position for the frontier models they deploy.

What the model returns also separates a model-level finding from a system-level one. An attribution telling you which features drove a classifier’s output says nothing about the retrieval step that selected the input, the tool an agent called afterwards, or the human who approved the result. Almost every technique here explains a model. Very little of it addresses the retrieval step, the tool call or the human approval, and none of it works on an agent doing all three.

The case for not explaining

Cynthia Rudin argued in Nature Machine Intelligence in 2019 that high-stakes decisions call for inherently interpretable models, and that explaining black boxes perpetuates bad practice and can cause serious harm. A substantial literature since has shown that interpretable models stay competitive on many tabular problems, using sparse rule lists, monotonic models and generalised additive models, without a second model to report what the first one did.

SHAP attributes a black-box output under a defined game, background distribution and masking scheme. An EBM exposes its fitted functional form directly. Where the two options reach comparable accuracy on your data, the second removes a failure mode that the first only instruments.

What an explanation costs

Post-hoc explanation adds an output channel to your model, and information moves along it in both directions.

Dylan Slack and colleagues demonstrated at AIES 2020 that an adversary controlling the model can fool LIME and SHAP. Their scaffolding technique detects the synthetic points those methods generate, which lets a biased classifier go on predicting as it did on real inputs while the explanation reports whatever the adversary chose. Reza Shokri, Martin Strobel and Yair Zick showed at AIES 2021 that backpropagation-based explanations leak substantial information about individual training records, because they expose statistics describing the decision boundary near an input.

That exposure connects to attacks I have covered separately. Explanations improve model extraction, they raise the success rate of inference attacks against training data, and gradient-based explanations supply the gradient information an attacker would otherwise estimate through repeated querying.

Instrument your models anyway. Put the explanation endpoint in the threat model, and decide deliberately who gets access to it. The companion article works through the attacks and through the EU AI Act’s explanation duty.