SAEExplainerInterpreting SAE Features with
Activation-Guided Preference Optimization

Jingyi He1,2Haiyan Zhao3Ruxue Shi4Yanguang Liu3Xin Wang4Fei Sun5Mengnan Du1,†
1 The Chinese University of Hong Kong, Shenzhen2 Shanghai Jiao Tong University3 NJIT4 Jilin University5 Institute of Computing Technology, CAS

† Corresponding author

Read the paper Code BibTeX

FROM ACTIVATIONS TO EXPLANATIONS

An explanation should activate
the feature it describes.

SAEExplainer turns a model’s own activation feedback into a learning signal—teaching an explainer to verify, correct, and refine its descriptions of sparse autoencoder features.

SAEExplainer framework: an SAE feature vector is injected into the explainer; SFT initializes the model, candidate explanations are scored by target-model activations, and preference pairs drive two rounds of DPO.
The SAEExplainer framework. Decode feature vectors into language, score candidate explanations with real activations, and improve the explainer through iterative preference optimization.

Abstract

Although Sparse Autoencoders (SAEs) have mitigated the opacity of large language models (LLMs) by decomposing dense representations into sparse features, explaining these features still remains a central challenge. Current explanation methods, however, typically operate within an open-loop paradigm, failing to leverage mechanistic feedback for further refinement. In this paper, we propose SAEExplainer, a training framework that utilizes activation scores as an objective reward signal to train the model for self-correction and iterative bootstrapping. By iteratively verifying and correcting foundational explanations through a two-round optimization process, SAEExplainer achieves continuous improvement in its explanatory capabilities. This mechanism significantly reduces explanation hallucinations and reinforces causal triggering patterns. Extensive experiments demonstrate our approach improves upon established baselines across most metrics, especially in causal triggering and discriminative activation.

THE METHOD

Explain. Verify. Refine.

A closed loop connects natural-language explanations to the target model’s mechanistic response.

01

Learn to explain

Inject the SAE feature vector into the explainer’s residual stream. Supervised fine-tuning on Neuronpedia explanations establishes a foundation for decoding features into language.

SFT warm-up
02

Verify with activations

Generate candidate explanations and synthesize texts from each one. Score these texts using the target feature’s activation, pairing strong candidates with challenging negatives.

Mechanistic preferences
03

Refine the explainer

Train with Direct Preference Optimization, then use the improved explainer to build a new preference dataset. A second round further sharpens explanation faithfulness.

Two rounds of DPO

EXPERIMENTS

More faithful. More specific.

Evaluated on held-out features from Gemma-2-9B, Llama-3.1-8B, and Gemma-2-27B.

6 / 6Best generative accuracyAcross all model–layer settings in Table 1
+9.26 ppGenerative accuracy gainGemma-2-27B, Layer 1, vs. strongest baseline
76.5%Fewer semantic collisionsRelative FSSR reduction vs. Neuronpedia

Generative accuracy (%)

Higher is better ↑
Generative accuracy from Table 1 of the paper, across three target models and two evaluated layers each.
MethodGemma-2-9BLlama-3.1-8BGemma-2-27B
Layer 1Layer 2Layer 1Layer 2Layer 1Layer 2
Activation Oracles20.3326.9731.1721.7729.6710.23
Neuronpedia21.0327.7733.9327.8737.6316.17
Neuronpedia w/ Claude40.0049.4355.6749.6744.3718.80
Neuronpedia w/ GPT-538.8749.3354.4746.7035.2718.60
SAEExplainer Ours46.1354.5058.4052.3353.6326.63

“Layer 1” and “Layer 2” label the two selected evaluation layers for each model, following Table 1; they are not Transformer layer indices. See the paper for the full set of metrics and experimental settings.

A CLOSER LOOK

From a broad topic to a precise trigger.

Gemma-2-9B · Layer 31 · Feature 1943. A case study from Table 2.

Neuronpedia

Baseline

EXPLANATION

“coding syntax and elements related to web development or programming instructions”

GENERATED TEXT

In the HTML template, wrap the button inside a div, then add class="primary" and set aria-label for accessibility.

Feature activation0

SAEExplainer

Ours

EXPLANATION (ABRIDGED)

References to the Dart programming language, dart:core, package:flutter/foundation.dart, the Zone class, async, and Future.

GENERATED TEXT

A comment in package:flutter/foundation.dart pointed to dart:core Zone behavior when an async Future was scheduled and later resolved.

Feature activation135.54

Activation values are specific to this feature and generated text; they are not percentages or aggregate benchmark scores.

REFERENCE

Citation

@inproceedings{he2026saeexplainer,
  title     = {{SAEExplainer}: Interpreting {SAE} Features with
               Activation-Guided Preference Optimization},
  author    = {He, Jingyi and Zhao, Haiyan and Shi, Ruxue and
               Liu, Yanguang and Wang, Xin and Sun, Fei and Du, Mengnan},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods
               in Natural Language Processing},
  year      = {2026}
}