SAE-SSV: Supervised Steering in Sparse Representation Spaces
for Reliable Control of Language Models

Zirui He1, Mingyu Jin2, Bo Shen1, Ali Payani3, Yongfeng Zhang2, Mengnan Du1,*

1 New Jersey Institute of Technology, 2 Rutgers University, 3 Cisco

* Corresponding author

SAE-SSV framework: encode model activations with a sparse autoencoder, select task-relevant dimensions using linear probes, and train a steering vector in the selected subspace with alignment, language-modeling, and sparsity losses.
Figure 1. SAE-SSV selects a task-relevant SAE subspace and optimizes a sparse steering vector within it. Figure from the paper. Select the image to view it full size.

Overview

Steering changes a language model’s internal activations to guide its output without retraining the model. SAE-SSV performs this intervention in a sparse autoencoder (SAE) representation space. It aims to identify a small set of dimensions that are relevant to a target behavior, then use those dimensions to control open-ended generation.

The method encodes activations into SAE features and uses supervised linear probes to rank task-relevant dimensions. A steering vector is initialized from positive and negative examples and optimized only within the selected subspace. Its training objective encourages the desired behavior while preserving generation quality and keeping the intervention sparse. The paper evaluates sentiment, political polarity, and truthfulness steering on Llama-3.1-8B and Gemma-2 models.

Results

Table 1 compares steering success rates with four baselines across three models and three tasks. SAE-SSV achieves the highest reported success rate in each model–task setting. The full paper also reports lexical diversity and entropy changes to assess generation quality.

ModelMethodSentiment ↑Politics ↑Truthfulness ↑
Llama-3.1-8BCAA45.643.728.7
Llama-3.1-8BRePe24.726.216.2
Llama-3.1-8BTop PC28.424.314.9
Llama-3.1-8BITI41.145.231.2
Llama-3.1-8BSAE-SSV63.260.534.1
Gemma-2-2BCAA39.645.424.6
Gemma-2-2BRePe27.236.611.6
Gemma-2-2BTop PC23.835.012.2
Gemma-2-2BITI41.242.122.3
Gemma-2-2BSAE-SSV52.861.331.7
Gemma-2-9BCAA42.339.319.8
Gemma-2-9BRePe19.722.49.2
Gemma-2-9BTop PC21.429.110.6
Gemma-2-9BITI41.233.821.4
Gemma-2-9BSAE-SSV48.555.027.2
Table 1. Steering success rate (SR, %) across model and task settings; higher is better. This view shows SR columns from the published table.

Interpretable Steering Features

Table 2 lists the ten SAE features with the largest learned weights in the sentiment steering vector for Llama-3.1-8B. Feature descriptions come from Neuronpedia; positive and negative values indicate the direction of each learned steering weight.

RankFeature explanationSAE feature #Weight
1Negative sentiments towards characters in movies2305−6.76
2Negative sentiments and criticisms related to performance or quality14086−3.24
3Phrases related to notable achievements in the entertainment industry123222.79
4Punctuation and symbols indicative of structural elements in text207672.32
5Mentions of achievements and recognition in a professional context288571.99
6Phrases and terminology related to legal injunctions and restrictions2268−1.89
7Components related to specific abilities or skills in performance29039−1.86
8References to historical or legendary figures and events28858−1.68
9Phrases indicating misinformation, contradictions, or inaccuracies14391−1.46
10Questions and expressions of disappointment or concern13758−1.43
Table 2. Top-ten SAE features in the sentiment steering vector for Llama-3.1-8B. Explanations and weights reproduce the published table.

BibTeX

@inproceedings{he-etal-2025-sae,
  title     = {{SAE}-{SSV}: Supervised Steering in Sparse Representation Spaces for Reliable Control of Language Models},
  author    = {He, Zirui and Jin, Mingyu and Shen, Bo and Payani, Ali and Zhang, Yongfeng and Du, Mengnan},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  pages     = {2207--2236},
  year      = {2025},
  publisher = {Association for Computational Linguistics},
  doi       = {10.18653/v1/2025.emnlp-main.112},
  url       = {https://aclanthology.org/2025.emnlp-main.112/}
}