SAIF: A Sparse Autoencoder Framework for Interpreting and Steering Instruction Following of Language Models

Zirui He, Haiyan Zhao, Yiran Qiao, Fan Yang, Ali Payani, Jing Ma, Mengnan Du*

* Corresponding author

ECML 2026

SAIF framework: extracting an instruction steering vector from SAE latent differences and applying it to guide generation.
Figure 1. SAIF extracts an SAE-based steering vector and applies it to control model output.

Overview

SAIF uses sparse autoencoders to study how language models follow instructions. It identifies instruction-relevant latent features and tests whether intervening on those features changes model behavior.

The analysis examines the semantic relationship between latent features and instructions, their causal effect on generated responses, and the conditions that affect steering, including feature selection, model layer, and instruction placement.

Results

The paper reports that a distinct set of sparse autoencoder latents is associated with instruction-following behavior. The identified features are semantically related to instructions and can steer responses when activated. Experiments examine how these findings transfer across autoencoders and language models of different sizes.

Figure 5: strict and loose steering accuracy across five instruction tasks for three language models.
Figure 5. Steering accuracy across translation, keyword inclusion, and summarization tasks.
Figure 4: examples of strict and loose instruction following in a French translation task.
Figure 4. Examples of strict and loose instruction following in French translation.

BibTeX

@inproceedings{he2026saif,
  title={SAIF: A Sparse Autoencoder Framework for Interpreting and Steering Instruction Following of Language Models},
  author={He, Zirui and Zhao, Haiyan and Qiao, Yiran and Yang, Fan and Payani, Ali and Ma, Jing and Du, Mengnan},
  booktitle={European Conference on Machine Learning (ECML)},
  year={2026},
  eprint={2502.11356},
  archivePrefix={arXiv}
}