AdaptiveK: Complexity-Driven Sparse Autoencoders for Interpretable Language Model Representations

Yifei Yao1, Hanrong Zhang2, Mengnan Du3,*

1 Zhejiang University, 2 University of Illinois Chicago, 3 The Chinese University of Hong Kong, Shenzhen

* Corresponding author

Framework diagram for AdaptiveK: Complexity-Driven Sparse Autoencoders for Interpretable Language Model Representations
Figure 2. AdaptiveK uses predicted context complexity to set an input-specific SAE sparsity budget. Select the image to view it full size.

Overview

AdaptiveK adjusts the number of active sparse autoencoder features according to the semantic complexity of each input. A linear probe predicts complexity from a language model representation. The predicted score determines an input-specific sparsity budget, allocating more features to complex contexts and fewer to simpler ones.

Results

Reconstruction–Sparsity Trade-off

Figures 5 and 6 compare AdaptiveK with SAE baselines across a range of sparsity levels. They report L2 reconstruction loss and unexplained variance for Pythia-70M, Gemma-2-2B, Qwen-3-8B, and Phi-14B.

Figure 5 Pythia-70M L2 reconstruction loss Pareto frontier(a) Pythia-70M
Figure 5 Gemma-2-2B L2 reconstruction loss Pareto frontier(b) Gemma-2-2B
Figure 5 Qwen-3-8B L2 reconstruction loss Pareto frontier(c) Qwen-3-8B
Figure 5 Phi-14B L2 reconstruction loss Pareto frontier(d) Phi-14B
Figure 5. L2 reconstruction loss Pareto frontier across four language models. Lower values indicate better reconstruction at a given sparsity level; figures from the paper.
Figure 6 Pythia-70M Unexplained variance Pareto frontier(a) Pythia-70M
Figure 6 Gemma-2-2B Unexplained variance Pareto frontier(b) Gemma-2-2B
Figure 6 Qwen-3-8B Unexplained variance Pareto frontier(c) Qwen-3-8B
Figure 6 Phi-14B Unexplained variance Pareto frontier(d) Phi-14B
Figure 6. Unexplained variance Pareto frontier across four language models. Lower values indicate better reconstruction at a given sparsity level; figures from the paper.

BibTeX

@misc{yao2026adaptivek,
  title={AdaptiveK: Complexity-Driven Sparse Autoencoders for Interpretable Language Model Representations},
  author={Yao, Yifei and Zhang, Hanrong and Du, Mengnan},
  year={2026},
  eprint={2508.17320},
  archivePrefix={arXiv}
}