Universal Activation Verbalizer: A Unified Framework for
Cross-Model Activation Explanation

Haiyan Zhao¹, Zirui He¹, Guanchu Wang², Ali Payani³, Yingcong Li¹, Mengnan Du^4,†

¹ New Jersey Institute of Technology, ² University of North Carolina at Charlotte,
³ Cisco Research, ⁴ The Chinese University of Hong Kong, Shenzhen

† Corresponding author

[Paper] [Code] [arXiv]

UAV framework: a donor model's activation is mapped by an adapter to soft tokens for a shared decoder. The adapter is trained for text reconstruction and then for question answering with decoder-side LoRA. — Figure 1. The UAV framework for cross-model activation verbalization. Figure from the paper. Select the image to view it full size.

Overview

Activation verbalization asks what information a language model’s hidden state contains and expresses it in natural language. Existing methods mainly explain a model’s own activations. Universal Activation Verbalizer (UAV) uses a shared decoder with donor- and layer-specific adapters, allowing one decoder to explain activations from different model families and sizes.

First, an adapter learns to turn a donor activation into decoder-readable soft tokens by reconstructing text. Next, instruction-following training teaches the verbalizer to answer questions about the activation, using decoder-side LoRA while the decoder backbone remains frozen. For a new donor, the learned decoder-side LoRA can be reused while a new adapter is trained. UAV evaluates classification, fact retrieval, and gist summarization; the source text is withheld from the verbalizer at evaluation time.

Results

Table 2 compares UAV with trained self-explanation baselines using overall ROUGE-L and BERTScore. On the Qwen3-4B donor, UAV exceeds Activation Oracle and LatentQA on both metrics. On the Llama-3.1-8B donor, UAV is competitive with LatentQA on ROUGE-L and higher on BERTScore.

Donor	Method	Decoder	ROUGE-L ↑	BERTScore ↑
Qwen3-4B	Activation Oracle	Qwen3-4B	0.198	0.234
Qwen3-4B	LatentQA	Qwen3-4B	0.235	0.327
Qwen3-4B	UAV · Full	Qwen3-4B	0.254	0.344
Qwen3-4B	UAV · Adapter-only transfer	Qwen3-4B	0.260	0.347
Llama-3.1-8B	Activation Oracle	Llama-8B	0.273	0.362
Llama-3.1-8B	LatentQA	Llama-8B	0.285	0.370
Llama-3.1-8B	UAV · Full, self-decoding	Llama-8B	0.286	0.379
Llama-3.1-8B	UAV · Full, cross-decoding	Qwen3-4B	0.274	0.369

Table 2. Overall generation scores from the paper; higher is better. “Full” denotes two-stage adaptation, while adapter-only transfer reuses frozen decoder-side LoRA from another donor.

Qualitative Example

In a fact-retrieval example from Table 19, UAV correctly recovers the name of the group from which Westfield Corporation was spun off. The decoder receives the question and activation-derived tokens rather than the source excerpt.

Source and reference

Source excerpt

Westfield Corporation was founded with the spin-off of the Westfield Group in 2014…

Question

What was the name of the group this entity split from?

Reference: Westfield Group

UAV

Model input

Question and activation-derived tokens; the source text is withheld.

Predicted answer

Westfield Group

ROUGE-L: 1.000

Table 19. One fact-retrieval example with an exact-match answer. A selected example does not represent average performance.

BibTeX

@inproceedings{zhao2026universal,
  title     = {Universal Activation Verbalizer: A Unified Framework for Cross-Model Activation Explanation},
  author    = {Zhao, Haiyan and He, Zirui and Wang, Guanchu and Payani, Ali and Li, Yingcong and Du, Mengnan},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing},
  year      = {2026},
  url       = {https://arxiv.org/abs/2605.25903}
}