Rep2Text: Decoding Full Text from a Single
LLM Token Representation

Haiyan Zhao¹, Zirui He¹, Yiming Tang², Fan Yang³, Ali Payani⁴, Dianbo Liu², Mengnan Du^5,†

¹ New Jersey Institute of Technology, ² National University of Singapore,
³ Wake Forest University, ⁴ Cisco Research, ⁵ The Chinese University of Hong Kong, Shenzhen

† Corresponding author

[Paper] [arXiv]

Rep2Text architecture: a target model's last-token hidden state passes through a trainable adapter into the token embedding space of a decoder that reconstructs the input text. — Figure 1. Rep2Text reconstructs text from a single last-token representation. Figure from the paper. Select the image to view it full size.

Overview

Rep2Text studies how much of an input can be recovered from one last-token hidden state in a language model. It uses a trainable adapter to map the target model’s representation into the token embedding space of a decoding model. The decoder then generates a reconstruction of the original sequence.

On 16-token Wikipedia sequences, the method recovers substantial lexical information and preserves much of the input’s meaning across several target and decoder combinations. The paper also studies how recovery changes with model choice, input length, and representation depth. Exact reconstruction remains difficult, especially when a detail is encoded imperfectly.

Results

Table 2 compares Rep2Text with an adapted Vec2Text baseline on Wikipedia inputs of at most 32 tokens. For the Mistral target, Rep2Text improves ROUGE-1, Token F1, and BERTScore. For the Llama target, it improves Token F1 and matches the baseline’s BERTScore, while Vec2Text Base has a slightly higher ROUGE-1.

Target model	Method	ROUGE-1 ↑	Token F1 ↑	BERTScore ↑
Mistral-7B-v0.1	Vec2Text Base	0.38	0.31	0.71
Mistral-7B-v0.1	Vec2Text + Corrector (50 steps)	0.38	0.31	0.71
Mistral-7B-v0.1	Rep2Text	0.41	0.42	0.73
Llama-3.1-8B	Vec2Text Base	0.37	0.29	0.70
Llama-3.1-8B	Vec2Text + Corrector (50 steps)	0.35	0.27	0.68
Llama-3.1-8B	Rep2Text	0.36	0.39	0.70

Table 2. Wikipedia results with Qwen-2.5-14B as the decoder for Rep2Text. Inputs contain no more than 32 tokens; higher is better.

Qualitative Example

In the example below from Table 3, the reconstruction retains the person’s name, birth date, and occupations. One letter differs in the given name: “Phillip” becomes “Philip.”

Original input

Wikipedia sequence

Phil LaMarr
Phillip LaMarr (born January 24, 1967) is an American actor, voice actor, comedian and writer.

Ground truth

Rep2Text

Reconstructed sequence

Phil LaMarr
Philip LaMarr (born January 24, 1967) is an American actor, voice actor, comedian and writer.

Token F1: 0.94 · BLEU: 0.902

Table 3. An in-distribution text-reconstruction example.

BibTeX

@inproceedings{zhao2026rep2text,
  title     = {{Rep2Text}: Decoding Full Text from a Single {LLM} Token Representation},
  author    = {Zhao, Haiyan and He, Zirui and Tang, Yiming and Yang, Fan and Payani, Ali and Liu, Dianbo and Du, Mengnan},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026},
  url       = {https://arxiv.org/abs/2511.06571}
}

Rep2Text: Decoding Full Text from a Single LLM Token Representation