Language Ranker: A Metric for Quantifying LLM Performance
Across High and Low-Resource Languages

Zihao Li1,*, Yucheng Shi2, Zirui Liu3, Fan Yang4, Ali Payani5, Ninghao Liu2, Mengnan Du1

1 New Jersey Institute of Technology, 2 University of Georgia, 3 University of Minnesota at Twin Cities,
4 Wake Forest University, 5 Cisco Research

* Work done during a remote internship at New Jersey Institute of Technology.

Language Ranker compares English and translated sentences through a language model, extracts their layer representations, and measures cosine similarity between them.
Language Ranker framework. English and translated texts are represented by the model and compared using cosine similarity. Select the image to view it full size.

Overview

Large language models are trained on corpora that differ greatly in coverage across languages. This paper introduces Language Ranker, an intrinsic metric for comparing a model’s representations of a language with those of corresponding English text. The resulting scores provide a way to examine relative multilingual capability, including for languages with limited evaluation resources.

The method uses parallel sentences from OPUS-100. For each pair, it takes the last-token representations at selected transformer layers, computes cosine similarity between the target language and its English translation, and averages the similarities across layers. Languages are then ranked by this score.

Experiments with Llama 2, Llama 3, Qwen, Mistral, and Gemma show higher scores for many high-resource languages and lower scores for low-resource languages. The analysis also compares the scores with language shares in pre-training corpora and with downstream multilingual reasoning tasks. The similarity score is an indirect measure of capability; the paper reports these comparisons to support its interpretation.

Results

Figure 1 shows the similarity to English across model layers for Llama 2, Gemma, Mistral, and Qwen 7B. In these selected languages, the high-resource group generally scores above the low-resource group. The pattern appears across all four models, although individual languages and layers differ.

Cosine similarity between English and ten other languages across model layers for Llama 2, Gemma, Mistral, and Qwen 7B. High-resource languages generally rank above low-resource languages.
Figure 1. Similarity to English across languages and model layers. Figure from the paper.

For Llama 2, the selected examples in Table 1 illustrate the relationship with pre-training data: German and French have similarity scores of 0.723 and 0.737, respectively, while Kannada and Urdu score 0.236 and 0.275. The paper reports their corresponding language shares as 0.17%, 0.16%, and at most 0.01% for Kannada and Urdu.

Why Language Ranker Is Meaningful

Figure 4 provides a view of the representation spaces behind these scores. In Gemma 7B, the four high-resource languages in the top row occupy broader regions of the projected space, while the four low-resource languages in the bottom row are concentrated in much narrower regions. This difference supports the paper’s interpretation that representation geometry reflects differences in language coverage. The projection offers qualitative evidence alongside the similarity scores and downstream evaluations.

Gemma 7B embedding-space projections for German, Spanish, French, and Chinese in the top row and Yiddish, Oriya, Kinyarwanda, and Kazakh in the bottom row. The bottom-row representations appear more narrowly concentrated.
Figure 4. Gemma 7B embedding-space visualization: four high-resource languages (top) and four low-resource languages (bottom). Figure from the paper.

BibTeX

@inproceedings{li2025language,
  title     = {Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages},
  author    = {Li, Zihao and Shi, Yucheng and Liu, Zirui and Yang, Fan and Payani, Ali and Liu, Ninghao and Du, Mengnan},
  booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
  volume    = {39},
  number    = {27},
  pages     = {28186--28194},
  year      = {2025},
  doi       = {10.1609/aaai.v39i27.35038}
}