Predicting Multilingual Classification and Translation Performance of LLMs with Cross-Lingual Alignment โ Is English Enough?¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/KathyHaem/cla-metrics
Area: Multilingual & Translation
Keywords: Cross-Lingual Alignment, Machine Translation Evaluation, Pointwise Mutual Information (PMI), English Pivot Hypothesis, Large Language Models
TL;DR¶
This paper comprehensively examines 27 cross-lingual alignment (CLA) variants for predicting LLM performance across classification and machine translation over 44 languages, proposes a target-language independent PMI translation evaluation metric, and demonstrates via convex combination analysis that LLMs predominantly use English as an internal pivot for translation.
Background & Motivation¶
Multilingual large language models exhibit remarkable zero-shot transfer capabilities across diverse languages, achieving strong performance on downstream natural language processing benchmarks without explicit task-specific parallel fine-tuning. To understand and estimate these multilingual capabilities without incurring the heavy computational cost of running full evaluation suites, recent literature has explored cross-lingual alignment (CLA) scores. Prior investigations have successfully used embedding-level and tokenizer-level CLA metrics to predict performance on non-English understanding tasks by measuring alignment with English representations. However, existing CLA research has remained almost entirely confined to classification benchmarks and encoder-only architectures, leaving causal decoder-only models and generative sequence tasks largely unexplored.
A critical missing link lies in generative tasks such as machine translation, where models must actively map concepts from a source language into a target language. When translating from a source to a target language, intuition suggests that translation quality should be governed by the direct alignment between the source and target representations. Yet, emerging mechanistic interpretability findings suggest that LLMs frequently translate foreign prompts into English in their intermediate layers before generating the final output. If English truly acts as an internal pivot, it raises a fundamental research question: are source-to-English and English-to-target alignments sufficient to predict translation quality, rendering direct source-to-target alignment redundant? Investigating this hypothesis across diverse target languages has been severely impeded by traditional evaluation metrics like chrF, whose scores strongly depend on the target language's writing system and character information density, thus preventing fair cross-target correlation analysis.
To overcome these barriers, this work presents a systematic study bridging embedding extraction, CLA metrics, and downstream translation across 44 typologically diverse languages. The core idea is to introduce a target-language independent Pointwise Mutual Information (PMI) translation metric and model translation quality through convex combinations of source-English-target alignments, providing quantitative evidence that LLMs rely predominantly on English as an internal pivot while bypassing it only for closely related language pairs.
Method¶
Overall Architecture¶
The proposed analytical framework evaluates how sentence representations and alignment metrics predict downstream LLM performance across both classification and generative translation. The pipeline first extracts fixed-length sentence embeddings from decoder-only language models using various pooling strategies and a novel multilingual few-shot extraction scheme. Next, cross-lingual alignment scores are computed via cosine similarity, margin-based retrieval accuracy, or neuron-wise correlation. For machine translation, a length-normalized PMI metric is introduced to neutralize target-language character and prior distribution biases. Finally, an ensemble convex combination grid search identifies the predictive contribution of each alignment direction (src-en, en-tgt, src-tgt) to downstream quality.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multilingual Parallel Corpus<br/>Flores-200 / 44 Languages"] --> B["Multilingual Few-Shot Embedding Extraction<br/>Wikidata entity description prompt"]
B --> C["Cross-Lingual Alignment Scoring Space<br/>Neuron correlation ANC / xSIM retrieval"]
C --> D["Target-Independent PMI Translation Metric<br/>Conditional likelihood minus unconditional prior"]
D --> E["Three-Way Convex Combination Modeling<br/>Grid search for optimal alpha and beta weights"]
E --> F["Downstream Prediction & Pivot Verification<br/>Quantifying internal English pivot mechanisms"]
Key Designs¶
1. Multilingual Few-Shot Embedding Extraction: Unbiased Sentence Semantic Condensation Because causal Transformers employ autoregressive attention masking, later tokens naturally accumulate broader contextual information than earlier ones. Traditional sentence extraction methods rely on last token hidden states, position-weighted means (p-wm), or English prompt wrappers (e.g., "This sentence means in one word:"). However, English prompt wrappers inherently bias representations toward English, while translating prompts into 44 languages introduces human translation inconsistencies and labor bottlenecks. To overcome this limitation, this paper introduces a training-free multilingual few-shot extraction prompt. Using Wikidata entity descriptions, the model is presented with five few-shot demonstrations pairing entity descriptions with their corresponding entity names in the target language. The sentence to be embedded is supplied as the final description, and the embedding is extracted from the hidden state immediately preceding the predicted entity token. This guides the LLM to compress the entire sentence semantics into a single vector without external English prompt contamination.
2. Cross-Lingual Alignment Scoring Space: Mitigating Anisotropy with Neuronal and Margin Metrics Given sentence embedding matrices for the source language \(S \in \mathbb{R}^{d \times D}\) and target language \(T \in \mathbb{R}^{d \times D}\) over \(D\) parallel sentences, direct cosine similarity often fails due to representation anisotropy. To achieve robust metric estimation across 27 representation-metric configurations, the framework evaluates Average Neuron-wise Correlation (ANC) and margin-based parallel retrieval (xSIM). ANC calculates the Pearson correlation coefficient along the feature dimension (across the rows of \(S\) and \(T\)) for each neuron \(j \in \{1, \dots, d\}\), effectively bypassing vector norm and angular drift issues. Meanwhile, xSIM mitigates scale inconsistencies by computing retrieval accuracy using margin distance (dist) or ratio over the \(k\)-nearest neighbors within each language's native space. All alignment scores are extracted across layers, and the layer exhibiting the peak alignment score is selected.
3. Target-Independent PMI Translation Metric: Decoupling Writing System Bias Conventional character n-gram translation metrics such as chrF are deeply biased by target-language orthography: ideographic scripts like Chinese pack substantially more semantic information per character than Latin alphabets, resulting in systematically lower chrF values regardless of actual translation quality. This discrepancy creates distinct language clusters in correlation plots and prevents pooling results across different target languages. To establish a universal metric that enables joint cross-target correlation, the authors formulate translation quality via Pointwise Mutual Information (PMI) by subtracting the unconditional (prior) log-likelihood of the translation from its conditional log-likelihood given the source text:
For a generated translation sequence of \(n\) tokens following an \(m\)-token few-shot prompt, the length-normalized log-likelihood is computed as:
The unconditional prior \(p(y)\) is evaluated by prompting the model with five target-language-only sentences from the same domain. Because language-specific factors influencing translation probability (e.g., tokenization granularity, vocabulary frequency, and character density) are equally present in both the conditional and prior distributions, taking their difference cancels out target-language bias and yields a comparable quality measure across language pairs.
4. Three-Way Convex Combination Modeling: Unraveling Internal Routing and Typological Boundaries To determine whether translation performance is governed by direct source-target alignment or mediated through English, the authors formulate an ensemble linear objective combining three alignment directions:
where \(\alpha \in [0, 1]\), \(\beta \in [0, 1 - \alpha]\), and the weights are optimized over a fine grid (step size \(1/25\)) to maximize the Pearson correlation with translation PMI. Across the full 44-language suite, the optimal direct weight \((1 - \alpha - \beta)\) consistently approaches zero, confirming that English alignments capture virtually all predictable variance. To test whether this pivot phenomenon persists when direct translation is linguistically trivial, the authors evaluate a targeted subset of seven closely related, mutually intelligible language pairs (14 directed pairs, such as Czech-Slovak, Spanish-Catalan, Afrikaans-Dutch, and Hindi-Urdu).
Key Experimental Results¶
Main Results¶
Experiments were conducted using three open-weight model families across their base and instruction-tuned variants: Qwen3-14B, gemma-3-12b, and Ministral-3-14B. For monolingual benchmarks (SIB-200 topic classification and Belebele reading comprehension), src-en CLA achieves strong correlations with task accuracy. For machine translation evaluated on Flores-200 and validated out-of-domain on BOUQuET, the optimal convex combinations achieve correlations exceeding 90%.
| Task / Benchmark | Model | Best Representation | Best Metric | Correlation (\(\rho\)) | Comparison to Direct src-tgt |
|---|---|---|---|---|---|
| SIB-200 (Classification) | Qwen3-14B-Base | prompt | xSIM abs | 81.8% | โ |
| SIB-200 (Classification) | Ministral-3-Base | prompt | xSIM abs | 97.5% | โ |
| Belebele (Comprehension) | Qwen3-14B-Base | prompt | ANC | 91.7% | โ |
| Belebele (Comprehension) | Ministral-3-Base | p-wm | ANC | 97.4% | โ |
| Flores MT (chrF) | Qwen3-14B-Base | p-wm | ANC | 82.0% | +22.6% over src-tgt (59.4%) |
| Flores MT (chrF) | Ministral-3-Base | fewshot | xSIM abs | 87.6% | +6.5% over src-tgt (81.1%) |
| Flores MT (PMI) | Qwen3-14B-Base | p-wm | cos | 91.2% | Direct src-tgt weight: 0.0% |
| Flores MT (PMI) | Ministral-3-Base | p-wm | ratio | 90.5% | Direct src-tgt weight: 4.0% |
Ablation Study¶
The ablation investigates the allocation of convex weights when predicting global PMI across all 44 languages versus the subset of seven closely related language pairs, alongside out-of-domain validation on BOUQuET.
| Language Evaluation Set | Model Variant | Optimal Config | Optimal \(\rho_{\oplus}\) | Val. \(\rho_{val}^{\oplus}\) | Direct Weight \((1-\alpha-\beta)\) | Core Mechanistic Implication |
|---|---|---|---|---|---|---|
| 44-Language Global Set | Qwen3-14B-Base | p-wm + cos | 91.2% | 91.1% | 0.0% | Complete reliance on English pivot |
| 44-Language Global Set | Qwen3-14B-Inst | p-wm + cos | 82.9% | 88.5% | 8.0% | English pivot dominates (92.0% weight) |
| 44-Language Global Set | gemma-3-12b-pt | last + ANC | 68.0% | 56.1% | 0.0% | Direct src-tgt provides zero gain |
| 44-Language Global Set | Ministral-3-Base | p-wm + ratio | 90.5% | 88.9% | 4.0% | English pivot captures 96.0% weight |
| 7 Related Language Pairs | Qwen3-14B-Base | mean + cos | 84.7% | 75.2% | 88.0% | Pivot bypassed; direct alignment dominates |
| 7 Related Language Pairs | gemma-3-12b-pt | โ + Eflo | 76.8% | 72.2% | 84.0% | Direct alignment dominates; tokenizer alignability active |
| 7 Related Language Pairs | Ministral-3-Base | mean + cos | 90.0% | 85.2% | 84.0% | Direct morphological transfer without English pivot |
Key Findings¶
- Empirical Confirmation of the English Pivot Hypothesis: Across the 44-language global translation benchmark, the optimal convex weight for direct src-tgt alignment is near-zero (0.0% to 12.0%), while src-en and en-tgt weights capture almost all predictive power. This provides behavioral evidence that LLMs internally route cross-lingual generation through an English semantic space.
- Bypassing the Pivot in Typologically Related Languages: When evaluating mutually intelligible language pairs with high lexical and syntactic overlap, the routing behavior inverts dramatically: direct src-tgt alignment weight surges to 84%โ88%, indicating that LLMs bypass English mediation when direct translation is trivial.
- Practical Recommendations for Metric Selection: Among all 27 combinations, position-weighted mean (p-wm) paired with Average Neuron-wise Correlation (ANC) achieves the most consistent performance for system-level evaluation. When sample-level scores are required, p-wm paired with xSIM abs is recommended. Tokenizer-level alignability (Eflomal) lags behind embedding scores but serves as a viable fallback when model hidden states are inaccessible.
Highlights & Insights¶
- PMI Overcomes Script-Specific Evaluation Discrepancies: Formulating translation quality via length-normalized Pointwise Mutual Information effectively eliminates language prior effects, transforming an otherwise clustered and script-biased metric into a smooth, cross-target correlatable proxy.
- Convex Ensembles as Probing Tools: Rather than treating alignment metrics as isolated predictors, optimizing convex combination weights offers an elegant methodology for uncovering hidden representation flow and internal language routing inside black-box LLMs.
- Unbiased Few-Shot Embedding Extraction: The Wikidata-driven entity prediction prompt extracts sentence-level semantic representations from decoder-only LLMs without requiring bilingual instruction translations or introducing English prompt biases.
Limitations & Future Work¶
- Domain Coverage Constraints: Due to the scarcity of massively multilingual parallel datasets, experiments rely heavily on Flores-200. Although out-of-domain validation on BOUQuET demonstrates stability, domain-sharing between alignment estimation and downstream evaluation could potentially inflate observed correlations.
- Limited Sample Size for Related Languages: The mutually intelligible language investigation is restricted to seven pairs and base models only; broader studies across dialect continuums and varied model families are needed.
- Residual Model Dependence in PMI: While PMI dramatically reduces target-language clustering compared to chrF, full theoretical independence cannot be established due to variations in individual LLM calibration and probability calibration errors.
Related Work & Insights¶
- vs. MEXA (Kargaran et al., 2025) & DALI (Ravisankar et al., 2026): Prior studies established that English alignment predicts performance on non-English classification and reading comprehension; this work expands the paradigm to generative sequence-to-sequence translation and models the tri-directional interaction between source, target, and pivot representations.
- vs. Do Llamas Work in English? (Wendler et al., 2024): While previous work demonstrated English mediation by applying vocabulary projection heads (Logit Lens) to intermediate hidden layers during word translation, this paper substantiates the English pivot hypothesis from an entirely independent behavioral and predictive correlation standpoint across full-sentence translations.
Rating¶
- Novelty: โญโญโญโญโญ First systematic study extending cross-lingual alignment prediction to machine translation while resolving target-language bias with PMI and revealing pivot dynamics.
- Experimental Thoroughness: โญโญโญโญโญ Rigorous evaluation across 3 model families (6 variants), 44 languages, 27 representation-metric combinations, and external validation on BOUQuET.
- Writing Quality: โญโญโญโญโญ Well-structured, mathematically rigorous, and seamlessly connects theoretical motivation with empirical discoveries.
- Value: โญโญโญโญโญ Provides actionable best practices for cross-lingual evaluation and delivers foundational insights into the internal linguistic representations of modern LLMs.