Skip to content

Apples to Apples? Towards Comparable Crosslingual Language Model Evaluation

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/xiulinyang/multilingual-eval.git
Area: LLM Reasoning
Keywords: Crosslingual Evaluation, Intrinsic Language Model Metrics, Tokenization Bias, Byte Premium, Sentence-level Negative Log-Likelihood

TL;DR

This paper demonstrates that popular normalized intrinsic metrics (PPL, BPB, BPC, Token-NLL, and MRR) introduce severe systematic crosslinguistic biases rooted in tokenization granularity, byte encodings, and orthographic conventions, proving theoretically and empirically that unnormalized sentence-level negative log-likelihood (Sent-NLL) evaluated on parallel texts is the only unbiased metric for crosslingual model comparison.

Background & Motivation

As large language models scale rapidly from English-centric environments to hundreds of typologically diverse languages across the globe, fair and comparable crosslingual evaluation has emerged as a fundamental yet critically under-appreciated challenge in multilingual natural language processing. Although recent multilingual foundation models such as BLOOM, mGPT, XGLM, Llama, and Qwen demonstrate increasingly broad language coverage, substantial performance discrepancies across language families persist. However, accurately measuring and interpreting these performance gaps remains deeply problematic: widely adopted probability-based intrinsic metrics—such as perplexity (PPL), bits-per-byte (BPB), bits-per-character (BPC), and token-level negative log-likelihood—often produce conflicting, contradictory, or outright inverted rankings when applied to the exact same models and benchmark corpora.

The fundamental tension underlying these conflicting evaluations is the pervasive confounding of intrinsic linguistic complexity with arbitrary representation-level engineering choices, specifically tokenization segmentation, orthographic systems, and byte encodings. In pursuit of "apples to apples" comparisons across languages, conventional metrics perform normalization by surface physical units (such as tokens, characters, or bytes). Crucially, these normalization schemes systematically introduce artificial representation premiums: non-Latin scripts suffer from high UTF-8 byte premiums that artificially deflate BPB scores and favor longer-byte languages; logographic scripts such as Chinese encode high information density per character and thus suffer severe penalties under character-normalized BPC; and token-normalized metrics like PPL and Token-NLL reward severe oversegmentation by deflating surprisal over fragmented single-byte sequences. While prior literature questioned the reliability of intrinsic metrics based on intuition or anecdotal observations, there has been a critical lack of controlled causal investigations isolating metric behavior from model parameter and data confounds.

To decouple metric properties from the confounding dynamics of multilingual models, this paper conducts a rigorous empirical investigation using a matrix of 50 controlled monolingual language models trained on strictly parallel data across 10 typologically diverse languages spanning 6 language families, each evaluated under 5 distinct vocabulary size settings, and validates these findings on 9 widely used multilingual LLMs. Core idea: abandon surface-level physical unit normalization (tokens, characters, bytes) in favor of unnormalized sentence-level negative log-likelihood (Sent-NLL) computed over semantically equivalent parallel content, anchoring crosslingual evaluation firmly in semantic invariance rather than representation artifacts.

Method

Overall Architecture

The paper develops a rigorous, four-tier experimental and theoretical diagnostic framework to audit and isolate the mathematical biases inherent in crosslingual evaluation metrics. First, the mathematical formulations of six mainstream probability-based metrics are decomposed to map their normalizers against concrete representational confounds (byte premium, length premium, corpus token count). Second, a matrix of 50 controlled monolingual GPT2-small models is trained from scratch on strictly parallel text (PARALLEL-10, encompassing 10 languages across 6 language families) with five vocabulary sizes, further scaled to GPT2-medium on the trilingual PARALLEL-3 corpus to test scaling behavior. Third, metric robustness is tested against semantic translation alternatives and varying sample sizes using multi-reference WMT datasets. Fourth, metric behavior is verified against empirical pretraining data allocations across nine open-source multilingual large language models (including BLOOM, mGPT, XGLM, Llama, and Qwen), extending the inquiry to downstream task comparability boundaries.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Parallel Evaluation Corpora<br/>FLORES-200 / PUD / WMT19"] --> B["Mathematical Decomposition & Bias Audit<br/>Disentangling surface units from semantic anchors"]
    B --> C["Controlled Monolingual Model Matrix<br/>10 languages × 5 vocabulary sizes = 50 models"]
    C --> D["Engineering Representation Sensitivity Tests<br/>Correlations with token counts, bytes, and characters"]
    D --> E["Robustness Across Translation Alternatives<br/>Multi-reference stability and sample-size scaling"]
    E --> F["Multilingual LLM Training Prior Verification<br/>Aligning actual rankings with documented data proportions"]
    F --> G["Crosslingual Comparability Protocol<br/>Establishing Sent-NLL validity and downstream guidance"]

Key Designs

1. Mathematical Decomposition & Representation Bias Audit: Unveiling Physical Unit Normalization Confounds To diagnose why common intrinsic metrics disagree, the paper formalizes the relationship between candidate metrics and representation-level variables. Given a sequence \(s\) comprising \(n\) tokens \(t_1, \dots, t_n\), \(c\) characters, and \(b\) UTF-8 bytes: - Bits-per-Byte (BPB): Defined as \(\text{BPB}(s) = -\frac{1}{b}\sum_{i=1}^n \log_2 P(t_i \mid t_{<i})\). Although designed to be tokenizer-agnostic, BPB introduces severe crosslingual bias due to orthographic encoding disparities: scripts like Khmer or Arabic require up to 3 bytes per character, whereas Latin scripts require only 1. Consequently, given identical model assignment of semantic probability, languages with higher byte premiums yield artificially lower (better) BPB scores simply because total surprisal is divided by a larger byte count \(b\). - Bits-per-Character (BPC): Defined as \(\text{BPC}(s) = -\frac{1}{c}\sum_{i=1}^n \log_2 P(t_i \mid t_{<i})\). Different writing systems exhibit vastly different length premiums; logographic Chinese conveys substantial semantic information per character (\(c\) is small), forcing BPC scores to be artificially high (penalized). - Perplexity (PPL) and Token-level NLL (Token-NLL): Defined as \(\text{PPL}(s) = \exp\left(-\frac{1}{n}\sum_{i=1}^n \ln P(t_i \mid t_{<i})\right)\) and \(\text{Token-NLL}(s) = -\frac{1}{n}\sum_{i=1}^n \ln P(t_i \mid t_{<i})\). Both divide total log-likelihood by the number of tokens \(n\). When evaluating an unseen or low-resource language script, standard BPE segmenters fragment words into many small single-byte pieces, driving up Corpus Token Count (CTC) and artificially deflating PPL and Token-NLL to deceptively low numbers. - Mean Reciprocal Rank (MRR): Defined as \(\text{MRR}(s) = \frac{1}{n}\sum_{i=1}^n \frac{1}{\text{rank}(t_i \mid t_{<i})}\), where ranks are bounded by vocabulary size; smaller vocabularies inherently compress the rank range and inflate MRR irrespective of modeling quality. - Sentence-level Negative Log-Likelihood (Sent-NLL): Formulated as \(\text{Sent-NLL}(s) = -\sum_{i=1}^n \ln P(t_i \mid t_{<i})\). By summing raw surprisal across the entire semantically equivalent sequence without dividing by surface counts, Sent-NLL shifts the normalization anchor from surface physical tokens to semantic information units.

2. Controlled Monolingual Matrix & Causal Sensitivity Auditing: Decoupling Representation from Model Quality To eliminate confounding factors inherent in multilingual models (such as shared parameter interference and skewed language pretraining distributions), the authors instantiate a strictly controlled setup: 50 monolingual GPT2-small models trained on PARALLEL-10, an aligned 10-language corpus (English, Arabic, Chinese, Turkish, French, German, Finnish, Polish, Russian, Korean; ~20M words per language) across five vocabulary sizes (8,192, 32,768, 49,152, 65,536, and 81,920). Scaling behavior is validated using GPT2-medium on PARALLEL-3 (~400M words per language for English, Chinese, and Arabic). By evaluating these models on parallel benchmarks (FLORES-200, in-domain test sets, and PUD treebanks) and computing Spearman's rank correlation \(\rho\) against potential confounds (#characters, #bytes, and CTC), the study quantitatively demonstrates that surface metrics are driven by engineering artifacts while Sent-NLL remains uncorrelated with surface length.

3. Robustness Across Translation Alternatives & Sample-Size Scaling: Distinguishing Paraphrase from Translation Addressing claims from prior literature (e.g., Poelman & de Lhoneux, 2026) that probability metrics are fundamentally unstable across paraphrases, the authors clarify the theoretical distinction between within-language paraphrasing and crosslingual translation. While paraphrases allow flexible wording and alter pragmatic/semantic information content (e.g., dative alternation or procedural vs. general competence), parallel translations are tightly bound by crosslinguistic semantic equivalence. Using WMT2019 English-German translation pairs alongside an independent human-generated reference set (DE1 and DE2) to construct pairwise extreme sets (\(\text{DE}_{\text{lowest}}\) and \(\text{DE}_{\text{highest}}\)), the framework examines metric variance across sample sizes (\(n=1\) to \(n=1997\) across 15 seeds). The analysis proves that Sent-NLL rankings achieve stable convergence once sample sizes reach \(n \ge 500\), consistently scoring English below all German variants.

4. Multilingual LLM Prior Verification & Downstream Task Boundary Assessment: Extrapolating to Real Foundations To demonstrate that identified metric distortions are universal mathematical properties rather than artifacts of small models, the framework evaluates nine prominent multilingual LLMs across four languages (Arabic, English, Russian, Chinese). For models with documented pretraining distributions (BLOOM-1B/7B, mGPT, XGLM-4.5B), empirical metric rankings are checked against theoretical expectations derived from training volume (e.g., BLOOM: EN 31.3% > ZH 18.3% > AR <1% > RU 0%). For models with undisclosed distributions (Llama-3, Qwen3-8B, Teuken-7B), dominant language alignment is assessed. Finally, the framework analyzes downstream evaluation pitfalls, detailing how translated benchmarks suffer from translationese, localized benchmarks (CMMLU, IndoMMLU, KMMLU) conflate language proficiency with cultural knowledge, and syntactic probes (MultiBLiMP) introduce non-parallel item imbalances and crosslinguistic typological disparities.

Benchmark & Evaluation Protocol

The experimental suite leverages strictly parallel and high-fidelity test corpora: - FLORES-200: High-quality multi-parallel machine translation benchmark spanning over 200 languages; - PARALLEL-10 and PARALLEL-3 Test Sets: Aligned in-domain test splits matching the controlled pretraining corpora (40,000 sentences each); - Parallel Universal Dependencies (PUD): 1,000 parallel sentences with dependency tree annotations; - WMT2019 Multi-Reference Dataset: English-German parallel references for testing translation alternative robustness; - MultiBLiMP / ZhoBLiMP: Linguistic minimal pair evaluation for zero-shot grammatical competence verification.

Key Experimental Results

Main Results: Correlation Analysis Between Metrics and Representation Confounds

Spearman's rank correlation coefficient \(\rho\) and statistical significance \(p\)-value between intrinsic metrics and their hypothesized representational confounds on FLORES-200 across 50 controlled monolingual GPT-2 models:

Metric Potential Confound Spearman's \(\rho\) \(p\)-value Metric Fairness Assessment
BPC (Bits-per-Character) Character count (#characters) -0.81 0.0082 Biased (driven by script length premium)
BPB (Bits-per-Byte) Byte count (#bytes) -0.94 < 0.001 Biased (driven by UTF-8 byte premium)
PPL (Perplexity) Corpus Token Count (CTC) -0.93 < 0.001 Biased (driven by token oversegmentation)
Token-NLL (Token-level NLL) Corpus Token Count (CTC) -0.93 < 0.001 Biased (identical sensitivity to PPL)
MRR (Mean Reciprocal Rank) Corpus Token Count (CTC) 0.89 0.001 Biased (constrained by vocabulary rank bound)
Sent-NLL (Ours) Character count (#characters) -0.38 0.28 (n.s.) Unbiased (robust to character length differences)
Sent-NLL (Ours) Byte count (#bytes) 0.14 0.71 (n.s.) Unbiased (invariant to UTF-8 byte encoding)
Sent-NLL (Ours) Corpus Token Count (CTC) -0.19 0.19 (n.s.) Unbiased (invariant to tokenizer vocabulary size)

Note: PPL is scaled by \(\log_{10}\). For a given corpus, vocabulary size and CTC maintain a strict monotonic correspondence. All metrics except Sent-NLL display significant correlation (\(|\rho| \ge 0.81, p < 0.01\)) with engineering artifacts.

Multilingual Foundation Model Evaluation vs. Pretraining Data Expectations

Comparison between documented pretraining data proportions and empirical metric rankings across 4 languages on FLORES-200 for BLOOM-7B1 (EN 31.3%, ZH 18.3%, AR <1%, RU 0%) and monolingual English GPT-2:

Model & Language Setting Expected Rank BPB Actual Rank BPC Actual Rank PPL / Token-NLL Rank Sent-NLL Rank MultiBLiMP Accuracy
BLOOM-7B1: English (EN) 1st 2nd 1st 2nd 1st (matches data) -
BLOOM-7B1: Chinese (ZH) 2nd 4th (worst) 4th (worst) 3rd 2nd (matches data) -
BLOOM-7B1: Russian (RU) 4th (0% data) 3rd 3rd 1st (artificially deflated) 4th (matches data) -
BLOOM-7B1: Arabic (AR) 3rd (<1% data) 1st 2nd 4th 3rd (matches data) -
GPT-2 (Monolingual): English 1st (trained) - - PPL: 49.77 (worst) Sent-NLL: 99.34 (best) 0.97 (fluent syntax)
GPT-2 (Monolingual): Chinese Unseen - - PPL: 32.83 Sent-NLL: 277.67 0.54 (chance level)
GPT-2 (Monolingual): Russian Unseen - - PPL: 9.45 (absurd best) Sent-NLL: 248.56 0.43 (below chance)
GPT-2 (Monolingual): Arabic Unseen - - PPL: 12.22 Sent-NLL: 251.00 0.50 (chance level)

Key Findings

  • The Oversegmentation PPL Trap in Unseen Scripts: Evaluating an English-only GPT-2 model on FLORES reveals that Russian achieves an astonishingly low PPL of 9.45 compared to 49.77 for English. Because the English BPE tokenizer decomposes Cyrillic text into single-byte fragments, the inflated token count \(n\) drastically dilutes average per-token surprisal, even though MultiBLiMP confirms GPT-2 possesses zero Russian syntactic knowledge (0.43 accuracy). Only Sent-NLL accurately places English (99.34) ahead of Russian (248.56).
  • Orthographic Inequity of BPB and BPC: In both controlled monolingual models and large models like BLOOM, Chinese is consistently ranked last under BPB and BPC due to its high per-character information density and compact sequence representation. Despite comprising 18.3% of BLOOM's pretraining mixture, Chinese scores worse than languages with less than 1% representation, invalidating claims that BPB is universally fair.
  • Statistical Stability Across Translation Variations: On the WMT19 multi-reference translation benchmark, small evaluation sets (\(n \le 100\)) show ranking instability across translation variants. However, once sample size exceeds \(n \approx 500\), Sent-NLL rankings become completely stable and immune to translation alternative noise, confirming that semantic invariance holds reliably across parallel corpora.

Highlights & Insights

  • Debunking the Myth of Universal BPB Neutrality: The paper cleanly disproves the common assumption that BPB is inherently fair across languages, establishing that while BPB neutralizes tokenizer differences within a single language, it introduces severe crosslinguistic bias across scripts with differing byte premiums.
  • Semantic Normalization as the Proper Anchor: By demonstrating that dividing surprisal by physical surface counts invariably introduces representation noise, the study establishes that parallel semantic content itself serves as the natural normalizer for crosslingual evaluation.
  • Clarifying the Linguistic Status of Paraphrases: The authors provide a crucial theoretical correction to recent evaluation debates by showing that within-language paraphrases inherently encode divergent information and pragmatic nuances, whereas crosslingual translations adhere to semantic equivalence constraints.

Limitations & Future Work

  • Language Sample Coverage: While spanning 10 typologically diverse languages across 6 language families, the controlled monolingual training experiments remain bounded by parallel corpus availability (~20M words per language), warranting future validation across broader low-resource typologies.
  • The Equi-complexity Hypothesis: Sent-NLL implicitly relies on the assumption that natural languages convey comparable amounts of information across parallel expressions. If natural languages systematically diverge in intrinsic information density, Sent-NLL will capture both modeling capability and linguistic information density—a fundamental challenge common to all crosslingual evaluation.
  • Dependence on High-Quality Parallel Data: The framework strictly requires aligned, human-validated parallel text, limiting immediate deployment in under-resourced languages and spoken dialects where multi-parallel benchmarks like FLORES do not yet exist.
  • vs Poelman & de Lhoneux (2026): While they concluded that all probability-based intrinsic metrics are inherently flawed due to paraphrase instability, this work demonstrates that their findings conflated within-language paraphrasing with crosslingual translation; under controlled parallel conditions, Sent-NLL provides stable, unbiased rankings.
  • vs Gao et al. (2020) & Shani et al. (2026): Prior work strongly advocated Bits-per-Byte (BPB) as a standardized crosslingual metric. This paper provides direct empirical proof that BPB introduces strong orthographic bias (\(\rho = -0.94\)), rendering it invalid for cross-script comparisons.
  • vs Cotterell et al. (2018) & Mielke et al. (2019): While earlier foundational studies used English-character normalization (BPEC) to study language modeling difficulty, this paper shows such formulations are monotonic transformations of Sent-NLL and provides the first large-scale empirical causal validation across controlled vocabulary scales.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Rigorously identifies and formalizes the mathematical and representational biases pervading crosslingual LM evaluation metrics.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ 50 controlled monolingual models across 5 vocabulary sizes, 9 multilingual LLMs, multi-reference translation tests, and syntactic probes.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear conceptual taxonomy, disciplined mathematical formulations, and seamless narrative progression.
  • Value: ⭐⭐⭐⭐⭐ Provides crucial practical guidelines and theoretical foundations that will directly reshape evaluation standards in multilingual NLP.