Assessing the Impact of Language Disparity on Multilingual Linguistic Ability in Large Language Models¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/EricZhanyuChen/multiblimp_eval
Area: Multilingual & Translation
Keywords: language disparity, multilingual linguistic ability, minimal pair evaluation, alignment tax, inarticulate gap
TL;DR¶
Evaluating six LLM families across 101 languages on the MultiBLiMP syntactic minimal-pair benchmark under four paradigms reveals that post-training imposes a linguistic alignment tax borne disproportionately by low-resource languages, while native-language prompting crucially recovers latent grammatical competence masked by unprompted probabilities.
Background & Motivation¶
Multilingual large language models (LLMs) have achieved unprecedented progress in processing, understanding, and generating diverse natural languages. Nevertheless, severe performance inequalities persist across linguistic communities, with cutting-edge capabilities clustered in a tiny fraction of dominant, high-resource languages. While prior investigations have mapped these disparities across high-level downstream applications like machine translation and question answering, systematic examinations of fundamental grammatical and syntactic competence—such as morphosyntactic subject-verb and subject-participle agreement—remain largely unexplored at scale. Crucially, how grammatical ability should be measured in LLMs remains actively debated: whether via unprompted raw string probabilities or interactive metalinguistic question answering.
This evaluation discrepancy harbors two deep tensions that have evaded rigorous empirical scrutiny. First, modern post-training pipelines—encompassing supervised instruction fine-tuning and reinforcement learning from human feedback—rely overwhelmingly on high-resource language corpora due to data scarcity. This English-centric post-training regime inevitably distorts the pre-trained token distribution and risks inducing severe linguistic degradation or catastrophic forgetting in non-English languages, yet the exact magnitude and distribution of this "alignment tax" across resource tiers have not been quantified. Second, recent findings demonstrate that English models fail to "introspect" about their grammatical knowledge—exhibiting an inarticulate gap where correct probability judgments cannot be verbalized via chat interfaces. Whether this dissociation generalizes cross-lingually or collapses in low-resource settings due to near-chance baselines has remained an open theoretical question.
To address these fundamental challenges, this paper leverages MultiBLiMP, a comprehensive benchmark comprising 121,305 minimal pairs across 101 languages, and systematically probes base and post-trained checkpoints across six prominent open-weight LLM families (Gemma 3, Gemma 4, Llama 3, Qwen 3, Qwen 3.5, and TinyAya) spanning parameter scales from 0.6B to 35B. Core idea: by constructing a multi-paradigm evaluation matrix spanning direct probability, English prompting, native-language prompting, and metalinguistic QA, the study formalizes the normalized linguistic alignment tax and inarticulate gap, demonstrating that low-resource languages bear the heaviest alignment costs and that native-language contextualization is indispensable to avoid systematically underestimating low-resource grammatical competence.
Method¶
Overall Architecture¶
The paper formulates an end-to-end framework to dissect cross-lingual grammatical competence, alignment distortion, and evaluation biases. Testing centers on MultiBLiMP, spanning 101 languages and 121,305 minimal pairs covering subject-verb and subject-participle agreement in number, person, and gender. Based on web-scale token frequencies from Common Crawl, all 101 languages are stratified into four resource tiers: Super High (7 languages, \(>10^{11}\) tokens), High (38 languages, \(10^9\)–\(10^{11}\) tokens), Mid (20 languages, \(10^7\)–\(10^9\) tokens), and Low/Zero (36 languages, \(\le 10^7\) tokens). For every grammatical sentence \(S_{cor}\) and minimally perturbed ungrammatical counterpart \(S_{inc}\), models are queried across four evaluation paradigms to compute log-likelihood margins, categorical predictions, normalized alignment taxes, and item-level directional flips.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Minimal Pair Input<br/>Grammatical Scor vs Ungrammatical Sinc"] --> B["Multi-Paradigm Evaluation Matrix<br/>Direct / English / Native / Meta"]
B --> C["Normalized Alignment Tax Quantification<br/>Isolating post-training loss above chance"]
C --> D["Dual Knowledge Decoupling & Inarticulate Gap<br/>Direct probability vs Meta QA evaluation"]
D --> E["Native-Language Prompt Activation<br/>Target language contextualization of latent ability"]
E --> F["Item-Level Flip & Margin Decay Analysis<br/>Tracing weak pre-trained knowledge attrition"]
Key Designs¶
1. Multi-Paradigm Evaluation Matrix: Unifying Likelihood and Meta Probing Prior linguistic probing predominantly relied on isolated evaluation formats, causing severe confounding between task-following ability and underlying grammatical knowledge. The authors implement four standardized protocols: (1) Direct Probability (zero-shot), evaluating sequence likelihoods on bare text without instructions, declaring success if \(P(S_{cor}) > P(S_{inc})\) (using base model format or model-specific chat templates without added instructions); (2) English-Prompted Probability, prepending a standardized English context string \(C_{eng}\) (“The following is a sentence in [Language].”); (3) Native-Prompted Probability, translating the prefix directly into the target language \(C_{native}\) (e.g., German “Dies ist ein Satz auf Deutsch.”), with attested phrases or shortened language-name cues for 14 ancient/low-resource languages; and (4) Meta Evaluation, restructuring each pair into a two-choice conversational question-answering format (“Which of the following sentences is grammatically correct in [Language]? Only respond with 1 or 2.”), with position randomized by character length parity to prevent positional option bias.
2. Normalized Linguistic Alignment Tax: Baseline-Calibrated Knowledge Loss Post-training commonly degrades raw probability estimation, creating a raw performance drop \(\Delta = \text{Acc}_{base} - \text{Acc}_{PT}\). However, raw accuracy drops are deceptive when comparing across resource tiers: an English base model at 99% accuracy has immense headroom to drop, whereas a low-resource language starting at 60% accuracy sits barely above the 50% random-chance threshold. To fairly quantify the degradation of above-chance knowledge, the paper formalizes the Normalized Linguistic Alignment Tax: $\(\text{Normalized Tax} = \frac{\text{Acc}_{base} - \text{Acc}_{PT}}{\text{Acc}_{base} - 0.5}\)$ The denominator isolates the model's genuine grammatical knowledge beyond random guessing. Languages with \(\text{Acc}_{base} \le 0.5\) are excluded, enabling scale- and resource-invariant benchmarking of post-training costs.
3. Dual Knowledge Decoupling and Inarticulate Gap: Implicit vs Explicit Dissociation To examine whether LLMs can introspect about their grammatical mastery across typologically varied languages, the paper formulates the Inarticulate Gap: $\(\text{Gap} = \text{Acc}_{Direct(PT)} - \text{Acc}_{Meta(PT)}\)$ A positive gap establishes that an instruction-tuned model possesses implicit grammatical knowledge in its sequence probability distribution that it cannot reliably articulate when explicitly queried through conversational prompts. Furthermore, base models are evaluated under the same format to establish a structural format-mismatch baseline. Cross-lingual analysis demonstrates that this gap is heavily shaped by pre-training representation quality and language resource depth.
4. Native-Language Prompt Activation and Item-Level Flip Audit: Mechanism Profiling To investigate whether linguistic knowledge is truly destroyed or merely latent, the authors measure the Prompt Gain: \(\text{Acc}_{Native(PT)} - \text{Acc}_{Direct(PT)}\). At the item level, all 121,305 minimal pairs are matched between base and post-trained checkpoints under the Direct paradigm. By calculating the base log-probability margin \(\log P(S_{cor}) - \log P(S_{inc})\), paired with logistic regressions over agreement features (e.g., dual number, second person), syntactic distance, and attractor nouns, the study tracks the micro-level dynamics of post-training erosion.
Key Experimental Results¶
Main Results¶
The table below reports average accuracy (%) across all four evaluation methods and extreme resource tiers for the largest model in each of the six evaluated families (mean ± standard deviation across languages).
| Model Family & Size | Evaluation: Direct (%) | Evaluation: Meta (%) | Evaluation: Native Prompt (%) | Evaluation: English Prompt (%) | Tier: Low/Zero (%) | Tier: Super High (%) |
|---|---|---|---|---|---|---|
| Gemma 3 (27B) base | 84.17 ± 19.4 | 73.17 ± 18.7 | 86.21 ± 18.1 | 85.77 ± 19.3 | 66.72 ± 22.1 | 98.60 ± 0.6 |
| Gemma 3 (27B) PT | 79.46 ± 18.9 | 74.02 ± 17.9 | 81.63 ± 17.5 | 79.69 ± 17.1 | 61.70 ± 19.5 | 95.15 ± 1.6 |
| Gemma 4 (31B) base | 85.56 ± 19.1 | 84.38 ± 16.1 | 88.36 ± 16.9 | 88.13 ± 17.9 | 68.61 ± 22.4 | 98.86 ± 0.7 |
| Gemma 4 (31B) PT | 76.09 ± 17.2 | 82.33 ± 18.1 | 78.06 ± 16.9 | 79.77 ± 17.0 | 61.08 ± 18.2 | 91.79 ± 2.6 |
| Llama 3 (8B) base | 80.88 ± 18.4 | 54.51 ± 9.9 | 83.24 ± 19.2 | 82.74 ± 18.7 | 64.96 ± 20.5 | 97.11 ± 1.7 |
| Llama 3 (8B) PT | 79.19 ± 18.1 | 57.31 ± 11.6 | 82.07 ± 18.8 | 81.39 ± 19.0 | 63.42 ± 19.9 | 95.31 ± 2.5 |
| Qwen 3 (30B-A3B) base | 80.37 ± 19.1 | 71.40 ± 17.6 | 82.56 ± 19.3 | 81.73 ± 19.5 | 63.26 ± 20.5 | 97.03 ± 1.7 |
| Qwen 3 (30B-A3B) PT | 78.14 ± 18.6 | 68.67 ± 17.1 | 80.36 ± 18.7 | 79.80 ± 19.4 | 61.58 ± 19.4 | 94.65 ± 2.0 |
| Qwen 3.5 (35B-A3B) base | 83.45 ± 19.0 | 81.91 ± 17.9 | 85.47 ± 18.8 | 85.56 ± 19.0 | 66.25 ± 21.5 | 97.91 ± 1.0 |
| Qwen 3.5 (35B-A3B) PT | 82.44 ± 18.6 | 80.48 ± 18.2 | 84.89 ± 17.8 | 84.59 ± 19.1 | 65.62 ± 20.7 | 96.96 ± 1.5 |
| TinyAya (3.35B) base | 76.68 ± 19.3 | 55.01 ± 10.0 | 78.06 ± 20.4 | 77.51 ± 21.2 | 62.07 ± 20.0 | 97.60 ± 1.3 |
| TinyAya (3.35B) PT | 74.61 ± 18.2 | 54.77 ± 11.0 | 76.65 ± 18.7 | 75.52 ± 19.8 | 61.81 ± 18.9 | 93.19 ± 2.7 |
Ablation Study¶
The table below details the progression of the Normalized Alignment Tax (%) and the post-trained Inarticulate Gap (Direct - Meta, %) across model parameter scales (1B to 27B) and resource tiers for the Gemma 3 family.
| Resource Tier / Metric | Normalized Tax (1B) | Normalized Tax (4B) | Normalized Tax (12B) | Normalized Tax (27B) | Inarticulate Gap (1B PT) | Inarticulate Gap (4B PT) | Inarticulate Gap (12B PT) | Inarticulate Gap (27B PT) |
|---|---|---|---|---|---|---|---|---|
| Super High Tier | 23.4% | 21.8% | 11.5% | 7.1% | 22.85% | 7.93% | 6.22% | 3.81% |
| High Tier | 33.6% | 27.2% | 14.7% | 9.3% | 19.83% | 9.91% | 4.32% | 4.43% |
| Mid Tier | 36.9% | 31.4% | 19.0% | 16.1% | 15.32% | 12.08% | 10.54% | 9.36% |
| Low/Zero Tier | 35.9% | 33.0% | 12.0% | 20.6% | 3.50% | 6.20% | 4.49% | 4.66% |
Key Findings¶
- Alignment tax is family-specific and spares high-resource tiers under scaling: Across families, the normalized alignment tax varies by an order of magnitude (TinyAya at 1.4% vs Gemma 4-31B at 30.9%). Within Gemma 3, scaling reduces tax dramatically for Super High languages (23.4% \(\to\) 7.1%), but Low/Zero languages retain a severe tax of 20.6% at 27B, demonstrating that scaling alone does not bridge the post-training deficit for under-resourced languages.
- The inarticulate gap is an artifact of high-resource visibility: For Gemma 3 PT, Super High languages exhibit a large gap at 1B (22.85%) that shrinks to 3.81% at 27B. In contrast, Low/Zero languages remain flat at 3.50%–4.66%. This occurs because low-resource Direct accuracy is already near the 50% chance baseline, leaving no statistical margin for an unarticulated knowledge gap to manifest.
- Flips are heavily one-directional: On Gemma 4 (31B), 14.5% of the 121,305 minimal pairs are lost after post-training, while only 2.9% are gained (a loss-to-gain ratio of 5.0:1). Items lost post-training had significantly smaller base log-probability margins (median margin 5.0) than items retained (median margin 8.0). Furthermore, low-resource items were 2.3–4.3 times more likely to be lost even when controlling for base margin.
- Native-language prompting unlocks hidden competence: Native prompting yields a strong positive Prompt Gain (+2.8 pp) in low-resource languages, whereas high-resource languages show negligible gains (\(\approx 0\)). This establishes that standard unprompted probability estimation underestimates low-resource competence due to the absence of target-language contextualization.
Highlights & Insights¶
- Formulates the normalized alignment tax to eliminate chance-level distortions across resource tiers, proving that post-training disproportionately penalizes low-resource languages.
- Uncovers the paradox of the inarticulate gap: it appears to be a high-resource phenomenon primarily because near-chance baselines in low-resource languages artificially conceal unarticulated knowledge.
- Demonstrates that native-language instruction framing is an effective mitigation tool that activates latent grammatical representations in under-resourced languages.
Limitations & Future Work¶
- Restricted syntactic scope: Evaluation is limited to subject-verb and subject-participle agreement in MultiBLiMP; whether findings generalize to complex word order, relative clauses, or long-range unbounded dependencies remains untested.
- Pre-training data opacity: Training mixtures are undisclosed for the evaluated models; Common Crawl frequencies serve as an imperfect proxy for true language representation during pre-training.
- Disabled reasoning modes: Evaluations were run with reasoning/thinking modes switched off; how multi-step reasoning affects grammatical articulation in low-resource contexts warrants future investigation.
Related Work & Insights¶
- vs Hu et al. (2024; 2026): Hu et al. established that string probabilities align better with human grammaticality judgments than prompting in English. This paper generalizes the comparison across 101 languages and proves that unprompted probabilities alone fail to elicit latent capabilities in low-resource languages without native framing.
- vs Song et al. (2025): While Song et al. identified the inarticulate gap strictly within English LLMs, this work demonstrates that the gap correlates negatively with the alignment tax across 101 languages and is masked in low-resource tiers due to floor effects.
- vs Jumelet et al. (2026) MultiBLiMP: This paper expands the benchmark from a purely causal probability evaluation into a comprehensive multi-paradigm audit framework dissecting alignment tax and prompting dynamics.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Comprehensive cross-lingual formalization of the alignment tax and inarticulate gap across 101 languages and 6 model families.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Exhaustive evaluation of 121,305 pairs across 23 checkpoint pairs, integrating log-margin regressions and bootstrap analyses.
- Writing Quality: ⭐⭐⭐⭐⭐ Crystal-clear methodology, rigorous metric normalization, and deep linguistic insights.
- Value: ⭐⭐⭐⭐⭐ Provides foundational benchmarks and methodological standards for equitable multilingual model development.