INCLUDE-2.0: Disentangling Knowledge Gaps in Multilingual LLM Evaluation¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://huggingface.co/datasets/epfl-nlp/include-2.0
Area: Reasoning
Keywords: Multilingual Evaluation, Regional Knowledge Disentanglement, Knowledge-Dark Items, Cross-Lingual Accessibility, Benchmark
TL;DR¶
INCLUDE-2.0 introduces a massive native multilingual benchmark covering 91 languages, 86 countries, and 127 language–country pairs with over 220K curated questions, establishing a four-way regional knowledge taxonomy and a paired-English evaluation framework that disentangles superficial linguistic failure from missing regional knowledge.
Background & Motivation¶
As large language models are deployed across global geographies, diverse linguistic communities, and heterogeneous cultural spheres, rigorously auditing whether they encode the requisite factual foundations to serve local populations has become paramount. However, conventional multilingual evaluations routinely conflate basic linguistic proficiency with the capacity to recall and reason over factual knowledge grounded in specific regions and communities. The overwhelming majority of existing multilingual benchmarks rely on translated English resources such as MMLU or GlobalMMLU, which inevitably preserve a Western-centric and English-dominated factual substrate. When an LLM fails on an evaluation prompt delivered in a non-English language, developers cannot determine whether the system stumbled due to tokenization and syntactic processing bottlenecks, lacked the underlying regional entity knowledge entirely in its parametric memory, or simply defaulted to English-centric priors when faced with locally situated queries.
The primary tension driving this diagnostic failure stems from two fundamental flaws in prior methodology. First, existing benchmarks frequently treat language as an unmediated proxy for geographic region, overlooking pluricentric languages such as Spanish, Arabic, English, and Portuguese, where the exact same linguistic medium is spoken across dozens of nations governed by conflicting legal frameworks, distinctive public institutions, and disparate cultural norms. Second, regional knowledge is historically treated as an undifferentiated monolith, indiscriminately lumping universal physical laws, globally documented historical landmarks, and locally embedded vernacular practices into a single aggregate score. This aggregation masks catastrophic blind spots: a high macro-averaged accuracy on high-resource languages conveys a false impression of universal competence, while the model remains virtually blind to the localized rules and vernacular practices essential to billions of regional users.
To overcome these foundational limitations, the evaluation paradigm must expand its empirical footprint to authentic native regional materials while introducing a principled conceptual framework that decouples regional dependency from linguistic comprehension. The core idea is to construct INCLUDE-2.0—a comprehensive native-source benchmark covering 91 languages and 127 language–country pairs across 220,000 curated exam items—and operationalize regionality via an orthogonal four-category taxonomy (region-agnostic, region-conditional, region-anchored, and region-embedded) paired with controlled English counterfactual translations to cleanly disentangle linguistic capability from regional factual availability.
Method¶
Overall Architecture¶
The INCLUDE-2.0 framework integrates native examination curation, rigorous quality control by native speakers, fine-grained regional metadata annotation, and a dual-condition diagnostic evaluation protocol. Multiple-choice questions (MCQs) are sourced from authentic academic examinations, professional licensure tests, and regional regulatory certifications across 86 countries. Each native sample is paired with a strictly constrained, meaning-preserving English translation and categorized into one of four regional knowledge types. By contrasting model predictions under native in-language (IL) prompts against their corresponding English (EN) translations, the framework systematically maps model failures into orthogonal error quadrants, identifying cases of language-specific processing failure versus systemic parametric knowledge gaps.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Native Multi-Source Exam Curation<br/>Academic / Professional / Regional Licensure (86 Countries / 91 Languages)"] --> B["Four-Way Regional Knowledge Taxonomy<br/>Agnostic / Conditional / Anchored / Embedded"]
B --> C["Constrained Parallel English Mapping<br/>Preserve Difficulty, Options, and LaTeX Formatting"]
C --> D["Dual-Condition Evaluation Protocol<br/>In-Language (IL) vs. Translated English (EN)"]
D --> E["Knowledge State Disentanglement<br/>Linguistic Processing Bottleneck vs. Knowledge-Dark Items"]
Key Designs¶
1. Four-Way Regional Knowledge Taxonomy: Disentangling Contextual Truth Conditions To overcome the ambiguity of treating all regional questions identically, INCLUDE-2.0 formulates a four-category taxonomy based on the ontological relationship between a query and its geographical or institutional context: - Region-agnostic: The ground-truth answer is entirely invariant across all geographic, national, or cultural contexts. Canonical examples include pure mathematics, physical constants, formal propositional logic, and biochemical reaction mechanisms. Because factual truth remains constant globally, cross-lingual accuracy disparities on this subset serve as an unconfounded control axis capturing pure linguistic and tokenization processing competence. - Region-conditional: The identical question prompt yields divergent correct answers depending strictly on the regional or legal jurisdiction to which it is anchored. Prominent examples include statutory speed limits, legal voting ages, prescription drug dispensing regulations, statutes of limitations, and national grading scales. In pluricentric languages (e.g., Spanish across 20 countries, Arabic across 8 countries), the language alone cannot determine the correct choice, requiring the model to condition dynamically on jurisdictional context. - Region-anchored: The query topic targets a specific place, country, or historic event, but the correct answer is a universally documented, fixed fact accessible in widely indexed global reference sources (e.g., English Wikipedia). Typical cases include national capitals, heads of state, major historical treaties, and international demographic statistics. - Region-embedded: The question possesses a fixed, unique ground truth, but the requisite factual knowledge is embedded within local linguistic communities, specialized regional customs, or vernacular traditions, remaining virtually absent from standard English-centric corpora. Examples include traditional musical instruments, regional culinary preparation techniques, vernacular idioms, and local historical figures without international recognition.
2. Constrained Parallel English Mapping: Separating Language Deficits from Parametric Gaps To isolate whether a model fails because it cannot parse the source language or because it completely lacks the underlying fact, INCLUDE-2.0 pairs every native-language question and choice set with an English translation generated under strict invariance constraints. The translation prompt forbids adding hints, altering the number or ordering of choices, or dropping formatting elements such as Markdown, LaTeX, and code blocks. By querying models under both in-language (\(IL\)) and English (\(EN\)) conditions, each item outcome is partitioned into four exhaustive outcomes: both correct, English-only correct, in-language-only correct, and both wrong. Items failing under both linguistic conditions across the entire 23-model population at or below the chance rate (\(\gamma = 0.25\)) are designated as "knowledge-dark," isolating facts that are fundamentally missing from modern LLM parametric pretraining rather than merely linguistically inaccessible.
3. Cross-Lingual Accessibility and Difference-in-Differences Control: Isolating Regional Representation Asymmetries When a model succeeds on an English prompt but fails in the original native language, the failure can be driven by either general low-resource language processing weaknesses or an inability to retrieve regionally coupled facts through native linguistic keys. To mathematically isolate regional knowledge inaccessibility from general language difficulty, the authors introduce a difference-in-differences formulation: $\(\Delta = D_{spec} - D_{agn}\)$ Here, \(D_{spec}\) represents the proportion of region-specific items (Region-anchored and Region-embedded) answered correctly in English but missed in the original language. \(D_{agn}\) represents the identical drop measured on universal, region-agnostic items. Because region-agnostic questions require identical native-language syntactic processing but involve zero regional factual bias, subtracting \(D_{agn}\) eliminates the general linguistic penalty. A strictly positive \(\Delta\) cleanly isolates factual knowledge that the model reliably encodes and produces in English, yet systematically fails to access when queried in the corresponding native tongue.
Benchmark & Evaluation Protocol¶
The benchmark evaluates 4 leading commercial frontier systems (Gemini 3.5 Flash, GPT-5.5, GPT-5.4-mini, Claude Opus 4.7) and 20 open-weight model checkpoints across the Gemma 3/4, Qwen 3/3.5, Llama 3.1, Aya Expanse, Tiny Aya, Apertus, and Mistral families. All experiments were conducted on the Swiss National Supercomputing Centre (CSCS) Alps infrastructure using Grace-Hopper (GH200) nodes, totaling over 600 GPU hours. Models were prompted zero-shot using an explicit <answer>X</answer> generation template. Answer parsing utilizes a two-tier extraction pipeline: primary strict XML-tag matching (em_hard) with an ordered, multi-stage fallback parser (em_soft) to ensure robust parsing across varying instruction-following tolerances. To analyze whether scaling open-weight parameters narrows the cross-lingual divide on region-agnostic items, the study fits a Bayesian mixed-effects logistic regression with family-level random slopes and language/item random intercepts.
Key Experimental Results¶
Main Results¶
The table below reports macro-averaged accuracy (%) across all 91 languages in INCLUDE-2.0, comparing native In-Language (IL) performance against the translated English (EN) condition across all regional knowledge categories.
| Model | Region-agnostic (IL / EN) | Region-conditional (IL / EN) | Region-anchored (IL / EN) | Region-embedded (IL / EN) | All Types Macro (IL / EN) |
|---|---|---|---|---|---|
| Gemini 3.5 Flash | 81.1 / 75.5 | 77.7 / 69.7 | 86.4 / 81.7 | 75.2 / 63.3 | 79.7 / 71.2 |
| GPT-5.5 | 78.7 / 76.0 | 76.2 / 71.6 | 86.1 / 82.5 | 72.5 / 63.4 | 77.5 / 71.3 |
| Claude Opus 4.7 | 77.0 / 76.0 | 75.9 / 67.7 | 83.3 / 82.2 | 67.0 / 60.8 | 74.5 / 70.2 |
| GPT-5.4-mini | 72.1 / 73.2 | 64.8 / 64.4 | 79.4 / 80.4 | 62.5 / 55.7 | 70.1 / 66.8 |
| Gemma 4-31B | 70.9 / 72.1 | 66.5 / 65.2 | 80.0 / 77.9 | 57.0 / 51.0 | 67.3 / 64.7 |
| Gemma 3-27B | 65.2 / 69.2 | 57.2 / 57.7 | 68.5 / 73.2 | 46.9 / 45.5 | 59.9 / 60.7 |
| Qwen 3-32B | 64.3 / 71.6 | 54.6 / 58.2 | 62.7 / 72.6 | 43.1 / 45.1 | 58.6 / 62.0 |
| Qwen 3.5-27B | 64.0 / 68.8 | 54.3 / 56.1 | 66.9 / 73.3 | 43.0 / 41.6 | 58.3 / 59.1 |
| Llama 3.1-70B-Inst | 56.7 / 66.8 | 55.1 / 58.8 | 65.4 / 73.0 | 45.3 / 45.6 | 54.5 / 59.2 |
| Aya Expanse-32B | 50.0 / 63.0 | 48.8 / 52.0 | 57.2 / 70.0 | 38.7 / 42.2 | 48.0 / 55.1 |
| Llama 3.1-8B-Inst | 37.5 / 53.8 | 36.9 / 43.6 | 38.4 / 57.6 | 29.5 / 35.4 | 34.7 / 47.0 |
Regional Knowledge-Dark Analysis and Cross-Lingual Tests¶
The table below details candidate knowledge-dark concentrations across selected nations alongside two-sided McNemar asymmetry tests and scaling regression parameters.
| Metric / Evaluation Dimension | Sample \(n\) | Dark-nat (%) | At Chance (%) | Below Chance (%) | Key Statistical Finding |
|---|---|---|---|---|---|
| Ghana | 5,708 | 34.2 | 17.6 | 16.6 | Highest dark concentration among large-sample nations |
| Nigeria | 8,406 | 30.0 | 14.1 | 15.9 | Extensive parametric blindness across regional African languages |
| Ethiopia | 2,013 | 25.3 | 9.2 | 16.1 | Severe degradation across Amharic, Oromo, and Tigrinya |
| Iran | 7,175 | 16.8 | 10.5 | 6.3 | Local cultural and civic regulations show clustered gaps |
| Frontier Models (4 systems) | 355 tested pairs | - | - | - | 210 pairs reject null (\(p < 0.05\)); 208 favor Native Language |
| Open-Weight Models (20 systems) | 1,800 tested pairs | - | - | - | 855 pairs reject null (\(p < 0.05\)); 674 favor English Translation |
| Mixed-Effects Regression \(\beta_3\) | Full sample | Estimate: 0.079 | 95% CI: [-0.214, 0.447] | Credible interval overlaps 0 | Model scaling does not reliably narrow the cross-lingual gap |
Key Findings¶
- Severe Performance Cliff on Locally Embedded Knowledge: Across all evaluated architectures, moving from globally documented regional facts (Region-anchored) to culturally embedded vernacular facts (Region-embedded) induces a severe performance drop. Commercial frontier models drop by an average of 14.5 percentage points (Gemini 3.5 Flash drops from 86.4% to 75.2%; Claude Opus 4.7 drops from 83.3% to 67.0%), while open-weight models drop by 14.4 percentage points on average. This proves that high scores on conventional benchmarks reflect memorization of global English Wikipedia entities rather than true grounding in native community knowledge.
- Inverted Cross-Lingual Asymmetry Between Frontier and Open Models: Frontier models achieve higher accuracy when queried in the original native language than in English (e.g., Gemini 3.5 Flash gains +8.5% in IL over EN). McNemar tests confirm that out of 210 statistically significant model-language pairs, 208 favor native in-language prompts. Conversely, 18 of 20 open-weight models perform substantially better in English than in their native tongue (e.g., Llama 3.1-8B is +12.3% higher in EN), indicating that open-source models suffer from severe linguistic processing barriers that impede factual recall.
- Breakdown of Global Leaderboard Generalizability: Aggregated global rankings fail to predict local model suitability. While Gemini 3.5 Flash ranks first globally across all 91 languages, it is outperformed by competitors in 36 individual languages (with statistically significant divergence in 8 languages). In low-resource tongues such as Oromo, Tangale, and Maithili, smaller specialized or alternate open-weight models significantly outperform the global leader.
- Scaling Fails to Narrow the Cross-Lingual Gap: Bayesian mixed-effects logistic regression demonstrates that while parameter scale improves overall baseline accuracy on region-agnostic items (\(\beta_1 = 0.430, p < 0.05\)), the interaction coefficient between model size and native language condition \(\beta_3 = 0.079\) possesses a 95% credible interval spanning zero (\([-0.214, 0.447]\)). Pure unguided scaling does not spontaneously resolve cross-lingual representational disparities.
Highlights & Insights¶
- Orthogonal Disentanglement of Language vs. Knowledge: The framework provides an elegant experimental template for separating linguistic processing failures from parametric factual gaps via region-agnostic controls and paired-English ablation.
- Sub-National Disparities in Pluricentric Languages: By analyzing residuals across 20 Spanish-speaking and 8 Arabic-speaking nations, the authors demonstrate that the benefit of native-language querying is highly heterogeneous across national borders, proving that a single language cannot be treated as a culturally uniform entity.
- Empirical Mapping of Regional Knowledge Blind Spots: The paper systematically identifies geographic regions (notably West and East Africa) where over 30% of locally relevant factual queries fall into "knowledge darkness," providing a concrete roadmap for future non-Western corpus curation.
Limitations & Future Work¶
- Multiple-Choice Questioning Format Constraints: All 220,000 items are structured as 4-option MCQs. While essential for scalable cross-lingual benchmarking, MCQs cannot evaluate open-ended generative reasoning, contextual nuance, or conversational explanation fidelity.
- Heuristic Availability Proxy: The partition between region-anchored and region-embedded categories relies on an annotation heuristic assessing availability in standard English reference materials, rather than an independent quantitative index of external web document frequency.
- Machine Translation Artifacts in Parallel Controls: While strictly prompt-engineered and spot-checked by humans, parallel English references were generated via GPT-5.5, which may occasionally smooth out regional idioms or subtly alter task difficulty.
Related Work & Insights¶
- vs. Translated MMLU / GlobalMMLU: Previous benchmarks evaluate non-English models by translating English tests, testing Western concepts through foreign words; INCLUDE-2.0 sources 100% of its content from authentic native regional examinations across 86 countries.
- vs. EXAMS / INCLUDE (1.0): Prior native exam benchmarks were capped at 16 to 44 languages without decomposing the underlying knowledge types; INCLUDE-2.0 expands coverage to 91 languages and formalizes the four-way regional taxonomy.
- vs. BLEnD / CulturalBench: Existing cultural benchmarks focus primarily on subjective lifestyle norms, crowdsourced conventions, or synthetic red-teaming; INCLUDE-2.0 anchors its evaluation on standardized academic curricula and official professional certifications with verified objective truth values.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneering four-way regional knowledge taxonomy and paired-language diagnostic framework]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Comprehensive evaluation spanning 91 languages, 86 countries, 24 models, and rigorous Bayesian mixed-effects modeling]
- Writing Quality: ⭐⭐⭐⭐⭐ [Exceptional conceptual clarity, disciplined empirical rigor, and compelling prose structure]
- Value: ⭐⭐⭐⭐⭐ [A foundational milestone for culturally grounded, region-aware multilingual LLM evaluation]