DeepSEA: A Multilingual Evaluation Benchmark for Southeast Asia¶
Conference: EMNLP 2026
Paper: https://aclanthology.org/2026.emnlp-main.1/
Code: https://github.com/google-research/deep-sea
Area: Multilingual & Machine Translation
Keywords: Multilingual Benchmark, Southeast Asian Languages, Resource Disparity, Cross-Lingual Reasoning, Machine Translation
TL;DR¶
Google Research introduces DEEPSEA, the first comprehensive human-translated benchmark covering 12 Southeast Asian languages across high- and low-resource tiers on 5 NLP tasks, uncovering severe cross-lingual performance collapse in open-weights models and an overwhelming reliance on English cognitive representations.
Background & Motivation¶
While Large Language Models (LLMs) have achieved remarkable breakthroughs in reasoning, comprehension, and generative tasks, their practical capabilities remain overwhelmingly concentrated in English and a handful of high-resource European languages. The Southeast Asian (SEA) region represents over 700 million inhabitants characterized by profound linguistic diversity, spanning multiple distinct language families and complex non-Latin writing systems such as Thai, Lao, Burmese, and Khmer. However, prior regional benchmarks suffer from critical shortcomings: evaluations are either heavily clustered around major national languages (primarily Indonesian, Vietnamese, and Thai) while completely omitting regional languages spoken by tens of millions (e.g., Cebuano, Ilocano, and Sundanese), or they are restricted to narrow task scopes that fail to capture multi-step reasoning and abstractive generation.
The core tension underlying this landscape is the prohibitively high cost and logistical challenge of building high-quality, uncontaminated evaluation benchmarks for severely underserved low-resource languages. Previous attempts frequently relied on automated machine translation or monolingual multi-choice exams, which inadvertently introduced severe translation artifacts and cultural alignment discrepancies, preventing rigorous, standardized cross-lingual and cross-tier comparisons. The research community and industry critically lack a unified, human-verified suite that spans multiple diverse tasks across both national and underserved regional languages.
This work addresses the challenge by assembling established benchmark suites and commissioning professional native translators to construct high-fidelity parallel evaluation data across 4 high-resource and 8 low-resource Southeast Asian languages. Core idea: establish DEEPSEA, the first professionally human-translated benchmark across 12 Southeast Asian languages spanning 5 core tasks (math and scientific reasoning, question answering, summarization, and machine translation), systematically diagnosing language accessibility bottlenecks, cross-lingual lexical interference, and the models' fundamental dependence on English Chain-of-Thought (CoT) representations.
Method¶
Overall Architecture¶
The DEEPSEA benchmark evaluates 12 Southeast Asian languages categorized into two resource tiers: high-resource languages comprising Thai (th), Indonesian (id), Vietnamese (vi), and Filipino (fil); and low-resource languages comprising Cebuano (ceb), Ilocano (ilo), Javanese (jv), Khmer (km), Lao (lo), Malay (ms), Sundanese (su), and Burmese (my). The suite systematically assesses model capabilities across five core natural language processing tasks: mathematical reasoning (MGSM), scientific reasoning (IndoMMLU), extractive question answering (XQuAD), abstractive summarization (CrossSum), and machine translation (WMT24++).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Source Benchmarks & Human Translation<br/>English / Indonesian Source"] --> B["DEEPSEA 12-Language Benchmark<br/>4 High-Resource + 8 Low-Resource Tiers"]
B --> C["Multi-Task Comprehensive Evaluation<br/>Math/Science Reasoning + QA + Summarization + MT"]
C --> D["Reasoning & Summarization Strategy Analysis<br/>Native CoT vs English CoT / Direct vs English Pivot"]
D --> E["Granular Error & Tokenizer Analysis<br/>Cross-Lingual Lexical Interference + Hallucination + Fertility Ratio"]
Key Designs¶
1. Rigorous Human Translation Protocol: To eliminate synthetic artifacts and translation hallucinations, the authors contracted professional native translation vendors. For MGSM, XQuAD, CrossSum, and WMT24++, translators worked directly from English source texts; for IndoMMLU, translations were conducted from Indonesian into the target low-resource regional languages. Explicit guidelines mandated translating only designated fields, strictly preserving numerical representations, punctuation, and capitalization without injecting explanatory or redundant content, while ensuring fair labor compensation well above local minimum wage baselines.
2. Dual-Pathway Chain-of-Thought Formulation: For reasoning-intensive tasks (MGSM and IndoMMLU), the authors systematically compared two prompting paradigms: (1) Native CoT, where instances and reasoning instructions are fully formulated in the target language; and (2) English CoT, where problem instances are presented in the target language but the model is explicitly instructed to execute step-by-step reasoning in English before generating the final answer. This setup cleanly decouples internal logical reasoning power from target-language textual expression proficiency.
3. English Pivot Decomposition for Abstractive Summarization: On CrossSum (English source documents summarized into SEA target languages), the evaluation contrasted direct cross-lingual summarization (Direct) against a two-stage decomposition (Pivot-based: first compressing the English source into an English summary, then translating the English summary into the target language). This design isolates information compression from target-language text generation, exposing the exact operational failure modes of weaker models.
Loss & Training¶
As a standardized diagnostic evaluation benchmark, DEEPSEA does not involve internal training objectives, adhering to rigorous task-specific evaluation metrics: - Standard Accuracy (Accuracy โ) for mathematical and scientific reasoning; - Token-level F1 score (Token-level F1 โ) for extractive question answering; - Character n-gram F-score (chrF โ) for cross-lingual abstractive summarization; - Neural metric MetricX-24 โ for machine translation (where lower scores indicate superior fidelity to human references). All models were evaluated under strict zero-shot greedy decoding (temperature = 0) to guarantee deterministic reproducibility.
Key Experimental Results¶
Main Results¶
The authors evaluated frontier proprietary models (Gemini-3.7-Flash, Gemini-3-Pro, GPT-5.6-Luna, Claude-Sonnet-5) against leading open-weights models (Gemma-3-27B/12B, Qwen2.5-7B-Inst, Llama-3.1-8B-Inst, SEA-LION-v4-27B, Sailor-2-20B):
| Model | Math Reasoning (MGSM Acc โ) En / High / Low |
Science Reasoning (IndoMMLU Acc โ) En / High / Low |
Question Answering (XQuAD F1 โ) En / High / Low |
Summarization (CrossSum chrF โ) En / High / Low |
Machine Translation (MetricX-24 โ) High / Low |
|---|---|---|---|---|---|
| Gemini-3.7-Flash | 99.2 / 96.5 / 96.4 | 86.5 / 86.3 / 86.1 | 87.9 / 85.2 / 81.7 | 37.4 / 36.4 / 36.2 | 2.4 / 3.5 |
| Gemini-3-Pro | 98.0 / 95.6 / 95.4 | 85.0 / 85.3 / 84.3 | 85.6 / 81.2 / 77.6 | 37.0 / 35.9 / 35.4 | 2.5 / 3.7 |
| GPT-5.6-Luna | 96.4 / 93.3 / 92.8 | 85.3 / 84.5 / 83.3 | 73.8 / 68.1 / 65.8 | 33.7 / 33.8 / 33.2 | 2.4 / 3.5 |
| Claude-Sonnet-5 | 98.8 / 96.4 / 95.4 | 84.9 / 84.6 / 82.6 | 87.1 / 84.3 / 80.3 | 35.7 / 35.8 / 34.0 | 2.6 / 3.8 |
| Gemma-3-27B | 96.0 / 91.7 / 83.6 | 79.2 / 76.2 / 71.5 | 85.5 / 79.9 / 73.9 | 33.1 / 33.5 / 29.7 | 2.9 / 6.1 |
| Gemma-3-12B | 92.8 / 89.7 / 77.2 | 71.0 / 68.5 / 63.6 | 84.7 / 77.3 / 71.5 | 33.4 / 32.5 / 27.5 | 3.1 / 8.6 |
| Qwen2.5-7B-Inst | 93.2 / 78.3 / 36.2 | 68.2 / 61.3 / 49.2 | 71.9 / 48.5 / 37.9 | 32.9 / 28.4 / 22.2 | 8.2 / 17.6 |
| Llama-3.1-8B-Inst | 87.6 / 67.3 / 38.8 | 52.0 / 42.3 / 24.5 | 71.1 / 62.9 / 37.2 | 34.8 / 30.3 / 21.5 | 5.9 / 14.8 |
| SEA-LION-v4-27B | 96.0 / 92.7 / 83.8 | 78.1 / 76.2 / 71.5 | 85.6 / 80.1 / 74.4 | 33.0 / 33.0 / 28.8 | 3.0 / 6.0 |
Ablation & Diagnostic Studies¶
Diagnostic experiments dissecting prompting paradigms revealed pronounced architectural phenomena:
| Diagnostic Axis / Model | Strategy Paradigm | High-Resource Tier | Low-Resource Tier | Delta & Key Observation |
|---|---|---|---|---|
| MGSM Reasoning (SEA-LION-v4) | Native CoT vs English CoT | 92.0% โ 93.5% | 82.3% โ 83.8% | English CoT provides consistent +1.5% accuracy gain |
| IndoMMLU Reasoning (Gemma-3-12B) | Native CoT vs English CoT | 68.5% โ 71.2% | 63.6% โ 69.2% | Low-resource gain surges to +5.6%, easing language bottlenecks |
| CrossSum Summarization (SEA-LION-v4) | Direct vs English-Pivot | 33.0 โ 34.0 chrF | 28.8 โ 33.2 chrF | Pivot yields massive +4.4 chrF leap, preventing low-resource collapse |
| Tokenizer Fertility (GPT-5 Tokenizer) | Ratio relative to English | ~1.5 - 2.5ร (Latin scripts) | Approaching 7.0ร (Lao, Burmese) | Extreme token fragmentation and inference cost inflation |
Key Findings¶
- Frontier Models Maintain Parity While Small Open Models Collapse: Gemini-3.7-Flash exhibits extraordinary robustness across resource tiers (MGSM drops negligibly from 99.2% in English to 96.4% in low-resource languages). Conversely, Qwen2.5-7B and Llama-3.1-8B undergo catastrophic degradation in low-resource mathematical reasoning, dropping from 93.2% and 87.6% to 36.2% and 38.8%โa drop exceeding 60%.
- Cross-Lingual Lexical Interference: Granular error tracing identified insidious semantic confusions within linguistically proximate language pairs. For example, in Ilocano, the token "lawas" signifies "week", but in the related Visayan language Cebuano it denotes "body". Models frequently misdirected their reasoning chains into biological health topics rather than temporal calculations, illustrating cross-lingual semantic leakage.
- Script-Induced Hallucination & Tokenizer Inflation: On non-Latin abugida scripts (Lao, Khmer, Burmese), models exhibited severe grounding failures in extractive QA, frequently substituting native script numerals with Western Arabic digits; concurrently, tokenizer fragmentation reached up to 7ร relative to English, drastically inflating latency and compute overhead.
Highlights & Insights¶
- Decoupling Reasoning Competence from Linguistic Accessibility: Smaller open-weights models are not inherently deficient in step-by-step mathematical logic; rather, their internalized knowledge fails to interface with low-resource surface tokens. Enforcing English CoT or Pivot prompting effectively unlocks latent reasoning capabilities.
- The Vital Value of Regionally Specialized Models: At the 27B parameter threshold, regionally pretrained SEA-LION-v4 matches or outperforms generalist models like Gemma-3-27B in XQuAD and WMT24++, demonstrating that continuous targeted pretraining on regional linguistic corpora is an indispensable strategy for equitable NLP.
Limitations & Future Work¶
- Residual Translation Artifacts: While translated under rigorous quality controls, parallel datasets derived from English or Indonesian source bases inevitably retain subtle structural traces of the original syntax. Incorporating organic, native-authored regional corpora remains a crucial future objective.
- Micro-Language and Dialectal Horizon: Southeast Asia encompasses hundreds of distinct dialects and hyper-low-resource indigenous languages. Extending the benchmark from 12 representative languages to deeper tail communities will be essential for genuine universal inclusivity.