Omnilingual MT: Machine Translation for 1,600 Languages via Scalable Multilingual Pretraining¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Agent
Keywords: Machine Translation, Multilingual Pretraining, Low-Resource Languages, OMT-LLaMA, OMT-NLLB
TL;DR¶
Omnilingual MT introduces a massively multilingual translation ecosystem covering over 1,600 languages by combining a 256K balanced tokenizer, backtranslation across 2,000+ languages, and the grammatically diverse seed dataset MeDLEy, demonstrating that specialized 1Bโ8B models (OMT-LLaMA and OMT-NLLB) strictly outperform 70B+ general-purpose LLMs and double the count of practically usable languages.
Background & Motivation¶
Neural machine translation (NMT) has achieved remarkable progress over recent years, most notably through Meta's No Language Left Behind (NLLB) initiative, which successfully scaled high-quality translation across 200 language varieties. However, the world encompasses approximately 7,000 spoken and written languages, leaving the overwhelming majority of marginalized and low-resource communities without computational language technology. While pioneer explorations into 1,000-language translation (e.g., Siddhant et al.) demonstrated theoretical viability, they failed to establish sustained, extensible systems. Concurrently, recent paradigms leveraging general-purpose large language models (LLMs) have yielded notable gains for high-resource tongues, yet their multilingual benefits remain deeply skewed toward dominant languages. Moreover, LLMs exhibit severe resource inefficiency, requiring tens of billions of parameters merely to approximate the accuracy of compact specialized translation architectures on low-resource language pairs.
This persistent divide stems from a severe structural asymmetry between cross-lingual comprehension and generation: modern foundation models can often interpret underserved source languages through cross-lingual transfer into English (XX-En), but they fail catastrophically when generating into low-resource target scripts (En-YY). Most leading LLM baselines degrade to near-random token generation once target varieties exceed 300 to 400 languages. This bottleneck is directly exacerbated by severe tokenization fertility caused by vocabulary underrepresentation in standard tokenizers, inadequate exploitation of raw monolingual text, and a critical lack of parallel seed corpora that capture fine-grained typological and grammatical diversity. Furthermore, standard neural evaluation metrics such as xCOMET and MetricX rely on underlying encoders with strictly bounded language coverage, creating an evaluation blind spot beyond 200 languages.
To bridge this barrier and establish equitable linguistic access, this work introduces an end-to-end full-pipeline translation system designed from inception to be multilingual. Core idea: by eliminating tokenization bottlenecks through an expanded 256K balanced vocabulary, curating the grammatically diverse seed dataset MeDLEy, and designing a multi-stage non-parallel autoencoding training objective, we construct specialized decoder-only (OMT-LLaMA 1B/3B/8B) and encoder-decoder (OMT-NLLB 3B) models that systematically outperform 70B general-purpose models across 1,600+ languages.
Method¶
Overall Architecture¶
Omnilingual MT provides a dual-track architectural paradigm designed for complementary deployment scenarios: OMT-LLaMA, a decoder-only autoregressive family built on LLaMA 3.1 (available at 1B, 3B, and 8B parameter scales), and OMT-NLLB, a compact 3B encoder-decoder model built upon the universal cross-lingual sentence embedding space of OMNISONAR.
The entire system pipeline spans five integrated stages: data curation, tokenization expansion, architectural modeling, multi-stage post-training, and holistic omnilingual evaluation. Training data is compiled from multi-billion token monolingual web and document corpora (CC-2000-Web and CC-2000-PDF spanning 2,000+ languages), the Bible across ~1,600 languages, PanLex dictionaries across ~1,000 languages, and MeDLEy, a manually curated grammatical seed dataset for 109 extreme low-resource languages. Monolingual corpora are transformed into parallel supervision via billion-scale OMNISONAR embedding mining with FAISS product quantization and quality-calibrated backtranslation (yielding ~230M sentences across ~2,000 languages). OMT-LLaMA undergoes 256K vocabulary warmup, 105B-token continual pretraining (CPT), supervised fine-tuning (SFT), and DAPO reinforcement learning. Meanwhile, OMT-NLLB progresses through frozen-encoder autoencoding, token-level cross-attention unbottlenecking, and end-to-end parallel fine-tuning. Evaluation is anchored by the 275-language BOUQuET benchmark, the 161-direction Met-BOUQuET human annotation set, and BLASER 3 quality estimation.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multilingual Corpora & Raw Text<br/>CC-2000 / Bible / PanLex / FineWeb-Edu"] --> B["Billion-scale Bitext Mining & MeDLEy Seed Dataset Curation"]
B --> C["256K Balanced Tokenizer Expansion & Sub-token Decomposition"]
C --> D["Dual-Track Architecture & Multi-Stage Staged Training"]
D --> E["XSTS+R+P Protocol & BLASER 3 Reference-Free Metric"]
E --> F["Omnilingual MT System Suite<br/>OMT-LLaMA (1B/3B/8B) & OMT-NLLB (3B)"]
Key Designs¶
1. Billion-scale Bitext Mining & MeDLEy Seed Dataset Curation: Addressing Severe Long-tail Data Scarcity -> Curating Typological Diversity and OMNISONAR Margin Filtering -> Delivering Robust Supervised Anchors for 2,000 Languages
Low-resource language corpora typically suffer from narrow topical domain bias (often confined to religious texts or sparse lexical dictionaries) and completely miss colloquial or varied syntactic structures. To establish a balanced foundation, the authors introduced MeDLEy (Multicentric, Domain-diverse, Linguistically-diverse, Easy-to-translate seed dataset), a manually curated parallel dataset covering 109 extremely low-resource languages (92 of which were absent from prior benchmarks like SMOL or NLLB Seed). Linguists first identified 61 universal cross-linguistic grammatical phenomena (such as grammatical case, aspectual distinctions, and evidentiality) and drafted 605 paragraphs (~2,200 sentences) across 5 distinct domains in 5 source languages. These were multiway-parallelized across 8 pivot languages and translated by professional translators into 109 low-resource targets. For broader coverage, monolingual web texts were segmented with SAT-12L-SM, filtered with GlotLID language identification, and translated using NLLB-200 or OMT-LLaMA. Parallel candidates were mined globally using FAISS product-quantized OMNISONAR embeddings with margin-based cosine similarity thresholds calibrated to language-specific Bible baseline means, filtering out length anomalies, repetitive hallucinations, and verbatim copies to yield 230M clean synthetic sentence pairs across ~2,000 languages.
2. 256K Balanced Tokenizer Expansion & Sub-token Decomposition: Mitigating Extreme Subword Fragmentation -> Continued BPE Merging and Embedding Averaging -> Halving Sequence Length and Boosting Generative Fluency
Standard LLM tokenizers (such as LLaMA 3's native 128K vocabulary) fail catastrophically on underserved scripts, fragmenting sentences from low-resource scripts into disjointed single-byte sequences, which multiplies inference latency, exceeds context windows, and disrupts semantic continuity. The authors extended the tokenizer from 128K to 256K tokens via continued byte-pair encoding (BPE) merging over a balanced multilingual distribution, alongside customized pre-tokenization rules for non-Latin writing systems. Rather than randomly initializing new token embeddings, each newly created token embedding \(e_{\text{new}}\) is initialized by computing the average of its sub-token decomposition embeddings: $\(e_{\text{new}} = \frac{1}{|S|}\sum_{t \in S} e_t\)$ This preserves existing semantic priors. On the FLoRes+ benchmark, this tokenization expansion nearly halved average sequence length from 80.7 to 44.8 tokens per sentence and yielded a 26% relative ChrF++ improvement in English-to-target generation (En-YY, jumping from 17.8 to 22.5) and a 7% relative gain in into-English understanding.
3. Dual-Track Architecture & Multi-Stage Staged Training: Overcoming Long-tail Generation Degradation -> Autoencoding Non-parallel Reconstruction and Attention Unbottlenecking -> Attaining High-efficiency Generation and Robust Understanding
To address distinct operational constraints, the authors developed two complementary model families with tailored optimization recipes: - Decoder-only OMT-LLaMA (1B, 3B, 8B): Initialized from LLaMA 3.1, the extended 256K vocabulary is first warmed up for 10K steps on 80 GPUs with the decoder backbone frozen. Continual pretraining (CPT) then proceeds for 50K steps across 256 GPUs over ~105B tokens using packed token batching (8,192 tokens per GPU rank). Post-training incorporates supervised fine-tuning (SFT) over 600K diverse instructions and MeDLEy translations to restore conversational obedience, followed by reinforcement learning using DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization) with a composite ChrF++ and MetricX reward regularized by KL divergence. - Encoder-decoder OMT-NLLB (3B): Initialized with a frozen OMNISONAR sentence encoder (which maps 1,600+ languages into a unified semantic space), OMT-NLLB adopts a 3-stage training scheme. In Stage 1 (Decoder Enhancement), the decoder is trained jointly on parallel MT data and monolingual autoencodingโreconstructing target sentences directly from pooled OMNISONAR representations, which injects fluency for low-resource languages. In Stage 2 (Removing the Bottleneck), the pooled sentence constraint is replaced with token-level encoder-decoder cross-attention, warming up the decoder to avoid optimization shocks. In Stage 3 (End-to-end Tuning), encoder and decoder are jointly updated on parallel corpora, unlocking fine-grained token-to-token interactions and boosting FLoRes+ ChrF++ to 50.9.
4. XSTS+R+P Protocol & BLASER 3 Reference-Free Metric: Overcoming Limited Metric Language Breadth -> Capturing Pragmatic Register and Discourse Coherence -> Establishing Reliable Quality Estimation Across 1,600 Languages
Evaluating translations across 1,600 languages is fundamentally restricted by the language coverage of automated metrics. The authors introduced the XSTS+R+P human evaluation protocol, extending cross-lingual semantic similarity (XSTS) with Register (R, penalizing formality and honorific mismatches) and Paragraph-level consistency (P, penalizing cross-sentence pronoun, grammatical gender, or temporal discrepancies). This elevated inter-annotator agreement to a Krippendorff's \(\alpha\) of 0.80 (compared to 0.57 for standard XSTS and 0.44 for MQM penalty). Using XSTS+R+P, they constructed Met-BOUQuET across 161 language directions and 119 unique languages. Leveraging this data alongside synthetic translations, they trained BLASER 3, a reference-free quality estimation model built atop frozen OMNISONAR embeddings. BLASER 3 achieves a mean per-direction item-level Spearman correlation of 0.55 on Met-BOUQuET (outperforming MetricX at 0.47 and xCOMET at 0.43), offering the first reliable, reference-free quality estimator spanning 1,600+ languages.
Loss & Training¶
During the CPT phase of OMT-LLaMA, standard autoregressive causal language modeling loss \(\mathcal{L}_{\text{CLM}}\) is applied over packed parallel and monolingual documents. The reinforcement learning phase optimizes policy \(\pi_\theta\) using DAPO against reference policy \(\pi_{\text{ref}}\): $\(\mathcal{L}_{\text{RL}}(\theta) = -\mathbb{E}_{(x, y) \sim \pi_\theta} \left[ \min\left(r_\theta(y|x)\hat{A}, \text{clip}(r_\theta(y|x), 1-\epsilon, 1+\epsilon)\hat{A}\right) \right] + \beta \mathbb{D}_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}})\)$ where the scalar reward combines character-level ChrF++ and reference-based MetricX predictions. In OMT-NLLB Stage 1, the autoencoding objective minimizes cross-entropy token prediction loss conditioned on pooled sentence embeddings \(z = \text{OMNISONAR}(x_{\text{mono}})\), while Stage 3 minimizes standard sequence-to-sequence cross-entropy across all mined and authentic parallel pairs.
Key Experimental Results¶
Main Results¶
The table below reports translation performance (ChrF++) on the multidomain BOUQuET benchmark (275 languages across 8 domains) and the Bible test set (1,560 languages), broken down by resource tier (High, Mid, Low, Very Low):
| System | Params | Architecture | XX-En (H / M / L / VL) | En-YY (H / M / L / VL) | BOUQuET Total | Bible XX-En Total |
|---|---|---|---|---|---|---|
| OMT-LLaMA 8B | 8B | Decoder-only | 65.1 / 53.5 / 38.7 / 27.6 | 60.7 / 45.8 / 30.8 / 18.8 | 36.7 | 38.8 |
| OMT-NLLB | 3B | Enc-Dec | 64.0 / 56.1 / 37.4 / 28.4 | 61.1 / 47.6 / 27.3 / 17.5 | 36.6 | 46.3 |
| OMT-LLaMA 3B | 3B | Decoder-only | 64.4 / 54.6 / 38.8 / 28.0 | 59.4 / 45.0 / 28.7 / 16.8 | 36.2 | - |
| NLLB-200 | 3.3B | Enc-Dec | 61.5 / 52.4 / 32.1 / 25.5 | 62.9 / 46.8 / 24.6 / 17.5 | 34.6 | 24.0 |
| GPT-OSS 120B | 120B | Decoder-only | 64.6 / 49.9 / 33.6 / 25.4 | 63.6 / 43.7 / 26.0 / 16.0 | 34.5 | 22.5 |
| Gemma 3 | 27B | Decoder-only | 65.3 / 51.3 / 34.8 / 27.4 | 62.6 / 40.9 / 24.0 / 14.8 | 34.1 | 24.5 |
| TranslateGemma | 12B | Decoder-only | 63.6 / 50.9 / 35.0 / 27.7 | 59.5 / 42.0 / 22.1 / 15.5 | 33.9 | 23.7 |
| OMT-LLaMA 1B | 1B | Decoder-only | 63.0 / 51.5 / 34.7 / 24.1 | 56.5 / 42.2 / 25.1 / 14.9 | 33.5 | - |
| LLaMA 3 70B | 70B | Decoder-only | 65.0 / 47.6 / 33.3 / 26.2 | 60.2 / 37.2 / 23.7 / 16.2 | 32.9 | 24.7 |
| MADLAD | 10B | Enc-Dec | 64.7 / 47.8 / 31.0 / 24.3 | 62.2 / 35.5 / 21.5 / 15.3 | 31.3 | - |
| Tower+ 72B | 72B | Decoder-only | 66.5 / 43.3 / 31.7 / 25.2 | 58.9 / 26.9 / 15.2 / 11.3 | 28.7 | - |
| LLaMA 3 8B | 8B | Decoder-only | 61.0 / 39.8 / 27.5 / 21.6 | 52.7 / 26.0 / 13.1 / 9.5 | 25.6 | 19.4 |
Ablation Study¶
The impact of individual architectural stages and adaptation techniques is evidenced across multiple benchmarks:
Table 1: Stepwise Training Progression of OMT-NLLB (FLoRes+ and Bible Benchmarks) | Stage Configuration | FLoRes+ ChrF++ | FLoRes+ xCOMET | FLoRes+ MetricX (lower is better) | Bible ChrF++ | Bible xCOMET | Bible MetricX (lower is better) | |---|---|---|---|---|---|---| | OMNISONAR Base (Frozen Enc-Dec) | 47.5 | 0.65 | 5.66 | 40.6 | 0.52 | 8.79 | | + Stage 1 (with Autoencoding) | 49.7 | 0.65 | 5.48 | 45.9 | 0.53 | 8.59 | | -- Ablation: Stage 1 (w/o Autoencoding) | 49.6 | 0.65 | 5.51 | 44.8 | 0.52 | 8.61 | | + Stage 2 (Unbottlenecked Cross-Attn) | 50.1 | 0.65 | 5.36 | 46.4 | 0.51 | 8.58 | | + Stage 3 (OMT-NLLB End-to-End Fine-tuned) | 50.9 | 0.66 | 5.32 | 46.3 | 0.50 | 8.61 |
Table 2: Extensibility via Fine-tuning (FT) and RAG across 20 Very Low (VL) and 5 Low (L) Resource Languages (BOUQuET Dev ChrF++) | System & Technique | 20 VL Languages En-YY | 20 VL Languages XX-En | 5 L Languages En-YY | 5 L Languages XX-En | |---|---|---|---|---| | OMT-LLaMA + FT + RAG | 29.3 | 36.7 | 46.7 | 53.2 | | OMT-LLaMA + FT | 28.9 | 29.0 | 47.5 | 42.7 | | OMT-NLLB + FT | 28.2 | 37.9 | 52.4 | 56.7 | | OMT-LLaMA + RAG | 26.8 | 36.2 | 44.8 | 54.0 | | OMT-NLLB (Zero-shot) | 24.4 | 35.8 | 51.4 | 56.8 | | OMT-LLaMA (Zero-shot) | 24.1 | 29.9 | 43.0 | 48.4 | | LLaMA-base + FT + RAG | 20.1 | 27.9 | 33.2 | 40.9 | | LLaMA-base (Unadapted zero-shot) | 9.3 | 20.7 | 16.1 | 28.7 |
Key Findings¶
- Specialization Trumps Brute-force Scale: The 1B OMT-LLaMA achieves an overall BOUQuET score of 33.5, cleanly outperforming the 70B parameter LLaMA 3 (32.9). At 8B parameters, OMT-LLaMA (36.7) matches or surpasses GPT-OSS 120B (34.5) while outperforming it by wide margins on low and very low resource tiers.
- Generation Bottleneck Pushed to 1,200 Languages: While all standard open-weight baselines degrade to near-random generation performance when target languages exceed 300โ400, OMT-LLaMA sustains meaningful generation fluency for approximately 1,200 languages on the 1,560-language Bible benchmark.
- Doubling of Usable Language Varieties: By establishing a MetricX-calibrated threshold corresponding to an XSTS+R+P score of 2.5 ("passable" quality preserving core message semantics), OMT-LLaMA covers 440 languages and OMT-NLLB covers 416 languagesโdoubling the coverage of NLLB-200 (221 languages).
- Synergistic Adaptation Dynamics: On underserved language pairs, fine-tuning primarily boosts into-target generation (+4.8 En-YY), whereas test-time retrieval (RAG) primarily assists source understanding (+6.3 XX-En). Combining both yields the strongest overall configuration (+5.1 En-YY / +6.8 XX-En). Crucially, fine-tuning unadapted base models fails to match even the zero-shot baseline of OMT models.
Highlights & Insights¶
- The Specialized Pareto Frontier: This paper empirically refutes the hypothesis that scale alone automatically resolves long-tail translation, establishing that targeted vocabulary rebalancing, linguistic seed design, and domain-continual pretraining produce a drastically superior Pareto frontier.
- Monolingual Autoencoding for Cross-Lingual decoders: Leveraging universal sentence embeddings (OMNISONAR) as bottleneck conditions for decoder autoencoding allows unaligned monolingual text to directly train target generation syntax, unlocking vast quantities of uncurated web data without parallel supervision.
- Pragmatically-Grounded Evaluation: Incorporating register and paragraph coherence into XSTS+R+P provides the MT community with a rigorous evaluation protocol, while BLASER 3 establishes an omnilingual reference-free quality estimation metric capable of operating on 1,600+ languages.
Limitations & Future Work¶
- Target Generation Asymmetry in OMT-NLLB: While OMT-NLLB understands 1,600 source languages, its generative decoding is currently capped at 250 target varieties, falling short of fully symmetric panlingual translation.
- Aging Base Architecture: The current implementation relies on the LLaMA 3.1 foundation, which is somewhat dated compared to modern architectures like LLaMA 3.3 or Qwen 2.5, though the authors note the recipe readily transfers to newer model families.
- Long-tail Evaluation Domain Bias: Extreme long-tail benchmarks (1,560 languages) remain inherently dependent on religious texts (the Bible), which may not accurately reflect informal registers, modern idioms, or contemporary web discourse.
Related Work & Insights¶
- vs NLLB-200: While NLLB-200 advanced translation to 200 symmetric language varieties, Omnilingual MT scales source understanding by eight-fold (1,600 languages) and expands generative capability to 1,200 languages while doubling the count of practically usable languages.
- vs MADLAD-400 & TranslateGemma: Specialized multilingual systems like MADLAD and TranslateGemma improved coverage over 200โ400 languages, but suffer performance cliffs on extreme long-tail varieties; OMT's MeDLEy seed dataset and autoencoding pretraining ensure robust generalization across thousands of languages.
- vs General-Purpose LLMs (LLaMA 3 70B / GPT-OSS 120B): Large foundation LLMs suffer extreme subword fragmentation, high computational footprint, and hallucinations in low-resource regimes; OMT demonstrates that targeted multilingual pretraining allows 1Bโ8B parameter models to decisively beat 70Bโ120B general models.
Rating¶
- Novelty: โญโญโญโญโญ Groundbreaking pipeline achieving benchmarked MT coverage across 1,600+ languages with novel autoencoding and grammatical seed curation.
- Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation across 275-language BOUQuET, 1,560-language Bible, 161 human evaluation directions, and toxicity auditing.
- Writing Quality: โญโญโญโญโญ Exceptionally clear narrative detailing structural bottlenecks, architectural trade-offs, and empirical findings.
- Value: โญโญโญโญโญ A monumental milestone for global linguistic inclusion and open-access language technologies for marginalized populations.