Omnilingual MT: Machine Translation for 1,600 Languages¶
Conference: EMNLP 2026
arXiv: 2603.16309
Paper: ACL Anthology
Code: https://huggingface.co/spaces/facebook/bouquet
Area: LLM Reasoning
Keywords: Multilingual MT, Extremely Low-Resource Languages, Vocabulary Extension, Decoder-Only MT, Quality Estimation
TL;DR¶
Meta AI introduces Omnilingual MT, scaling translation support to over 1,600 languages through a 256K extended tokenizer, grammatically diverse seed dataset MeDLEy, and a novel three-stage auto-encoding training scheme, allowing 1B–8B specialized models to consistently outperform 70B general-purpose LLMs in cross-lingual reasoning and long-tail generation.
Background & Motivation¶
Multilingual neural machine translation has achieved remarkable breakthroughs over recent years, with milestone efforts such as No Language Left Behind (NLLB) establishing high-quality translation across 200 languages as the standard evaluation benchmark. However, human civilization encompasses more than 7,000 living languages, and the vast majority of the world's underserved and indigenous tongues remain completely neglected by modern natural language processing pipelines. Early preliminary explorations venturing toward 1,000-language translation demonstrated empirical feasibility, yet they stopped short of producing sustained, extensible systems capable of continual multilingual evolution. Simultaneously, mainstream large language models (LLMs) have advanced high-resource translation through massive web-scale pretraining, but their multilingual capability remains disproportionately concentrated in dominant commercial languages, demanding tens of billions of parameters while proving computationally inefficient and brittle on long-tail dialects.
The core tension underlying massively multilingual translation lies in a fundamental operational asymmetry: modern foundation models can often interpret and understand inputs from unencountered or marginal languages through cross-lingual transfer in shared representation spaces, but they fail catastrophically when required to generate into those same languages. Standard subword tokenizers incur exorbitant fertility on underserved non-Latin scripts, which shreds semantic attention contexts, while existing training pipelines rely excessively on repetitive religious texts lacking modern grammatical diversity. Furthermore, evaluation methodology has remained stalled at the 200-language horizon; reference-free neural metrics are biased toward well-represented target distributions, leaving the research community without principled means to measure whether models truly attain passable translation quality across the long tail of human speech.
To resolve this systematic barrier, this paper contends that linguistic inclusivity requires purpose-built engineering across the entire data, modeling, and evaluation stack rather than brute-force general parameter scaling. Core idea: by combining a grammatically diverse multiway parallel seed dataset (MeDLEy) and massive backtranslation with a 256K-token expanded vocabulary, we build complementary decoder-only (OMT-LLaMA) and encoder-decoder (OMT-NLLB via three-stage autoencoding) models, accompanied by the 275-language BOUQuET benchmark and the BLASER 3 metric, to deliver robust translation and generation across 1,600+ languages.
Method¶
Overall Architecture¶
Omnilingual MT is structured around an integrated four-part pipeline comprising data synthesis, dual-paradigm model training, preference post-training, and massively multilingual evaluation. The system collects web-scale monolingual and mined texts for over 2,000 languages, complemented by the newly curated MeDLEy seed corpus covering 109 long-tail languages across 61 grammatical functions and synthetic backtranslation filtered via OMNISONAR embeddings. On the modeling side, two complementary paths are pursued: OMT-LLaMA extends LLaMA3 with a custom 256K tokenizer through continual pretraining and DAPO reinforcement learning, while OMT-NLLB adapts an encoder-decoder architecture via a three-stage progression (autoencoding decoder enhancement \(\to\) unbottlenecked token-level cross-attention \(\to\) end-to-end finetuning). Finally, models are verified using the 275-language BOUQuET benchmark, the 161-direction Met-BOUQuET human annotation set, and the BLASER 3 reference-free metric.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Long-Tail Multilingual Corpora<br/>CC-2000-Web/PDF + Bible Benchmark"] --> B["1. Seed Data Curation and Backtranslation Filtering<br/>MeDLEy 61 Grammatical Functions + OMNISONAR Filtering"]
B --> C["2. Vocabulary Extension and Continual Pretraining<br/>128K→256K BPE Expansion + 105B Token CPT"]
C --> D["3. Three-Stage Encoder-Decoder Training<br/>Auto-Encoding Decoder + Bottleneck Removal + End-to-End Tuning"]
D --> E["4. Post-Training via Instruction SFT and RL Alignment<br/>Multilingual SFT + DAPO (ChrF++/MetricX) Optimization"]
E --> F["5. Massively Multilingual Evaluation and Quality Estimation<br/>BOUQuET 275 Languages + BLASER 3 QE Verification"]
Key Designs¶
1. Seed Data Curation and Backtranslation Filtering: Bridging the grammatical and domain gap for low-resource languages
Existing long-tail parallel corpora are overwhelmingly concentrated in formal religious texts, preventing translation systems from generalizing to realistic syntactic constructions and everyday vocabulary. The authors construct MeDLEy, a manually curated seed corpus for 109 extremely low-resource languages (including 92 languages completely absent from prior seed datasets such as SMOL or NLLB Seed). MeDLEy covers 61 linguistic grammatical phenomena (case markings, evidentiality, aspectual distinctions) across 605 paragraphs (approximately 2,200 sentences) in 5 domains, multiway-parallelized across 8 pivot languages and translated by professional native linguists. For unannotated web corpora across 2,000+ languages, a synthetic backtranslation pipeline segments text via SAT-12L-SM, filters language purity with GlotLID, backtranslates using NLLB-200 or early OMT checkpoints, and retains roughly 230 million high-fidelity sentence pairs based on calibrated OMNISONAR embedding similarity and model-free hallucination heuristics (length ratio, unique character ratio, and edit distance).
2. Vocabulary Extension and Continual Pretraining: Eliminating subword fertility barriers on rare writing systems
Off-the-shelf LLM tokenizers fragment rare writing systems into excessively long sequences of byte-level tokens, severely degrading semantic representations and computational efficiency. The method modifies pre-tokenization rules and executes continued BPE merging on balanced multilingual data, expanding the vocabulary from 128K to 256K tokens. This halves the average sentence token length on FLoRes+ from 80.7 to 44.8, with new token embeddings initialized via the mean of their constituent sub-token vectors. OMT-LLaMA models (1B, 3B, and 8B parameters) are subsequently trained across 105B tokens using packed token batching (2.1M tokens per optimizer step), consisting of a 10K-step embedding warmup on 80 GPUs followed by 50K steps of joint language modeling and translation continual pretraining across 256 GPUs.
3. Three-Stage Encoder-Decoder Training: Harnessing non-parallel data via representation auto-encoding
Standard encoder-decoder architectures struggle when scaled to 1,600 languages due to severe parallel data scarcity, while sentence-level cross-attention bottlenecks (such as original OMNISONAR pooling) restrict expressive sequence-to-sequence mapping. OMT-NLLB (3B parameters) solves this through a progressive three-stage optimization recipe: Stage 1 initializes from a frozen OMNISONAR encoder and trains the decoder on parallel data alongside a monolingual auto-encoding objective, compelling the decoder to reconstruct native target sentences directly from pooled language-agnostic sentence representations; Stage 2 removes the pooled representation bottleneck and introduces unconstrained token-level encoder-decoder cross-attention, warming up decoder cross-attention while the encoder remains temporarily frozen; Stage 3 executes end-to-end fine-tuning across all encoder and decoder parameters simultaneously, fully unlocking token-level multilingual cross-attention dynamics.
4. Post-Training via Instruction SFT and RL Alignment: Stabilizing generation format and enforcing translation rewards
Continually pretrained base models frequently suffer from off-target language hallucination and conversational drift. The post-training recipe applies supervised fine-tuning (SFT) over approximately 600K diverse instruction examples spanning 10 languages combined with parallel data from SMOL and MeDLEy, restoring precise instruction-following compliance. Following SFT, Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) is applied using a composite metric reward function combining ChrF++ and MetricX under Kullback–Leibler divergence regularization:
This reinforcement learning stage delivers cumulative score increases across difficult subsets without degrading performance on any translation direction.
5. Massively Multilingual Evaluation and Quality Estimation: Establishing rigorous human and neural metrics at scale
Traditional evaluation setups such as FLORES-200 fail to capture broader language families or stylistic nuances. The authors extend BOUQuET to 275 languages across 8 diverse domains, and curate Met-BOUQuET, the largest MT human evaluation dataset covering 161 language directions with the XSTS+R+P protocol. XSTS+R+P incorporates register (R) fidelity and paragraph-level (P) consistency penalties, achieving a high inter-annotator agreement (Krippendorff's \(\alpha = 0.80\), compared to \(0.44\) for standard MQM). To resolve the dependency on reference translations, BLASER 3 is introduced as a compact neural quality estimation model trained multi-task across 6 evaluation protocols over frozen OMNISONAR embeddings, reaching an item-level Spearman correlation of 0.55 on Met-BOUQuET and outperforming MetricX (0.47) and xCOMET (0.43).
Loss & Training¶
The OMT-LLaMA decoder-only pipeline optimizes standard causal language modeling objectives during CPT and SFT, followed by policy gradient updates via DAPO under clipping thresholds. OMT-NLLB incorporates cross-entropy translation loss alongside auto-encoding sequence reconstruction in Stage 1, transitioning to end-to-end label-smoothed cross-entropy in Stage 3. For inference-time adaptation on scarce languages, optional retrieval-augmented generation (RAG) fetches parallel examples using hybrid lexical search and OMNISONAR embedding similarity.
Key Experimental Results¶
Main Results¶
On the comprehensive BOUQuET benchmark (275 languages across 8 domains), OMT models establish a new Pareto frontier across both translation into English (XX-En) and translation from English (En-YY). Specialized 1B–8B models match or surpass 70B+ general-purpose models across nearly all resource tiers.
| Model | Size | XX-En (High / Mid / Low / V.Low) | En-YY (High / Mid / Low / V.Low) | Overall Total ChrF++ |
|---|---|---|---|---|
| OMT-LLaMA 8B | 8B | 65.1 / 53.5 / 38.7 / 27.6 | 60.7 / 45.8 / 30.8 / 18.8 | 36.7 |
| OMT-NLLB | 3B | 64.0 / 56.1 / 37.4 / 28.4 | 61.1 / 47.6 / 27.3 / 17.5 | 36.6 |
| OMT-LLaMA 3B | 3B | 64.4 / 54.6 / 38.8 / 28.0 | 59.4 / 45.0 / 28.7 / 16.8 | 36.2 |
| OMT-LLaMA 1B | 1B | 63.0 / 51.5 / 34.7 / 24.1 | 56.5 / 42.2 / 25.1 / 14.9 | 33.5 |
| NLLB-200 | 3.3B | 61.5 / 52.4 / 32.1 / 25.5 | 62.9 / 46.8 / 24.6 / 17.5 | 34.6 |
| GPT-OSS 120B | 120B | 64.6 / 49.9 / 33.6 / 25.4 | 63.6 / 43.7 / 26.0 / 16.0 | 34.5 |
| Gemma 3 | 27B | 65.3 / 51.3 / 34.8 / 27.4 | 62.6 / 40.9 / 24.0 / 14.8 | 34.1 |
| TranslateGemma | 12B | 63.6 / 50.9 / 35.0 / 27.7 | 59.5 / 42.0 / 22.1 / 15.5 | 33.9 |
| LLaMA 3 70B | 70B | 65.0 / 47.6 / 33.3 / 26.2 | 60.2 / 37.2 / 23.7 / 16.2 | 32.9 |
| MADLAD | 10.7B | 64.7 / 47.8 / 31.0 / 24.3 | 62.2 / 35.5 / 21.5 / 15.3 | 31.3 |
| Tower+ | 72B | 66.5 / 43.3 / 31.7 / 25.2 | 58.9 / 26.9 / 15.2 / 11.3 | 28.7 |
| LLaMA 3 8B | 8B | 61.0 / 39.8 / 27.5 / 21.6 | 52.7 / 26.0 / 13.1 / 9.5 | 25.6 |
Ablation Study¶
The table below highlights controlled ablations over tokenizer capacity, post-training stages, and the three-stage progression of the OMT-NLLB architecture.
| Configuration / Stage | Benchmark & Direction | Metric Score | Key Insight & Mechanism |
|---|---|---|---|
| LLaMA3 1B (128K Tokenizer) | FLoRes+ (En-YY Avg) | ChrF++ 17.8 | High fertility creates severe sequence dilation on unseen scripts |
| LLaMA3 1B (256K Tokenizer) | FLoRes+ (En-YY Avg) | ChrF++ 22.5 (+26.4% rel.) | Vocabulary expansion halves token count and dramatically boosts generation |
| OMT-LLaMA (CPT Base) | BOUQuET (XX-En / En-YY) | xCOMET .725 / .621 | Multilingual continual pretraining establishes core foundation |
| + SFT Tuning | BOUQuET (XX-En / En-YY) | xCOMET .732 / .611 | Formats instructions and sharply aligns translation fidelity |
| + RL Alignment (DAPO) | BOUQuET (XX-En / En-YY) | xCOMET .741 / .616 | Combined ChrF++/MetricX reward optimizes challenging long-tail translations |
| OMNISONAR Baseline | FLoRes+ / Bible | ChrF++ 47.5 / 40.6 | Bottlenecked sentence embeddings limit expressive generation |
| + Stage 1 (with Auto-Encoding) | FLoRes+ / Bible | ChrF++ 49.7 / 45.9 | Monolingual auto-encoding strongly primes target generation fluency |
| Stage 1 (w/o Auto-Encoding) | FLoRes+ / Bible | ChrF++ 49.6 / 44.8 | Removing auto-encoding drops Bible long-tail performance by 1.1 ChrF++ |
| + Stage 2 (Token-Level Attention) | FLoRes+ / Bible | ChrF++ 50.1 / 46.4 | Unbottlenecked cross-attention enables fine-grained semantic transfer |
| OMT-NLLB (Stage 3 End-to-End) | FLoRes+ / Bible | ChrF++ 50.9 / 46.3 | Full joint optimization balances encoder and decoder representations |
Key Findings¶
- Specialization Trumps Scale: The 1B OMT-LLaMA matches or exceeds general LLMs 3×–8× its size, while the 8B OMT-LLaMA consistently outperforms LLaMA 3 70B and Tower+ 72B on low- and very-low-resource language subsets.
- Breaking the Generation Bottleneck: On the 1,560-language Bible benchmark, standard baselines degrade to random token emission beyond 300–400 languages, whereas OMT-LLaMA sustains meaningful generation quality across approximately 1,200 languages.
- Doubling Passable Language Coverage: Extrapolating to the XSTS+R+P human threshold of \(\ge 2.5\) ("passable" quality), OMT-LLaMA achieves passable into-English translation for 440 languages and OMT-NLLB for 416 languages, doubling the 221 languages supported by NLLB-200.
- Decisive Human Evaluation Win Rate: In Met-BOUQuET Round 2 human evaluation across 57 challenging directions (80 unique languages), OMT-LLaMA achieves an overall win rate of 77% against the strongest competing baseline per direction, jumping from 2.67 to 3.38 in mean XSTS+R+P score (improving from "minimally useful" to "useful").
Highlights & Insights¶
- Grammatically Targeted Seed Curation: MeDLEy shifts data engineering from opportunistic scraping to systematic coverage of 61 grammatical functions, proving essential for bootstrapping generative translation on under-resourced languages.
- Representation Auto-Encoding for Extensible Decoders: Training the decoder to reconstruct non-parallel monolingual text from language-agnostic sentence representations bridges the resource divide without requiring massive parallel corpora.
- Holistic Evaluation Ecosystem: Delivering the 275-language BOUQuET dataset alongside BLASER 3 and OmniTOX equips the research community with foundational benchmarks to assess translation adequacy, register, and safety at an unprecedented scale.
Limitations & Future Work¶
- Target Language Asymmetry: While OMT-LLaMA generates into ~1,200 languages, the more compact OMT-NLLB architecture supports only 250 target languages, leaving a gap between understanding and generation in encoder-decoder setups.
- Baseline Foundation Model Age: The current systems are built upon the LLaMA 3 family, leaving exploration of newer architectures (e.g., LLaMA 3.3 or Qwen 2.5) to future technical iterations.
- Domain and Register Skew in Long-Tail Corpora: Long-tail language data remains skewed toward religious and formal domains; verifying informal conversational registers and colloquial fluency will require extensive native-speaker field studies.
Related Work & Insights¶
- vs NLLB-200: NLLB-200 pioneered reliable 200-language translation but lacked extensible mechanisms for long-tail scaling; OMT scales coverage by an order of magnitude to 1,600+ languages and doubles passable translation varieties.
- vs General-Purpose LLMs (LLaMA3 70B / GPT-OSS 120B): Large frontier LLMs concentrate capability on commercially viable languages; OMT proves that targeted vocabulary extension, synthetic backtranslation, and specialized CPT yield a superior Pareto efficiency frontier for low-resource translation.
- vs Specialized MT Models (TranslateGemma / Tower+): Prior specialized LLM translation models supported up to several dozen or hundreds of languages; OMT provides the first validated roadmap for scaling both decoder-only and encoder-decoder pipelines into thousands of languages.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Establishes the first end-to-end framework scaling machine translation to 1,600+ languages with novel auto-encoding decoder objectives.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 275-language BOUQuET, 1,560-language Bible stress testing, and 161-direction human evaluation.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally structured and transparent, clearly articulating empirical trade-offs, architecture choices, and evaluation nuances.
- Value: ⭐⭐⭐⭐⭐ A milestone contribution toward global linguistic inclusion, open-sourcing essential benchmarks and quality estimation models.