Skip to content

Parallel Tokenizers: Rethinking Vocabulary Design in Cross-Lingual Transfer of Low-Resource Languages

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/
Area: Multilingual & Translation
Keywords: Parallel Tokenizers, Low-Resource Languages, Vocabulary Alignment, Cross-Lingual Transfer, Fertility Balance

TL;DR

Addressing semantic index fragmentation and severe fertility imbalances in conventional multilingual tokenizers, this paper introduces Parallel Tokenizers, which build monolingual tokenizers aligned at the word level via machine translation and broadcast language identity embeddings to achieve superior cross-lingual transfer across 13 low-resource languages.

Background & Motivation

In multilingual pretrained language modeling, standard practice trains a single shared subword tokenizer (typically based on Byte Pair Encoding or WordPiece) over a combined multilingual corpus. However, this shared approach suffers severely from the curse of multilinguality in low-resource settings: high-resource languages overwhelmingly dominate the shared vocabulary budget, leaving underrepresented languages severely fragmented. Consequently, low-resource sentences are partitioned into excessively long token sequencesโ€”exhibiting disproportionately high fertility scores that inflate inference latency, memory consumption, and overall computational costs.

Beyond sequence length disparities, a more fundamental structural failure lies in semantic index separation and cross-lingual interference. In traditional multilingual tokenizers, semantically equivalent words across languages (such as "eat" in English, "ci" in Hausa, and "้ฃŸในใ‚‹" in Japanese) are mapped to entirely distinct discrete vocabulary indices, erecting an artificial barrier against sharing embedding representations. Conversely, accidental orthographic overlap creates false cognates, where identical subword surface forms carrying conflicting meanings in different languages are forced to share identical embeddings. These limitations prevent pretrained models from exploiting discrete semantic equivalence at the input layer, compelling transformer layers to compensate via expensive latent alignment.

While prior research has explored vocabulary capacity rebalancing or script clustering, these strategies do not eliminate semantic redundancy where identical meanings occupy disjoint indices. This paper tackles the problem from a fresh angle: rather than learning a single compromised tokenizer across languages, one can train clean monolingual tokenizers and explicitly align their discrete word-level vocabularies. Core idea: train monolingual pivot tokenizers and align their word-level vocabularies using machine translation, paired with globally broadcast language identity embeddings, ensuring semantically equivalent words share identical discrete indices while isolating language-specific syntactic features.

Method

Overall Architecture

Parallel Tokenizers replace the traditional single-tokenizer framework with a language-specific set of parallel tokenizers \(\mathcal{T} = \{T_1, T_2, \dots, T_k\}\). For any input text, an explicit language identifier first routes the sequence to its corresponding language-specific tokenizer \(T_j\). The resulting token IDs are mapped through a shared embedding matrix, and the final input representation is formed by summing token embeddings, segment embeddings, positional embeddings, and a dense language identity embedding broadcast across all token positions.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Text + Language Identifier Signal"] --> B["Parallel Tokenizer Selection<br/>Routes to language-specific tokenizer"]
    B --> C["Tokenization & Vocabulary Mapping<br/>Aligned word indices + language subwords"]
    C --> D["Input Representation Fusion<br/>Token/Segment/Position + Language Identity"]
    D --> E["Transformer Encoder<br/>Shared underlying multilingual representation"]

Key Designs

1. Word-Type Vocabulary Translation and Alignment: Establishing Shared Semantic Anchors Existing bilingual dictionaries such as PanLex suffer from severe coverage and data quality limitations in low-resource regimes, achieving only an average of 21.5% token alignment. To overcome this, the authors first train an English monolingual WordPiece tokenizer with a vocabulary size of 30,522 on English Wikipedia and isolate its word-type tokens (which account for 65.87% of the total vocabulary, filtering out numbers, short words, and subwords). These English word-type tokens are then translated into each target language using machine translation (Google Translate) and verified via back-translation filtering to eliminate multiword or malformed phrases. In the resulting parallel vocabularies, translated equivalent words are assigned identical token indices, allowing cross-lingual equivalents to directly share word embeddings. Across 13 low-resource languages, an average of 82% of word-type tokens (61% of all tokens in the vocabulary) achieve exact semantic alignment.

2. Target Language Monolingual Tokenizer Expansion and Concatenation: Preserving Morphological Nuance Because translated words cover only a subset of the full vocabulary, each parallel tokenizer is expanded to the full capacity of 30,522 entries by incorporating language-specific subwords and characters. A monolingual WordPiece tokenizer is trained on the target language's Wikipedia corpus to identify natural morphological subword boundaries. The final parallel vocabulary is formed by prioritizing: (1) aligned translated word-type tokens, (2) special tokens (e.g., [CLS], [SEP], [UNK]) and a 1,000-character English foundational subset, and (3) the target language's monolingual vocabulary. Duplicates are removed and the list is capped at exactly 30,522 entries, guaranteeing identical parameter counts to the standard multilingual baseline while maintaining compact segmentation.

3. Language Identity Embedding Broadcasting: Disambiguating Unaligned Tokens and Syntax Directly sharing discrete indices introduces potential cross-lingual ambiguities for the remaining 39% unaligned tokens and interlingual homographs (words that share spelling but differ in meaning). Rather than inserting a discrete language token into the input sequence, the language identifier selects a continuous language identity embedding \(\mathbf{E}_{\text{lang}}\) that is broadcast across all \(m\) tokens: $\(\mathbf{E}_{\text{final}} = \mathbf{E}_{\text{token}} + \mathbf{E}_{\text{seg}} + \mathbf{E}_{\text{pos}} + \mathbf{E}_{\text{lang}}\)$ This design provides an essential syntactic cue and disambiguation signal for unaligned subwords, allowing the model to enjoy shared semantic embeddings without conflating distinct grammatical rules or linguistic behaviors.

Key Experimental Results

Main Results

The framework is evaluated by pretraining a Transformer encoder from scratch on 13 low-resource languages spanning Latin and Amharic scripts (Acehnese, Amharic, Balinese, Hausa, Igbo, Javanese, Kinyarwanda, Minangkabau, Oromo, Sundanese, Swahili, Tigrinya, Twi) against a standard shared multilingual baseline (Single-13L).

Task / Benchmark Metric Single-13L (Baseline) Parallel-13L (Ours) Gain
FLORES+ Fertility Score (Tokens/Word) โ†“ Tokens/Word 1.89 1.57 โ†“ 0.32
FLORES+ Parity Score (Token Count Ratio) โ†“ Parity Score 1.14 1.07 โ†“ 0.07
NusaX-senti Sentiment Analysis (100% data) Macro-F1 76.09 (ยฑ1.27) 76.16 (ยฑ1.05) +0.07
AfriHate Hate Speech Detection (100% data) Macro-F1 69.61 (ยฑ1.19) 69.80 (ยฑ1.20) +0.19
EthioEmo Emotion Classification (100% data) Weighted-F1 54.98 (ยฑ0.98) 57.01* (ยฑ0.84) +2.03*
BRIGHTER Emotion Classification (100% data) Weighted-F1 48.26 (ยฑ0.94) 49.68* (ยฑ1.89) +1.42*
Downstream Classification Average (100% data) Average F1 62.24 63.16 +0.92
FLORES+ Bitext Mining Error Rate โ†“ Avg. err. xsim 83.56 74.08 โ†“ 9.48
FLORES+ Best Bitext Mining Pairs (out of 78) โ†‘ # of best xsim 12 66 +54 pairs

Ablation Study and Limited Data Scenarios

In real-world low-resource settings, training data in the target language is frequently restricted or absent. The table below reports cross-lingual transfer under partial-shot (50%) and zero-shot (0%) target language fine-tuning conditions, where auxiliary languages are available at 100%:

Data Availability Benchmark Single-13L (Baseline) Parallel-13L (Ours) Difference
50% Target Language Data NusaX-senti (Macro-F1) 72.47 73.52* +1.05
50% Target Language Data AfriHate (Macro-F1) 67.26 67.40 +0.14
50% Target Language Data EthioEmo (Weighted-F1) 51.33 54.27* +2.94
50% Target Language Data BRIGHTER (Weighted-F1) 45.77 46.76* +0.99
0% Target Language Data (Zero-shot) NusaX-senti (Macro-F1) 71.83 72.70 +0.87
0% Target Language Data (Zero-shot) AfriHate (Macro-F1) 37.27 40.67 +3.40
0% Target Language Data (Zero-shot) EthioEmo (Weighted-F1) 23.75 23.20 -0.55
0% Target Language Data (Zero-shot) BRIGHTER (Weighted-F1) 15.04 19.73 +4.69

Key Findings

  • Substantial Fertility Improvements: Parallel-13L reduces the average fertility score from 1.89 to 1.57 tokens per word, approaching the 1.52 score of specialized monolingual tokenizers. The improvement is especially pronounced for non-Latin scripts such as Amharic (2.43 down to 1.82) and Tigrinya (2.61 down to 1.83), substantially reducing sequence truncation and inference latency.
  • Enhanced Cross-Lingual Semantic Cohesion: In bitext mining over FLORES+, the xsim error rate drops by 9.48 points, and Parallel-13L wins in 66 out of 78 language pairs. PCA visualizations reveal tight cross-lingual clustering driven by semantic equivalence rather than separate language-family clusters.
  • Superior Low-Resource & Zero-Shot Transfer: Under extreme zero-shot transfer (0% target data), Parallel-13L outperforms the baseline by up to +4.69 F1 on BRIGHTER and +3.40 F1 on AfriHate, proving that discrete index alignment provides a strong structural prior for cross-lingual knowledge transfer.

Highlights & Insights

  • Direct Discrete Prior Injection: Instead of delegating cross-lingual alignment entirely to the latent attention weights, Parallel Tokenizers align semantic equivalents directly at the discrete vocabulary index level, giving the model shared initial anchors across languages.
  • Monolingual Specialization Meets Multilingual Sharing: The three-step construction (pivot translation alignment + target language subword extraction + unified vocabulary capping) elegantly resolves the long-standing tension between monolingual segmentation quality and multilingual parameter sharing.
  • Efficient Disambiguation via Broadcast Embeddings: Broadcasting continuous language identity embeddings bypasses the overhead of explicit prompt-style language tokens, allowing unaligned subwords and false cognates to retain language-specific behavior without breaking semantic transfer.

Limitations & Future Work

  • Dependency on Machine Translation Fidelity: Vocabulary alignment relies on the accuracy of word-level machine translation. Human verification revealed an average translation accuracy of 83% across local Indonesian languages (with Minangkabau dropping to 76%), indicating that translation noise and malformed outputs remain an open challenge.
  • Exploration Restricted to Encoder Architectures: Experiments focus strictly on encoder-only architectures for classification and representation retrieval; evaluating whether parallel tokenizers scale effectively to autoregressive decoder-based LLMs remains for future work.
  • Language Scalability Overhead: Integrating a new language requires obtaining translated word lists and training a separate monolingual tokenizer, which requires external dictionary or MT infrastructure rather than pure unsupervised corpus harvesting.
  • vs XLM-V / Balanced Multilingual Tokenization (Chung et al., 2020; Liang et al., 2023): Prior works increase vocabulary size or reweight subword sampling across languages to mitigate fertility gaps, yet semantically identical words still occupy separate indices. Parallel Tokenizers achieve both fertility reduction and shared semantic representations within a compact 30,522 vocabulary.
  • vs Tokenization-Free / Byte-Level Models (ByT5, CANINE, Xue et al., 2022): While byte/character encoders eliminate vocabulary design altogether, they drastically inflate sequence length and discard lexical priors. Parallel Tokenizers preserve high token efficiency while delivering strong cross-lingual generalization in low-resource regimes.

Rating

  • Novelty: โญโญโญโญโ˜† An insightful rethink of vocabulary index allocation that directly bridges multilingual lexical gaps at the discrete embedding layer.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorous evaluation across 13 diverse low-resource languages, four classification benchmarks, zero/partial-shot setups, and bitext mining similarity.
  • Writing Quality: โญโญโญโญโญ Well-structured narrative with crisp visual illustrations and solid empirical analysis.
  • Value: โญโญโญโญโ˜† Offers valuable architectural principles for democratizing multilingual NLP and optimizing low-resource representation learning.