Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ljvmiranda921/polyglot-teachers
Area: Multilingual & Machine Translation
Keywords: Multilingual LLMs, Synthetic Data Generation, Knowledge Distillation, Multilingual Evaluation, Teacher Model Selection
TL;DR¶
Addressing the common ad-hoc practice of selecting the largest available LLMs as synthetic data generators for multilingual supervised finetuning (SFT), this paper proposes POLYGLOT SCOREโa holistic metric combining intrinsic data quality with extrinsic student downstream performanceโevaluates 10 state-of-the-art models across 6 typologically diverse languages via 1.4M synthesized examples and 240 student models, and demonstrates that model scale does not predict teacher effectiveness while model family matching and method adaptation yield optimal data recipes.
Background & Motivation¶
Supervised finetuning (SFT) has become the standard mechanism for adapting foundation language models to target languages and diverse regional contexts. However, constructing high-quality instruction-tuning datasets requires substantial human annotation effort and linguistic domain expertise, creating an acute bottleneck for mid- and low-resource languages where paired prompt-response data is notoriously scarce. To circumvent this manual bottleneck, synthetic data generation using language models has gained rapid traction, where a capable teacher model generates synthetic instruction pairs to distill task capability and linguistic knowledge into smaller student models. Yet, the selection of teacher models in current research remains largely unguided, defaulting almost universally to the largest state-of-the-art models or top benchmark performers under the assumption that greater English prowess and higher benchmark rankings naturally translate into superior data generation across all target languages.
This prevailing assumption is problematic because even top-tier large models often suffer from acute capability disparities and representation deficiencies in non-English languages. Discrepancies in pretraining data representation, tokenization coverage, and cross-lingual conceptual alignment cause large models to produce subtle grammatical hallucinations, Western-centric biases, and stylistic artifacts when forced into non-English generation. When such models are deployed as teachers, their generation weaknesses are propagated directly to the distilled student models, impairing downstream generalization and causing small models to learn flawed target language distributions.
The core difficulty in resolving this issue lies in the absence of a unified, principled evaluation framework that ties the intrinsic textual properties of synthesized data directly to extrinsic downstream student performance. Prior studies have evaluated synthetic data either purely through intrinsic heuristic and embedding metrics or solely via downstream English task benchmarks, leaving practitioners with no clear understanding of what actually makes an effective multilingual teacher. Core idea: this paper introduces POLYGLOT SCORE, an integrated evaluation framework uniting intrinsic data quality and extrinsic student performance gap recovery, and demonstrates through the generation of 1.4M examples and 240 finetuned student models that model scale fails to predict teacher effectiveness, whereas data properties like prompt diversity and fluency alongside teacher-student model family matching are decisive.
Method¶
Overall Architecture¶
To systematically unpack how teacher choice, generation strategy, and language typology interact during multilingual synthetic data creation, the authors design an end-to-end evaluation pipeline. The process begins with a curated seed pool spanning 7 languages, applies three foundational generation methods across 10 diverse teacher models, and subsequently finetunes standardized student base models on the synthesized corpora. Finally, intrinsic quality metrics and extrinsic benchmark gains are harmonized into the unified POLYGLOT SCORE.
The end-to-end flow of the multilingual teacher evaluation pipeline is illustrated below:
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Multilingual Seed Dataset<br/>D_seed across 7 languages, 139k samples"] --> B["Three Generation Method Paradigms<br/>Few-shot Generate / Translate / Respond"]
B --> C["Dual-Track Quality Evaluation<br/>Intrinsic diversity & fluency + Extrinsic PGR"]
C --> D["Family-Matched Teacher-Student Alignment<br/>Cross-base validation and tokenizer alignment"]
D --> E["Output: Optimal Multilingual Synthetic Recipe<br/>Target language downstream SFT generalization"]
Key Designs¶
1. Three Generation Method Paradigms: Disentangling synthesis mechanisms across resource tiers Prior multilingual data efforts apply disjointed prompting recipes, making it difficult to isolate which generation strategy suits specific language settings. The authors formalize the space of multilingual synthetic data creation into three canonical generation methods: - Generate: The teacher model is provided with \(k\) sampled prompt-response pairs from the seed pool as in-context demonstrations and prompted to generate entirely new, distinct instruction-response pairs \((x_i, y_i)\) in the target language. This method relies heavily on the teacher's internal multilingual world knowledge and capacity to maintain task diversity without prompt degeneration. - Translate: English prompts from high-resource instruction datasets are forward-translated into the target language \(\ell\) to produce \(x_i\), after which the teacher model synthesizes the corresponding response \(y_i\) in the target language. This pipeline imports complex reasoning tasks from English into lower-resource languages but risks introducing translationese. - Respond: Target language prompts \(x_i\) are sampled directly from the native or localized seed dataset, and the teacher model is only tasked with producing the natural, detailed response \(y_i\). This isolates the teacher's answering fluency from its ability to formulate native instructions.
2. Dual-Track Quality Evaluation: Harmonizing intrinsic data properties and extrinsic student gains Evaluating teacher models solely through student finetuning is computationally prohibitive and sensitive to SFT hyperparameters, whereas relying solely on surface data heuristics fails to guarantee downstream utility. The authors establish a complementary dual-track measurement: - Intrinsic Data Quality Track: Measures semantic space dispersion, grammatical fluency, and human-aligned quality. The authors compute corpus-level prompt diversity \(d_x\) and response diversity \(d_y\) as the mean pairwise cosine distance among embeddings extracted via Llama-Embed-Nemotron-8B. Response fluency is measured via the conditional perplexity (PPL) of the base model over \(y_i\) given \(x_i\). In addition, the open multilingual judge M-Prometheus 14B evaluates each pair on a 1โ5 rubric covering fluency, naturalness, and cultural appropriateness, yielding reward score \(R\). - Extrinsic Student Performance Track: Evaluates the distilled student model \(S_{T,\ell}\) on three distinct capabilities across all languages: Cultural and factual understanding (Global-MMLU Lite), General chat alignment (M-RewardBench), and Mathematical reasoning (M-GSM). Rather than reporting raw accuracy, improvements are normalized via the Performance Gap Recovered (PGR) relative to an instruction-tuned reference model \(S_{ref}\) derived from the same base model checkpoint.
3. Family-Matched Teacher-Student Alignment: Exploiting architectural synergy over raw scale While practitioners often default to massive external teacher models, empirical investigation reveals that teacher scale and benchmark scores do not reliably correlate with student improvements. Instead, architectural and tokenization congruence between teacher and student emerges as a primary performance driver. When teacher and student share the same model family, they benefit from identical subword vocabulary boundaries, shared tokenization splits, and compatible representation manifolds. This alignment substantially eases the transfer of synthetic knowledge during SFT. Consequently, model family matching provides a robust, cost-effective default strategy whenever target-language teacher quality is uncertain.
Formal Definition and Evaluation Protocol¶
Given teacher model \(T\), target language \(\ell\), and seed dataset \(D_{seed,\ell}\), the synthesized dataset of size \(N\) is denoted as \(X_{T,\ell} = \{(x_i, y_i)\}_{i=1}^N\): $\(X_{T,\ell} = T(D_{seed,\ell})\)$ The student model \(S_{T,\ell}\) is produced by finetuning base model \(S_{base}\) on \(X_{T,\ell}\): $\(S_{T,\ell} = \text{SFT}(S_{base}, X_{T,\ell})\)$
The intrinsic quality score \(\text{Intrinsic}_{T,\ell}\) aggregates the z-score normalized metrics: $\(\text{Intrinsic}_{T,\ell} = \frac{1}{|M|} \sum_{m \in M} \text{z-score}(m(X_{T,\ell}))\)$ where \(M = \{d_x, d_y, -\log(1 + \text{PPL}), R\}\).
Extrinsic downstream evaluation computes the Performance Gap Recovered (PGR) across benchmarks \(B = \{\text{CULTURE}, \text{CHAT}, \text{MATH}\}\): $\(\text{PGR}_b(S_{T,\ell}) = \frac{\text{score}_b(S_{T,\ell}) - \text{score}_b(S_{base})}{\text{score}_b(S_{ref}) - \text{score}_b(S_{base})}\)$ $\(\text{Extrinsic}_{T,\ell} = \frac{1}{|B|} \sum_{b \in B} \text{PGR}_b(S_{T,\ell})\)$ where \(S_{base}\) is OLMo 3 7B and \(S_{ref}\) is OLMo 3 7B Instruct SFT in the primary setup.
The unified POLYGLOT SCORE is defined as the z-score of the combined intrinsic and extrinsic metrics: $\(\text{PG-SCORE}_{T,\ell} = \text{z-score}(\text{Intrinsic}_{T,\ell} + \text{Extrinsic}_{T,\ell})\)$ A generalized formulation \(\text{PG-SCORE}_{T,\ell} = \alpha I + (1 - \alpha)E\) with \(\alpha \in [0, 1]\) confirms that rankings remain highly stable across neighboring weights (\(\rho \ge 0.90\)), validating equal weighting (\(\alpha = 0.5\)) as an unbiased standard.
Loss & Training¶
All student models are trained using full-parameter supervised finetuning with standard autoregressive cross-entropy loss: $\(\mathcal{L}_{\text{SFT}}(\theta) = -\sum_{i=1}^N \sum_{t=1}^{|y_i|} \log P(y_{i,t} \mid x_i, y_{i,<t}; \theta)\)$ Training uses the Unsloth framework on NVIDIA Grace Hopper GH200 superchips. For the 7B student models, training is configured with a global batch size of 32, 4 gradient accumulation steps, AdamW optimizer, an initial learning rate of \(5 \times 10^{-5}\) with linear scheduling, sequence length limit of 16,384 tokens, weight decay of 0.001, and runs for 2 epochs (approximately 1.5 wall-clock hours on 2 nodes).
Key Experimental Results¶
Main Results¶
The primary benchmark evaluates 10 teacher models across 6 typologically diverse target languages: Arabic (ar), Czech (cs), German (de), Spanish (es), Indonesian (id), and Japanese (ja). The resulting PG-SCORE values with OLMo 3 7B as the student base are summarized below:
| Teacher Model | Average | Arabic (ar) | Czech (cs) | German (de) | Spanish (es) | Indonesian (id) | Japanese (ja) |
|---|---|---|---|---|---|---|---|
| Gemma 3 27B Inst. | 0.726 | 0.145 | 0.360 | 1.655 | 1.358 | 0.214 | 0.626 |
| Aya Expanse 32B | 0.706 | โ0.058 | 0.222 | 1.468 | 1.129 | 1.153 | 0.320 |
| Gemma 3 12B Inst. | 0.595 | โ0.464 | 0.327 | 1.756 | 1.228 | 0.151 | 0.573 |
| Command A (104B) | 0.546 | โ1.360 | 0.114 | 1.673 | 1.102 | 1.063 | 0.683 |
| Gemma 3 4B Inst. | 0.469 | โ0.488 | 0.330 | 1.644 | 0.929 | โ0.105 | 0.504 |
| GPT-4o mini | 0.461 | โ1.117 | 0.015 | 1.766 | 0.908 | 1.003 | 0.189 |
| IBM Granite 4.0 (3B) | 0.312 | โ0.072 | โ0.031 | 1.000 | 0.734 | โ0.079 | 0.321 |
| IBM Granite Micro (0.4B) | 0.304 | โ0.282 | 0.290 | 1.102 | 0.783 | โ0.329 | 0.264 |
| Llama 3.1 70B Inst. | 0.140 | โ0.964 | 0.109 | 1.195 | 0.688 | 0.182 | โ0.373 |
| Llama 3.1 8B Inst. | โ0.356 | โ1.693 | โ0.974 | 0.891 | 0.182 | 0.322 | โ0.863 |
The results highlight that Gemma 3 27B (0.726) and Aya Expanse 32B (0.706) significantly outperform much larger models like Llama 3.1 70B Instruct (0.140). Even smaller variants like Gemma 3 12B (0.595) and Gemma 3 4B (0.469) surpass Llama 3.1 70B.
Cross-Base Generalization and Method Ablation¶
To verify whether PG-SCORE rankings remain valid across different student base architectures, the authors evaluated teachers across Llama 3.1 8B, Gemma 3 4B, Qwen 3 8B, and OLMo 3 7B (averaged across Arabic, German, and Indonesian):
| Teacher Model | OLMo 3 7B Base | Gemma 3 4B Base | Qwen 3 8B Base | Llama 3.1 8B Base |
|---|---|---|---|---|
| Gemma 3 27B Inst. | 0.672 | 0.810 | 1.301 | 0.800 |
| Aya Expanse 32B | 0.854 | 0.762 | 1.183 | 0.793 |
| Gemma 3 12B Inst. | 0.481 | 0.666 | 1.393 | 0.804 |
| Gemma 3 4B Inst. | 0.350 | 0.712 | 0.545 | 1.062 |
| Command A | 0.459 | 0.725 | 0.974 | 0.737 |
| GPT-4o mini | 0.551 | 1.022 | 1.005 | 0.621 |
| Qwen 3 8B | 0.398 | 0.615 | 1.421 | 0.662 |
| OLMo 3 7B Inst. | 0.731 | 0.295 | 0.581 | 0.277 |
| Llama 3.1 70B Inst. | 0.138 | 0.338 | 1.039 | 0.497 |
| Llama 3.1 8B Inst. | โ0.160 | โ0.133 | 0.365 | 0.048 |
Furthermore, the choice of generation method produces substantial performance variations depending on language resource availability:
| Language | Best Generation Method | Gemma 3 27B Gain | Aya Expanse 32B Gain | Llama 3.1 70B Gain |
|---|---|---|---|---|
| Arabic (ar, low-resource/challenging) | Respond | +453.1% | +355.2% | +77.7% |
| German (de, high-resource) | Generate | +29.3% | +0.3% | +16.4% |
| Indonesian (id, mid-resource) | Translate | +458.9% | +39.3% | โ14.8% |
Key Findings¶
- Absence of Scale Predictability: Mixed-effects regression reveals that log parameter size (\(\beta = 0.053, p = 0.507\)) and average multilingual benchmark performance (\(\beta = 1.387, p = 0.529\)) are not statistically significant predictors of PG-SCORE. Conventional assumptions equating raw size with data generation efficacy are unsupported.
- Predictive Power of Intrinsic Metrics: Principal Component Analysis (PCA) shows that four principal components capture 93.3% of the variance in intrinsic data quality metrics. PC1 (42.2%) represents response fluency and distinctiveness, while PC2 (22.1%) captures prompt diversity and length. A linear model trained on these intrinsic PCs predicts student downstream performance with \(R^2 = 0.664\) and \(\text{RMSE} = 0.440\), providing a fast proxy for screening teachers without SFT runs.
- Resource-Dependent Method Selection: In high-resource languages like German with rich seed data, few-shot Generate yields the highest gains. For less-resourced languages like Arabic and Indonesian, relying on existing prompts via Respond or transferring from English via Translate yields massive improvements (exceeding +450% PG-SCORE).
Highlights & Insights¶
- Debunking Scale Superiority in Data Distillation: The finding that Gemma 3 27B and Aya Expanse 32B systematically outperform 70B+ models provides an immediate empirical rationale for practitioners to discard costly proprietary or oversized generators in favor of compact, multilingually balanced architectures.
- Family Matching Advantage: Demonstrating that pairing student models with teachers of the same family (e.g., Gemma with Gemma, Qwen with Qwen) consistently places them among top performers shows that shared tokenizer segmentation boundaries and pretraining dynamics mitigate distillation losses.
- Intrinsic Data Quality as an Efficient Screening Proxy: Establishing an \(R^2 = 0.664\) linear mapping from prompt diversity, length, and perplexity to student downstream performance gives practitioners a reliable metric for filtering candidate datasets without full-scale student fine-tuning.
Limitations & Future Work¶
- Language Set Breadth: Although the 6 target languages were selected based on typological variation and benchmark availability, validating findings across endangered and extremely low-resource indigenous languages remains open.
- Translationese and Cultural Nuance: The Translate paradigm presupposes accessible English prompts, inheriting structural translation biases and difficulty in capturing localized cultural idioms.
- Representation Gaps in Pretraining: PG-SCORE correlates positively with language prevalence in CommonCrawl (\(\rho = 0.886, p < 0.05\)), highlighting structural disparities in pretraining data that still challenge synthetic pipelines for tail languages.
Related Work & Insights¶
- vs. Bactrian-X / MultiAlpaca / xP3-MT: Prior multilingual synthetic datasets relied on single rigid methods (e.g., pure Google Translate or pure multilingual Self-Instruct). This paper decouples and compares Generate, Translate, and Respond across diverse teachers, showing that optimal generation strategy depends heavily on target language resource tier.
- vs. Kim et al. (2025) / Xu et al. (2025b): While previous English-focused studies observed that stronger models do not always make better teachers, this work demonstrates that the phenomenon is exacerbated in multilingual settings, where tokenizer overlap and multilingual pretraining quality outweigh benchmark-measured English strength.
Rating¶
- Novelty: โญโญโญโญโญ [Introduces the first holistic multilingual teacher evaluation benchmark uniting intrinsic metrics and student performance gap recovery, overturning conventional scale assumptions.]
- Experimental Thoroughness: โญโญโญโญโญ [Generates 1.4M synthetic pairs, trains 240 student models across 10 teachers and 6 languages, validated on 4 base models and an unseen language.]
- Writing Quality: โญโญโญโญโญ [Exceptionally structured, rigorous statistical modeling, cohesive empirical analysis, and clear practical recipes.]
- Value: โญโญโญโญโญ [Delivers immediate, actionable guidance and open-source recipes for cost-effective multilingual language model post-training.]