ShapleyLaw: A Game-Theoretic Approach to Multilingual Scaling Laws¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/teijyogen/ShapleyLaw
Area: LLM Reasoning
Keywords: Multilingual Pretraining, Scaling Laws, Cooperative Game Theory, Shapley Value, Data Mixture Optimization
TL;DR¶
ShapleyLaw is the first game-theoretic multilingual scaling law that treats pretraining languages as players in a cooperative game, quantifies cross-lingual transfer via In-Run Data Shapley values, and enables accurate per-language loss prediction and provably optimal language-level mixture ratio optimization.
Background & Motivation¶
As modern large language models serve an increasingly global user base, multilingual pretraining on multi-language corpora has become standard practice for building foundational architectures. The performance of a pretrained model on any target language depends not only on the raw model parameters and total token budget, but also critically on the precise proportion of each language within the training corpus. Because linguistic knowledge transfers unevenly across languages, determining the optimal language mixture ratios that minimizes multilingual evaluation loss is a fundamental challenge in scalable LLM engineering.
Prior approaches have predominantly relied on heuristic smoothed sampling or overly simplified assumptions. The current state-of-the-art multilingual scaling law, FamilyLaw, attempts to tame the combinatorial complexity of cross-lingual interactions by grouping languages into families and enforcing a strict family-independence assumption—presuming that cross-lingual transfer occurs exclusively within the same language family. This assumption fails in practice: for example, although Chinese, Japanese, and Spanish belong to entirely different language families, Japanese training data contributes significantly more to Chinese language performance than Spanish due to extensive shared logographic script and lexical overlap. Consequently, family-level scaling laws cannot predict single-language task performance, nor can they optimize fine-grained sampling allocations within families.
To resolve these limitations, this paper reframes multilingual pretraining through the lens of cooperative game theory. Every participating language in the pretraining mix is viewed as a rational player whose collaborative effort reduces the test loss on a target evaluation language. Core idea: formulate multilingual pretraining as a cooperative game, quantify cross-lingual transfer via the axiomatic Shapley Value using an \(O(1)\) single-run gradient approximation, and construct ShapleyLaw by modulating Chinchilla base scaling with aggregate transfer to enable language-level mixture optimization.
Method¶
Overall Architecture¶
The ShapleyLaw pipeline consists of four tightly integrated stages: formulating the Multilingual Pretraining Game (MPG), approximating and normalizing pairwise Shapley values via In-Run Data Shapley (IRDS), fitting the aggregate transfer scaling law on small-scale anchor models, and solving the resulting convex optimization problem for optimal language-level sampling ratios under corpus size constraints. The workflow takes multilingual raw text, tracks language-wise gradient projections during a single lightweight reference training run to compute transfer coefficients, and outputs both loss predictions for arbitrary configurations and mathematically optimal data mixture weights for downstream multi-billion parameter pretraining.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multilingual Corpora<br/>K training languages with token limits"] --> B["Multilingual Pretraining Game & Shapley Metric<br/>Formulate MPG and assess marginal loss reduction per language"]
B --> C["In-Run Data Shapley Single-Run Estimation<br/>Accumulate step-wise gradient inner products in O(1) training run"]
C --> D["Aggregate Cross-Lingual Transfer Modulation<br/>Compute mixture-weighted normalized SVs and modulate Chinchilla base"]
D --> E["KKT-Constrained Convex Mixture Optimization<br/>Solve Lagrange multipliers under simplex and token budget caps"]
E --> F["Output: Optimal language mixtures & predicted task losses"]
Key Designs¶
1. Multilingual Pretraining Game & Shapley Metric: Rigorous axiomatic quantification of asymmetric cross-lingual transfer
Heuristic transfer measures lack theoretical grounding and fail to disentangle overlapping language contributions when multiple languages are trained simultaneously. The authors define the Multilingual Pretraining Game (MPG) \(G = (\mathcal{K}, N, D, \mathcal{V})\), where each language \(i \in \mathcal{K}\) acts as a player, and the payoff \(v_j^G(S)\) of a coalition \(S \subseteq \mathcal{K}\) on target task \(V_j\) is the test loss reduction achieved by pretraining a model using uniform data from subset \(S\). Under this cooperative formulation, the Shapley value (SV) is the unique attribution metric satisfying efficiency, linearity, symmetry, and null player properties. The marginal contribution of language \(i\) to task \(j\) is computed as: $\(\phi_{i,j}^{\text{SV}}(G) = \sum_{S \subseteq \mathcal{K} \setminus \{i\}} \frac{|S|!(|\mathcal{K}|-|S|-1)!}{|\mathcal{K}|!} \left( v_j^G(S \cup \{i\}) - v_j^G(S) \right)\)$ To ensure fair comparability across tasks with differing intrinsic difficulty, the authors apply exponential normalization: $\(\phi_{i,j}^{\text{NSV}}(G) = \exp \left( \phi_{i,j}^{\text{SV}}(G) - \max_{i'} \phi_{i',j}^{\text{SV}}(G) \right)\)$ which pegs the dominant self-transfer coefficient strictly to 1 and smoothly maps cross-lingual transfers into \((0, 1]\), providing a normalized transfer matrix suitable for downstream scaling.
2. In-Run Data Shapley Single-Run Estimation: Avoiding exponential pretraining costs via dynamic gradient projections
Computing exact Shapley values requires evaluating all non-empty language coalitions, which necessitates pretraining \(2^K - 1\) distinct models—a computationally intractable \(O(2^K)\) barrier for realistic language sets. To overcome this, the authors adapt In-Run Data Shapley (IRDS), which measures how language \(i\) contributes to validation loss reduction along a single multilingual pretraining trajectory. Using a first-order Taylor expansion on local validation loss, the IRDS estimator aggregates gradient dot-products across optimization steps: $\(\hat{\phi}_{i,j}^{\text{IRDS}} = \sum_{t=0}^{T-1} \eta_t \langle g_{t,j}^{\text{val}}, g_{t,i} \rangle\)$ where \(g_{t,j}^{\text{val}} = \nabla \ell_j(w_t)\) is the validation gradient on target language \(j\), \(g_{t,i} = \sum_{z \in B_{t,i}} \nabla \ell(w_t, z)\) is the aggregated training gradient for language \(i\), and \(\eta_t\) is the learning rate. This reduces computation from \(O(2^K)\) models to \(O(1)\) single pretraining run. Empirically, IRDS exhibits a 0.9584 cosine similarity and 0.9390 Pearson correlation with exact SVs. Crucially, the SV matrix proves highly transferable across model scales: the correlation between SVs calculated at 50M and 243M reaches 0.999, allowing practitioners to precalculate transfer matrices on cheap small models and reuse them for scaling laws of multi-billion parameter models.
3. Aggregate Cross-Lingual Transfer Modulation: Parameterizing scaling laws with language mixture variables
Standard scaling laws (such as Chinchilla) assume homogeneous data and cannot account for heterogeneous language compositions. ShapleyLaw introduces the aggregate cross-lingual transfer quantity \(\Theta_j\), defined as the mixture-weighted sum of normalized transfer coefficients: $\(\Theta_j = \sum_{i \in \mathcal{K}} p_i \cdot \phi_{i,j}^{\text{NSV}}\)$ The test cross-entropy loss on target language \(j\) is then modeled as the product of the base scale-dependent loss and a transfer-induced modulation factor: $\(L_j(N, D, p \mid \{\phi^{\text{NSV}}\}) = L_{\text{chin}}^j(N, D) \cdot (\Theta_j)^{-\gamma_j} = \left( E_j + \frac{A_j}{N^{\alpha_j}} + \frac{B_j}{D^{\beta_j}} \right) (\Theta_j)^{-\gamma_j}\)$ where \(E_j, A_j, B_j, \alpha_j, \beta_j, \gamma_j\) are task-specific empirical fitting parameters. When model size \(N\) and data budget \(D\) are held constant, swapping language ratios directly modifies \(\Theta_j\) according to the respective transfer strengths, accurately predicting the resulting shifts in target loss.
4. KKT-Constrained Convex Mixture Optimization: Provably optimal resource allocation under corpus budget limits
With ShapleyLaw predicting language losses, the overall multilingual objective is defined as the weighted sum \(J(p) = \sum_{j=1}^K w_j L_j(N, D, p)\). For \(\gamma_j > 0\), the function \(x \mapsto x^{-\gamma_j}\) is strictly convex, and because \(\Theta_j\) is affine in \(p\), \(J(p)\) is guaranteed to be strictly convex over the probability simplex \(\Delta^{K-1}\). In the self-dominant diagonal approximation, the KKT conditions yield an elegant closed-form optimal allocation \(p_i^\star \propto (w_i C_i \gamma_i)^{1/(\gamma_i + 1)}\), which is further refined via a first-order perturbation term capturing off-diagonal transfers. When languages face physical corpus caps \(p_i \le \bar{p}_i = \bar{D}_i / D\), the unconstrained optimum is projected onto the truncated simplex: data-limited languages saturate their caps (\(p_i^\star = \bar{p}_i\)), and remaining probability mass is iteratively redistributed among data-rich languages according to their marginal utility, guaranteed to converge in at most \(K\) steps.
Loss & Training¶
Pretraining is conducted using Megatron-LM on 8-node clusters of \(8 \times \text{H200}\) GPUs. Models follow a decoder-only Transformer architecture with RMSNorm, RoPE, SwiGLU, and the 100k-vocabulary llm-jp tokenizer. Hyperparameters include an Adam optimizer (\(\beta_1=0.9, \beta_2=0.95\)), base learning rate of \(3 \times 10^{-4}\) with cosine decay, sequence length 4,096, and global batch size 512. Normalization values and transfer matrices are estimated using an MPG run configured at \(N=50\text{M}\) and \(D=50\text{B}\), and validated across model sizes \(N \in \{50\text{M}, 243\text{M}, 684\text{M}, 1460\text{M}, 3171\text{M}\}\) and \(D=100\text{B}\).
Key Experimental Results¶
Main Results¶
The authors pretrain models from scratch at \(N=243\text{M}, D=100\text{B}\) using mixtures derived from eight distinct allocation strategies across 10 languages spanning 8 language families, evaluating under both Unweighted Sum and Normalized Sum (weighted by the inverse of monolingual loss) objectives.
| Strategy | en | de | fr | es | zh | ja | ko | fi | hr | ms | Unweighted CE Loss ↓ | Normalized CE Loss ↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Uniform | 10.0 | 10.0 | 10.0 | 10.0 | 10.0 | 10.0 | 10.0 | 10.0 | 10.0 | 10.0 | 24.440 | 11.985 |
| Smoothed (\(\alpha=0.5\)) | 13.2 | 14.5 | 12.6 | 13.6 | 19.1 | 11.4 | 4.9 | 4.7 | 3.7 | 2.4 | 24.683 | 12.180 |
| Common Crawl | 61.4 | 8.3 | 6.4 | 6.4 | 7.2 | 8.2 | 1.2 | 0.5 | 0.3 | 0.1 | 25.491 | 12.715 |
| FineWeb-2 | 80.6 | 3.3 | 2.3 | 2.7 | 6.7 | 3.1 | 0.5 | 0.4 | 0.3 | 0.1 | 27.133 | 13.613 |
| FamilyLaw (He et al., 2025) | 14.1 | 14.1 | 1.8 | 1.8 | 12.1 | 11.6 | 10.2 | 12.6 | 11.0 | 10.7 | 24.749 | 12.153 |
| ODM (Shukor et al., 2025) | 1.5 | 30.0 | 5.1 | 7.2 | 6.0 | 30.0 | 1.5 | 1.5 | 15.7 | 1.5 | 25.544 | 12.592 |
| DML (Ye et al., 2025) | 1.5 | 36.7 | 4.7 | 17.5 | 5.9 | 5.0 | 1.5 | 5.3 | 8.9 | 13.0 | 25.330 | 12.304 |
| ATLAS (Longpre et al., 2026) | 1.5 | 1.5 | 5.1 | 1.7 | 3.4 | 28.0 | 23.3 | 12.4 | 12.5 | 10.5 | 25.353 | 12.518 |
| ShapleyLaw (Ours) | 26.0 | 13.5 | 1.5 | 1.5 | 20.2 | 13.4 | 10.3 | 5.3 | 6.8 | 1.5 | 24.108 | 11.813 |
Note: In the Normalized Sum setting, ShapleyLaw selects en 18.1%, de 15.4%, fr 1.5%, es 1.5%, zh 19.7%, ja 9.3%, ko 9.6%, fi 10.3%, hr 12.0%, ms 2.6%, achieving the minimum CE loss of 11.813.
Ablation & Downstream Evaluation¶
To assess the precision of IRDS estimation versus exact SVs, as well as downstream task alignment, comprehensive evaluations were conducted.
Table A: Exact SV vs. IRDS Approximation Fitting Error (\(PE_{\text{fit}}\))
| Metric / Language | de | en | fr | es | ja | ko | zh | Global Alignment |
|---|---|---|---|---|---|---|---|---|
| Exact SV Fitting Error (\(PE_{\text{fit}}\)) | 0.015 | 0.004 | 0.015 | 0.014 | 0.006 | 0.009 | 0.015 | Cosine Sim = 1.000 |
| IRDS Approximation (\(PE_{\text{fit}}\)) | 0.015 | 0.003 | 0.036 | 0.017 | 0.008 | 0.023 | 0.013 | Cosine Sim = 0.9584 / Pearson r = 0.9390 |
| Difference (\(\Delta PE_{\text{fit}}\)) | 0.000 | -0.001 | +0.021 | +0.003 | +0.002 | +0.014 | -0.002 | Mean \(\ell_1\) Distance \(\approx 10.1\) |
Table B: Downstream Multilingual Reasoning Accuracy on HellaSwag (5-Shot)
| Optimization Strategy | en (%) | de (%) | fr (%) | es (%) | hr (%) | 5-Language Avg. Accuracy ↑ |
|---|---|---|---|---|---|---|
| Uniform | 20.0 | 20.0 | 20.0 | 20.0 | 20.0 | 0.343 |
| Smoothed (\(\alpha=0.5\)) | 25.2 | 22.9 | 21.9 | 23.6 | 6.4 | 0.344 |
| Common Crawl | 73.9 | 10.0 | 7.7 | 7.8 | 0.6 | 0.354 |
| FineWeb-2 | 90.3 | 3.7 | 2.5 | 3.0 | 0.5 | 0.355 |
| FamilyLaw | 31.7 | 31.7 | 4.1 | 4.1 | 28.4 | 0.346 |
| ODM | 0.1 | 88.2 | 0.1 | 11.5 | 0.1 | 0.321 |
| DML | 0.1 | 21.1 | 0.1 | 78.6 | 0.1 | 0.331 |
| ATLAS | 0.1 | 19.5 | 38.4 | 15.7 | 26.3 | 0.338 |
| ShapleyLaw (Ours) | 95.7 | 0.1 | 0.1 | 0.1 | 4.0 | 0.356 |
Key Findings¶
- Language-level modeling invalidates family independence: In controlled experiments where the Chinese ratio was fixed at 5% while swapping Japanese and Spanish proportions, FamilyLaw failed completely (\(R^2 \le 0\)) because both were considered out-of-family, while ShapleyLaw captured the cross-lingual transfer accurately (\(R^2 = 0.9312\)).
- Actual pretraining validates predicted mixture quality: The optimized mixture discovered by ShapleyLaw produced the lowest cross-entropy test loss under actual pretraining (24.108 unweighted, 11.813 normalized), proving that game-theoretic optimization yields superior empirical data mixtures compared to ATLAS (25.353), FamilyLaw (24.749), and Common Crawl heuristic distributions (25.491).
- Pretraining loss reliably drives downstream reasoning: Across multilingual benchmarks (HellaSwag, XWinograd, XStoryCloze), pretraining cross-entropy showed a strong negative correlation with downstream few-shot accuracy (\(|r| > 0.90\), average \(R^2 = 0.933\)). The exponential calibration model \(s = a \cdot \exp(-\ell) + b\) demonstrated that lower pretraining loss directly translates into superior downstream reasoning capability.
- Robust extrapolation to larger scales: The learned scaling curves accurately predicted held-out test losses for models scaled up to \(N = 3.17\text{B}\) parameters, with held-out prediction errors as low as 0.004 (English) and 0.008 (German, French, Japanese).
Highlights & Insights¶
- Game-theoretic reformulation of multilingual learning: Framing pretraining as a cooperative game allows researchers to leverage the century-old, axiomatically proven Shapley value, replacing empirical curve-fitting with principled contribution attribution.
- Efficient \(O(1)\) approximation via In-Run Data Shapley: Replacing the combinatorial \(O(2^K)\) pretraining requirement with step-wise gradient inner product tracking makes game-theoretic data allocation feasible for large-scale production workflows.
- Scale-invariance of transfer matrices: The discovery that normalized Shapley values remain virtually unchanged across model parameter scales (\(r = 0.999\) between 50M and 243M) provides a practical methodology: measure linguistic transfer cheaply on small models, then optimize data mixtures for massive flagship models.
Limitations & Future Work¶
- Upper bound on validated model scale: While validated up to 3.2B parameters, empirical confirmation on 70B+ frontier model scales remains constrained by academic compute budgets.
- Tokenizer sensitivity: Shapley value estimation relies on the chosen tokenizer; differences in vocabulary representation efficiency across low-resource languages could slightly bias gradient inner products.
- Dynamic stage-wise mixtures: Future work could extend ShapleyLaw to dynamic curriculum settings where data mixtures adaptively shift across different pretraining phases to optimize foundational linguistic competence before specialized reasoning.
Related Work & Insights¶
- vs FamilyLaw (He et al., 2025): FamilyLaw aggregates languages into families and assumes zero cross-family transfer, forcing intra-family allocations to rely on crude heuristic smoothing; ShapleyLaw models fine-grained language-level transfer across all language pairs, enabling precise individual mixture optimization.
- vs ATLAS (Longpre et al., 2026): ATLAS uses ad-hoc empirical coefficients to decompose language transfer; ShapleyLaw derives transfer strictly from cooperative game theory, resulting in lower empirical test loss and superior downstream task performance.
- vs DoReMi / ODM / DML (Xie et al., 2023; Shukor et al., 2025; Ye et al., 2025): Prior data mixture optimizers optimize ratios only under fixed compute budgets without scaling law extrapolation; ShapleyLaw provides joint extrapolation across model scale, token budget, and language proportions.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneering integration of cooperative game theory and Shapley values into multilingual scaling laws.]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Spans 10 languages, 8 language families, model sizes from 50M to 3.2B, with end-to-end pretraining and downstream validation.]
- Writing Quality: ⭐⭐⭐⭐⭐ [Mathematically rigorous, lucid narrative flow, and comprehensive ablations covering approximations and normalization.]
- Value: ⭐⭐⭐⭐⭐ [Offers an actionable, computationally efficient blueprint for optimizing pretraining data mixtures in modern multilingual foundation models.]