Towards a Comprehensive Scaling Law of Mixture-of-Experts¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Mixture-of-Experts, Scaling Laws, Sparse Activation, Shared Experts, Architecture Design
TL;DR¶
Addressing the failure of classical dense scaling laws in Mixture-of-Experts (MoE) architectures, this work systematically decouples five core factors—data size, total model size, activated model size, active experts, and shared expert ratio—to establish a comprehensive joint MoE scaling law across 450 controlled experiments, deriving theoretically invariant structural optima and efficiency-aware sparsity trajectories.
Background & Motivation¶
The notable reasoning and generalization frontiers achieved by modern Large Language Models (LLMs) are fundamentally anchored in the continuous expansion of parameter capacity and pretraining computational budgets. Classical scaling laws for dense architectures, notably established by Kaplan et al. and the Chinchilla framework, have served as indispensable foundations for predicting model validation loss, balancing compute allocation, and guiding pre-training runs. However, as frontier industry architectures pivot toward Mixture-of-Experts (MoE) to substantially decouple total parameter capacity from per-token inference FLOPs, these dense formulations collapse completely. The core premise of dense laws—that every parameter participates uniformly in every forward pass—is violated by dynamic routing and conditional computation, which introduce complex, multi-dimensional architectural dependencies.
Existing efforts to formulate MoE-specific scaling laws suffer from severe factor omissions and overly simplistic monotonic assumptions. Prior formulations often examine isolated variables such as activated model size or expert granularity in isolation, failing to account for the intricate structural couplings among hyperparameters. More critically, empirical observations demonstrate that structural factors—including the number of activated experts per token, the ratio of shared experts, and the ratio of activated parameters to total parameters—exhibit distinctly non-monotonic, hook-like relationships with validation loss. Standard power-law formulations cannot accommodate performance degradation caused by overly granular expert partitioning or structural distortion from excessively high activation ratios, leading to substantial prediction residuals and offering no principled guidance for architectural design.
To bridge this fundamental gap, this work introduces a rigorous, fine-grained methodology to establish a unified joint MoE scaling law spanning both scale and structural dimensions. Through systematic variable-controlled experimentation, the authors decouple marginal effects, model the non-monotonic dependencies using hook-like and quadratic formulations, and evaluate the joint law across both interpolation and significant extrapolation regimes. Core idea: Decouple MoE scaling behavior into a structural term governed by active and shared experts and a scale term modulated by total and activated parameters, capturing non-monotonic marginal effects to derive a unified joint scaling law alongside closed-form theoretical and efficiency-aware optimal architectural configurations.
Method¶
Overall Architecture¶
The technical pipeline for formulating the joint MoE scaling law decomposes the design space into five governing factors: training dataset size \(D\), total model parameter size \(N\), activated model parameter size \(N_a\), total activated experts per token \(G = n_k + n_s\) (comprising \(n_k\) activated routed experts and \(n_s\) shared experts), and the shared expert ratio \(S = n_s / G\). The research workflow proceeds through a disciplined four-stage pipeline: single-factor marginal effect analysis, hyperparameter cross-factor independence verification, multiplicative joint scaling formulation, and closed-form derivation of optimal configurations under theoretical and practical constraints.
The empirical foundation comprises 450 rigorously controlled pretraining experiments using a standardized transformer backbone on the Dolma V1.7 corpus. The final unified loss formulation couples an architectural structure factor \(\Phi(G, S)\) multiplicatively with an MoE-specific scale modulation factor \(\Psi(N, N_a)\), superimposed onto Chinchilla-style power-law baselines reflecting data and overall parameter bounds.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Five Governing Factors (D, N, Na, G, S)"] --> B["Five-Dimensional Decoupling & Controlled Experiments<br/>450 controlled runs isolating non-monotonic marginal trends"]
B --> C["Non-Monotonic Hook & Shared Structural Modeling<br/>G linear-inverse balance + S quadratic symmetry"]
C --> D["Multiplicative Structure-Scale Joint Scaling Law<br/>Φ(G, S) · Ψ(N, Na) joint validation loss formulation"]
D --> E["Closed-Form Optima & Efficiency-Aware Analysis<br/>Analytical G_opt, S_opt, and practical Na/N Pareto frontiers"]
E --> F["Output: Accurate Loss Extrapolation & Industrial MoE Prescriptions"]
Key Designs¶
1. Five-Dimensional Decoupling & Controlled Experiments: Isolating true marginal dynamics from coupled structural side-effects
In standard MoE configurations, naively altering the activated parameter size \(N_a\) or expert count \(G\) inevitably induces confounding changes in the total parameter footprint \(N\) or expert hidden dimensions. To isolate genuine single-factor marginal effects, the authors derive an exact parameter preservation constraint linking the scaling ratio \(u\) of expert dimensions to the scaling ratio \(v\) of routed experts: $\(v = \left(\frac{1}{u} - 1\right) \cdot \frac{S \cdot G + n_e}{n_e \cdot u}\)$ By scaling expert hidden dimension and routed expert counts inversely according to this formulation, the total parameter count \(N\) remains strictly invariant while \(N_a\) varies across controlled ranges. The resulting 450 experimental runs span fitting intervals of \(N \in [133\text{M}, 3.4\text{B}]\) and \(D \in [10\text{B}, 50\text{B}]\), with validation sets pushed to \(N = 30\text{B}\), \(N_a = 6.6\text{B}\), and \(D = 100\text{B}\), providing an uncorrupted empirical corpus for robust regression.
2. Non-Monotonic Hook & Shared Structural Modeling: Capturing granularity and shared capacity limits
Marginal experiments reveal that validation loss varies non-monotonically with both activated parameter size \(N_a\) and activated expert count \(G\). Specifically, holding \(N\) constant while expanding \(N_a\) initially reduces loss according to a power law, but subsequently causes validation loss to rise due to severe structural distortion—the individual expert dimension balloons while the total expert count shrinks, collapsing combinatorial routing diversity. Similarly, increasing \(G\) under constant \(N_a\) fragments expert representation capacity. The authors formalize this behavior via an elegant hook-like function combining linear growth and an inverse penalty: $\(L(G) = e G + \frac{f}{G} + \tau\)$ For the shared expert ratio \(S\), empirical curves exhibit a distinct quadratic bowl: models completely lacking shared experts suffer significant performance degradation, while excessive shared allocation diminishes routing capacity. The marginal loss follows a symmetric parabola \(L(S) = m S^2 + n S + \psi\), where the optimal ratio is independent of dataset scale.
3. Multiplicative Structure-Scale Joint Scaling Law: Unifying five dimensions under Occam's razor
By analyzing hyperparameter trajectories across across-factor regressions, the authors discover that data size \(D\) is strictly decoupled from architectural factors \(G\) and \(S\), while the amplitude of structural perturbations is modulated by parameter scale \(N\) and \(N_a\). Applying Occam's razor to eliminate redundant parameters, the complete five-factor joint MoE scaling law is formalized as: $\(L(N, D, N_a, G, S) = \Phi(G, S) \cdot \Psi(N, N_a) + \frac{a}{N^\alpha} + \frac{b}{D^\beta} + \frac{c}{N_a^\alpha} + \epsilon\)$ where the structural and scale components are defined respectively as: $\(\Phi(G, S) = e G + \frac{f}{G} + m S^2 + n S, \quad \Psi(N, N_a) = \frac{1}{N^\alpha} + \frac{k}{N_a^\alpha} + h \frac{N_a}{N}\)$ Here, the baseline term \(\frac{a}{N^\alpha} + \frac{b}{D^\beta} + \frac{c}{N_a^\alpha} + \epsilon\) captures primary power-law capacity limits, while the multiplicative term \(\Phi(G, S) \cdot \Psi(N, N_a)\) precisely models how expert granularity and shared allocations scale in efficacy across varying model sizes.
4. Closed-Form Optima & Efficiency-Aware Analysis: Bridging theoretical upper bounds with deployment constraints
Setting the first-order partial derivatives of \(L\) with respect to structural parameters to zero yields analytical, scale-invariant optimal settings. Because \(\Phi(G, S)\) decouples from scale, the optimal activated expert count is fixed at \(G_{\text{opt}} = \sqrt{f/e} \approx 6.78\), independent of \(N\) and \(D\). Similarly, the optimal shared expert ratio is invariant at \(S_{\text{opt}} = -n / (2m) \approx 0.31\), translating directly to an operational recommendation of 1 to 2 shared experts within an active budget of \(G \approx 7\). For the activation ratio \(r = N_a/N\), setting \(\frac{\partial L}{\partial r} = 0\) yields: $\(\left(\frac{N_a}{N}\right)_{\text{optt}} = \frac{1}{N} \left[\frac{\alpha (k \Phi + c)}{h \Phi}\right]^{\frac{1}{\alpha + 1}}\)$ which proves that theoretical loss-optimal MoE models must become increasingly sparse as \(N\) expands. To align with real-world serving throughput and memory constraints, the authors develop an efficiency-aware algorithm with step size \(\Delta N_a = 0.01 N\). Terminating when marginal loss reduction drops below \(\Delta \text{Loss} \le 0.005\) establishes a practical recommended activation ratio bounded tightly around 5%, forming an actionable Pareto operating window \([(N_a/N)_{\text{opte}}, (N_a/N)_{\text{optt}}]\).
Benchmark & Evaluation Protocol¶
All model variations are pretrained on Dolma V1.7 using AdamW (\(\beta_1=0.9, \beta_2=0.95\), weight decay 0.1, clip grad norm 1.0) with sequence length 2048 and a batch size of 2M tokens. Training implements the Warmup-Stable-Decay (WSD) scheduler, allowing identical warmup and stable stages to be shared across varying token budgets \(D\) to minimize compute waste. Validation loss values are sampled during the decay phase using a sliding window of size 10. Nonlinear least-squares optimization is executed via scipy.optimize.curve_fit, accompanied by extensive grid searches across initial values to prevent boundary convergence and guarantee reproducibility.
Key Experimental Results¶
Main Results: Fitting Accuracy Comparison Across MoE Scaling Formulations¶
To ensure rigorous benchmarking, all baseline scaling laws were refitted on the identical 450 experimental data points collected in this study. The evaluation encompasses both within-distribution interpolation points and challenging extrapolation configurations up to 30B total parameters and 100B tokens.
| Scaling Law Formulation | Modeled Factors | Key Structural Characteristics | Mean Validation Loss Error (MAE) | Relative Error Reduction |
|---|---|---|---|---|
| Krajewski et al. (2024) | \(N, D, G\) | Models expert granularity; omits \(N_a\) and \(S\) | 0.0193 | Baseline |
| Abnar et al. (2025) | \(N, D, S\) (sparsity) | Penalizes inactive ratio via power-laws; omits hook behavior | 0.0170 | -11.9% |
| Ludziejewski et al. (2025) | \(N, N_a, D\) | Incorporates \(N_a\) but ignores expert topology | 0.0143 | -25.9% |
| Base Two-Factor Law \(L(N, D)\) | \(N, D\) | Classical Chinchilla-style dense formulation | 0.0195 | +1.0% |
| Joint MoE Scaling Law (Ours) | \(N, D, N_a, G, S\) | Unified 5D formulation with multiplicative structure-scale coupling | 0.0060 | -68.9% |
Ablation Study: Validation on Mainstream Frontier MoE LLMs¶
To verify the external validity of the derived optimal configurations, the table below compares the theoretical optima and practical efficiency-aware thresholds predicted by our scaling law against the actual architectural specifications of leading production models.
| Model Name | Actual Parameters (\(N_a - N\)) | Actual Activation Ratio (\(N_a/N\)) | Theoretical Optimal Ratio \((N_a/N)_{\text{optt}}\) | Practical Efficiency-Aware Optimal Ratio \((N_a/N)_{\text{opte}}\) (\(\Delta \text{Loss} \le 0.005\)) |
|---|---|---|---|---|
| Hunyuan-A13B | 13B - 80B | 16.25% | 33.16% (26.5B) | 6.00% (4.8B) |
| DeepSeek-V4-Flash | 13B - 284B | 4.58% | 25.99% (73.8B) | 6.00% (17.0B) |
| Hy3 preview | 21B - 295B | 7.12% | 25.80% (75.1B) | 6.00% (17.7B) |
| DeepSeek-V3.1 | 37B - 671B | 5.51% | 22.02% (147.8B) | 5.00% (33.6B) |
| GLM-5.1 | 40B - 744B | 5.38% | 21.59% (160.6B) | 5.00% (37.2B) |
| Kimi-K2.6 | 32B - 1T | 3.20% | 20.40% (204.0B) | 5.00% (50.0B) |
| Ling-2.6-1T | 63B - 1T | 6.30% | 20.40% (204.0B) | 5.00% (50.0B) |
| DeepSeek-V4-Pro | 49B - 1.6T | 3.06% | 18.63% (298.1B) | 5.00% (80.0B) |
Key Findings¶
- Substantial Residual Compression: The proposed joint scaling law cuts average validation loss prediction error to 0.0060, delivering a 68.9% error reduction over prior state-of-the-art formulations. High fitting fidelity is preserved even when extrapolating to 30B total parameter sizes and 100B pretraining tokens.
- Scale Invariance of Optimal Granularity: Both analytical derivations and empirical fittings confirm that the optimal active expert count \(G_{\text{opt}} \approx 6.78\) and optimal shared expert ratio \(S_{\text{opt}} \approx 0.31\) are invariant across model scale and token budget. Setting approximately 7 active experts with 1–2 shared experts represents a universally optimal topology across small and frontier LLMs alike.
- Monotonic Sparsification of Frontier Models: The theoretical loss-optimal activation ratio decreases steadily as total model size expands (falling from 33.16% at 80B to 18.63% at 1.6T). Under practical efficiency constraints where marginal loss gains drop below 0.005, the recommended ratio converges to approximately 5%, directly corroborating the design choices of systems like DeepSeek-V3 and Kimi-K2.
Highlights & Insights¶
- Rigorous Invariant-Decoupled Experimental Design: By deriving and enforcing the exact inverse relation between expert dimension and expert counts, the study eliminates parameter-scale confounders during structural ablations, establishing a gold-standard methodology for MoE architecture research.
- Analytical Formalization of Non-Monotonic Latent Bottlenecks: Moving beyond naive power-law assumptions, the introduction of hook-like and quadratic terms provides a mathematically tractable representation of structural capacity limits caused by over-fragmentation or loss of combinatorial routing diversity.
- Dual-Track Guidance for Production Systems: Rather than offering purely theoretical minima that neglect inference overheads, the paper formalizes both an unconstrained loss-optimal bound and an efficiency-aware Pareto boundary, providing concrete, operational prescriptions for industrial LLM pretraining.
Limitations & Future Work¶
- Scope Limited to Standard Top-K Routing: The empirical investigations primarily focus on classical token-choice Top-K routing with auxiliary load-balancing losses, leaving alternative routing mechanisms (such as Expert Choice, soft routing, or autonomous expert selection) unmodeled.
- Uniform Sequence Lengths and Objective Functions: Experiments were conducted under a fixed 2048-token context length and next-token prediction objective. The impact of ultra-long context windows and post-training reinforcement learning on optimal routing topologies warrants dedicated future investigation.
- Uncoupled Optimization Hyperparameters: Training hyperparameters such as peak learning rate and batch size were fixed to reasonable heuristics; future iterations aim to integrate critical batch sizes and learning rate decay schedules into a grand unified scaling hyper-surface.
Related Work & Insights¶
- vs Krajewski et al. (2024): Krajewski et al. pioneered expert granularity analysis in MoE scaling laws, but their formulation treated active expert counts without decoupling activated parameter size \(N_a\) or modeling shared experts; our formulation cuts validation error by nearly 70% and provides analytical optima.
- vs Abnar et al. (2025): Abnar et al. examined parameter sparsity penalties using monotonic power laws, missing the sharp performance degradation that occurs when granularity exceeds optimal thresholds; our hook-like formulation accurately captures this turning point.
- vs Hoffmann et al. (2022) (Chinchilla): While Chinchilla dictates proportional scaling between model size and data tokens for compute-optimal dense models, this work proves that MoE scaling decouples along two distinct axes: total parameters and activated parameters, with the optimal activation ratio becoming progressively sparser as total compute scales.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ First unified joint scaling law incorporating five MoE factors with explicit modeling of non-monotonic structural boundaries.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ 450 controlled pretraining runs spanning extensive parameter and token configurations with rigorous out-of-distribution validation.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear mathematical derivations, elegant visual illustrations, and seamless transitions between theory and empirical practice.
- Value: ⭐⭐⭐⭐⭐ High-impact architectural roadmap that directly demystifies the structural design choices of modern frontier MoE models.