Skip to content

Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilization

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://aclanthology.org/2026.emnlp-main.1449/
Area: LLM Reasoning
Keywords: Mixture-of-Experts, Elastic inference, Matryoshka representation learning, Dynamic routing, Adaptive computation

TL;DR

This paper introduces Matryoshka Mixture-of-Experts (M-MoE), a training framework that systematically randomizes the per-layer number of active experts during pre-training, forcing the router to learn globally coherent rankings and high expert specialization so that a single checkpoint achieves robust, continuous inference elasticity across varying computational budgets.

Background & Motivation

Mixture-of-Experts (MoE) architectures have emerged as the leading paradigm for scaling large language models to hundreds of billions of parameters without a proportional explosion in per-token FLOPs. Modern foundational MoE models—including DeepSeek-V2 and Qwen3-MoE—predominantly rely on a rigid Top-K routing mechanism where each Transformer block activates a fixed subset of \(k\) expert feed-forward sub-networks (e.g., \(k=8\)) whose outputs are combined via gating scores. However, real-world deployment landscapes are fraught with volatile computational demands, strict latency service level agreements (SLAs), and heterogeneous hardware constraints. Under such dynamic conditions, deploying separate dedicated models for distinct operating points is economically prohibitive, giving rise to an urgent need for elastic inference: the ability of a single model checkpoint to dynamically expand or contract its active computational footprint on the fly.

Unfortunately, empirical investigations into existing open-weights MoE models reveal a counter-intuitive and severe brittleness: naively varying the number of activated experts at test time triggers an immediate performance collapse. When the activation count is truncated below the training setting (e.g., evaluating a \(k=8\) model at \(k=1\) or \(k=2\)), downstream benchmark accuracy plummets precipitously; conversely, activating additional experts beyond \(k=8\) yields negligible performance improvements. This vulnerability stems from the rigid co-adaptation inherent to standard fixed Top-K training. All experts jointly optimize the objective as a fixed-size committee, diffusing essential information indiscriminately across the ensemble. As a result, individual sub-networks never learn to function independently, and the gating router's ranking ability remains localized to the exact top-\(k\) boundary, collapsing as soon as the team size is altered.

Drawing inspiration from Matryoshka Representation Learning (MRL), which enforces a nested, coarse-to-fine structure within high-dimensional dense embeddings, the sparse MoE architecture offers an innate topological potential for elasticity along its width dimension—the number of concurrently active experts. If the top-ranked experts can be encouraged to independently capture foundational, coarse-grained semantics while lower-ranked experts incrementally learn residual, fine-grained details, the model will inherently behave like a nested Russian doll. Core idea: by systematically randomizing the number of activated experts during training—especially via layer-wise independent sampling—the gating network and expert ensemble internalize a robust coarse-to-fine functional hierarchy, enabling a single MoE model to deliver smooth, scalable performance across diverse inference budgets.

Method

Overall Architecture

The core principle of Matryoshka MoE (M-MoE) is to instill nested inductive bias directly into the sparse gating mechanism and expert ensemble during the pre-training phase. In a conventional MoE layer, an input token \(x\) is processed by a router \(G\) to produce gating logits \(s = \text{Softmax}(x \cdot W_g)\), and only the top \(k\) experts \(T\) are executed to compute the blended output \(y = \sum_{i \in T} w_i E_i(x)\). M-MoE dismantles the invariant \(k\) assumption by dynamically drawing the activation budget from a specified range during each forward pass. This setup requires the highest-scoring expert to serve as a reliable standalone predictor, while subsequently selected experts contribute progressive residual refinements.

While global-batch and micro-batch sampling randomize \(k\) uniformly across all layers, the authors discover that decoupling the stochasticity across Transformer layers (Layer-wise M-MoE) achieves the highest efficacy. In this layer-decoupled regime, each of the 56 Transformer layers independently samples its active expert count \(k_l \in [k_{\min}, k_{\max}]\), forcing internal representations at each stage to remain resilient against variable upstream computational widths and driving deeper functional orthogonalization among experts.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Token Representation x"] --> B["Router Gating Score Calculation<br/>s = Softmax(x · Wg)"]
    B --> C["Layer-wise Decoupled Sampling<br/>kl ~ U[kmin, kmax] independently"]
    C --> D["Capacity-Aware Weighted Sampling<br/>P(kl=k) ∝ k^(1/τ) power-law bias"]
    D --> E["Nested Expert Ensemble Residuals<br/>Top-1 coarse foundation + Top-2..k refinements"]
    E --> F["Asymmetric Layer-wise Budget Allocation<br/>Prioritizing early layers for high efficiency"]
    F --> G["Elastic Multi-budget Output y"]

Key Designs

1. Layer-wise Decoupled Sampling: Breaking cross-layer co-adaptation for robust representation resilience

Batch-level randomization forces all layers in the network to share an identical active budget \(k_{\text{dyn}}\), which preserves the rigid assumption that representation width across adjacent stages is homogeneous. To maximize representation robustness, Layer-wise M-MoE delegates the activation decision to each individual Transformer block. For any forward pass, layer \(l\) independently samples its expert budget from a discrete uniform distribution \(k_l \sim \mathcal{U}[k_{\min}, k_{\max}]\) (e.g., \(k_{\min}=1, k_{\max}=6\)). Under this layer-decoupled stochasticity, downstream layers frequently receive representations produced by narrower or wider upstream configurations. Consequently, intermediate features become invariant to varying layer capacity, and the router is compelled to learn a truly generalizable hierarchy. The implementation is remarkably simple: a single line of random sampling in the router forward pass, introducing zero architectural changes and complete compatibility with standard distributed parallelisms.

2. Capacity-Aware Weighted Sampling: Mitigating under-training in high-capacity expert regimes

Uniform sampling treats each expert count with equal probability, yet configurations activating more experts inherently possess higher parameter capacity and greater representational degrees of freedom, demanding more training volume to fully converge. To prevent high-capacity configurations from being under-optimized, the authors introduce a principled, temperature-controlled sampling framework. Defining a monotonic capacity function \(f(k) = \log(k)\), the per-layer activation probability is formulated via a temperature-scaled softmax: $\(P(k_l = k) \propto \exp\left(\frac{f(k)}{\tau}\right) = k^{1/\tau}\)$ When \(\tau \to \infty\), the distribution smoothly recovers uniform sampling; when configured with a finite positive temperature (e.g., \(\tau=2\)), the sampling follows a controlled power-law distribution. This guarantees that while \(k=1\) and \(k=2\) receive sufficient steps to anchor the foundational coarse capabilities, higher-capacity settings (\(k=4, 6\)) receive amplified optimization signals, effectively closing the performance gap between the elastic model and dedicated high-budget specialist baselines.

3. Nested Expert Residual Collaboration: Promoting ranking stability and orthogonal specialization

In conventional MoE models, experts evolve severe functional overlap, resulting in high cosine similarity among router weight vectors and brittle ranking correlations when evaluated away from the target \(k\). M-MoE's nested subset objective mandates that the minimal subset (Top-1) independently optimizes the language modeling loss, establishing a robust core representation. Subsequent experts are forced to learn the residual error uncaptured by the preceding subset, establishing an explicit hierarchical decomposition analogous to principal components. Empirically, the Mean Off-Diagonal Similarity (MODS) of router weight vectors drops substantially, indicating that experts are driven toward mutual orthogonality and occupy distinct functional niches. Furthermore, measuring the Focused Spearman Correlation between router logits under large and small budgets demonstrates that M-MoE preserves a globally consistent expert ordering across all layers.

4. Asymmetric Layer-wise Budget Allocation: Exploiting early-layer sensitivity for non-uniform inference

Because M-MoE decouples expert activation layer by layer, it naturally unlocks heterogeneous inference policies where different depths of the network operate under disparate computational budgets. Partitioning the 56-layer model into four sequential 14-layer quartiles, the authors evaluate performance under asymmetric budget reallocations while fixing the average global expert count (e.g., average \(k=2.5\) or \(k=1.5\)). The empirical findings uncover a striking architectural asymmetry: early Transformer layers are significantly more critical to model performance than later layers. Allocating extra capacity to the first half of the model (e.g., \([3, 3, 2, 2]\)) yields substantial accuracy gains over the uniform baseline, whereas allocating the same compute to the second half (\([2, 2, 3, 3]\)) produces marginal improvements. Conversely, shrinking capacity in early layers (\([1, 1, 2, 2]\)) causes catastrophic degradation. This offers practical guidance for production serving: front-loading compute optimizes accuracy under strict FLOP budgets.

Loss & Training

The main evaluation is conducted in a continual pre-training setup using a 20B parameter base MoE model (56 layers, GQA, 96 experts per layer, activating 0.5B parameters at \(k=1\) and 1.5B at \(k=6\)). Starting from a checkpoint pre-trained on 1T tokens (using a cost-efficient \(k=1\) regime), models are trained for an additional 80B tokens across 8× NVIDIA A100 GPUs using Megatron-LM with pipeline, context, and expert model parallelism. The training uses AdamW (\(\beta_1=0.9, \beta_2=0.95\), weight decay 0.1), peak learning rate \(2.6 \times 10^{-4}\) with a 2,000-step warmup and Weighted Sample Decay (WSD). Evaluations are automated via lm-evaluation-harness across 7 benchmarks: MMLU, ARC Challenge (ARC-C), BoolQ, HellaSwag (HellaS.), LogiQA, OpenBookQA (OBQA), and Winogrande (WinoG.) in strict 5-shot protocols.

Key Experimental Results

Main Results

The table below contrasts specialist Top-k models (each trained on a static \(k\)) against elastic M-MoE variants over the 80B-token continual pre-training setup. Asterisks (*) indicate specialist performance at native training counts.

Training Method Inf. \(k\) MMLU ARC-C BoolQ HellaS. LogiQA OBQA WinoG. Average (Avg)
Top-k (k=1 specialist) 1* 52.01 52.22 67.43 70.38 29.95 40.00 67.64 54.23
Top-k (k=2 specialist) 1 35.54 32.59 65.08 49.09 30.11 32.60 61.17 43.74
Top-k (k=2 specialist) 2* 52.16 52.30 70.73 71.47 31.49 42.00 69.93 55.73
Top-k (k=4 specialist) 1 41.50 40.02 63.46 59.28 28.42 33.60 62.19 46.92
Top-k (k=4 specialist) 2 50.42 51.62 66.33 70.41 29.03 41.20 68.67 53.95
Top-k (k=4 specialist) 4* 53.43 54.44 69.11 72.41 29.95 43.40 69.93 56.10
Top-k (k=6 specialist) 1 35.52 43.17 61.93 58.47 29.34 29.00 60.38 45.40
Top-k (k=6 specialist) 2 48.90 52.17 62.75 69.87 29.11 38.40 66.38 52.51
Top-k (k=6 specialist) 4 53.25 55.03 69.30 72.22 30.57 43.80 69.61 56.25
Top-k (k=6 specialist) 6* 54.32 55.46 70.92 72.88 31.34 43.80 70.48 57.03
Top-p (p=0.1 dynamic) 1 35.54 43.01 63.09 55.52 27.96 30.40 60.06 45.08
Top-p (p=0.1 dynamic) 6 53.06 54.86 69.57 71.89 30.72 41.20 70.48 55.97
M-MoE-global-batch 1 50.78 51.54 67.52 71.01 29.34 41.20 67.96 54.19
M-MoE-global-batch 6 52.28 54.35 70.09 72.25 29.80 42.80 70.32 55.98
M-MoE-micro-batch 1 51.00 53.24 69.69 70.50 30.11 42.20 68.27 55.00
M-MoE-micro-batch 6 52.89 54.78 69.91 72.05 29.80 43.20 70.40 56.15
M-MoE-layer 1 51.69 51.19 69.39 69.96 32.26 41.00 66.77 54.61
M-MoE-layer 2 52.71 53.50 71.44 72.43 31.49 42.40 68.82 56.11
M-MoE-layer 4 53.77 54.95 72.84 72.39 31.64 42.00 69.22 56.69
M-MoE-layer 6 53.56 54.95 72.72 71.70 31.95 43.00 69.14 56.72
M-MoE-layer (\(\tau=2\)) 1 50.62 52.82 67.09 68.83 30.41 38.60 68.43 53.83
M-MoE-layer (\(\tau=2\)) 2 53.42 53.75 67.43 72.41 31.80 43.00 70.40 56.03
M-MoE-layer (\(\tau=2\)) 4 54.33 55.20 68.84 72.17 32.10 43.80 71.74 56.88
M-MoE-layer (\(\tau=2\)) 6 54.14 55.55 69.36 71.79 32.10 43.80 71.82 56.94

Ablation Study & Asymmetric Inference

The table below explores non-uniform layer-wise inference on M-MoE-layer across four sequential 14-layer quartiles, investigating sensitivity to computation placement under constrained average budgets:

Activation Pattern Avg. \(k\) MMLU ARC-C BoolQ HellaS. LogiQA OBQA WinoG. Average (Avg)
Baseline [2, 2, 2, 2] 2.0 52.71 53.50 71.44 72.43 31.49 42.40 68.82 56.11
Front-loaded [3, 3, 2, 2] 2.5 53.35 53.33 71.01 72.92 32.26 43.00 68.98 56.41
Center-loaded [2, 3, 3, 2] 2.5 53.13 53.84 71.44 72.46 30.88 42.40 67.80 55.99
Rear-loaded [2, 2, 3, 3] 2.5 52.90 53.75 71.74 72.25 30.88 41.80 68.98 56.04
Front-reduced [1, 1, 2, 2] 1.5 51.63 53.75 69.88 71.51 31.20 41.40 68.59 55.42
Center-reduced [2, 1, 1, 2] 1.5 52.56 52.05 71.62 71.26 31.95 41.20 68.19 55.55
Rear-reduced [2, 2, 1, 1] 1.5 52.78 52.90 71.07 71.23 31.95 40.60 68.11 55.52

From-scratch pre-training for 80B tokens (Table 6 in the paper) further corroborates that this resilience is fundamental: Top-4 and Top-6 specialist models crash to 35.23 and 36.07 respectively when evaluated at \(k=1\), whereas M-MoE-layer retains an average score of 41.08, exceeding even the dedicated Top-1 specialist baseline (40.96).

Key Findings

  • Elimination of Co-adaptation Brittleness: While the static Top-6 specialist drops by 11.63 points (from 57.03 to 45.40 Avg) when tested at \(k=1\), M-MoE-layer maintains a score of 54.61 at \(k=1\) and 56.72 at \(k=6\), completely decoupling expert utility from a fixed committee size.
  • Superiority of Layer-wise Decoupling: M-MoE-layer outperforms both global-batch and micro-batch variants across all activation widths, demonstrating that layer-independent stochasticity provides the strongest regularization against representation rigidity.
  • Early-Layer Bottleneck: When compute is curtailed to an average \(k=1.5\), preserving early-layer capacity ([2, 2, 1, 1]) preserves an MMLU of 52.78, whereas truncating early layers ([1, 1, 2, 2]) leads to an MMLU of 51.63.
  • Zero Additional Training Overhead: Training throughput for M-MoE-layer is 3,230.2 tokens/GPU/s, essentially matching the fixed Top-4 specialist (3,211.6 tokens/GPU/s), meaning elastic capabilities across \(k \in [1, 6]\) are acquired within the training budget of a single standard model.

Highlights & Insights

  • Extending Matryoshka Learning to Discrete Structural Width: Whereas MRL focuses on truncating dense vector prefixes, M-MoE generalizes the nested residual principle to the routing selection of sparse neural modules, realizing actual FLOPs scaling at inference time.
  • Focused Spearman Correlation for Routing Diagnosis: By computing rank correlations across the union of experts selected by large and small budgets, the paper introduces a rigorous metric to verify that routing decisions form proper nested subsets rather than arbitrary permutations.
  • Demonstration of Expert Orthogonalization via MODS: Mean Off-Diagonal Similarity confirms that M-MoE drives expert routing vectors toward mutual orthogonality, suppressing functional redundancy.

Limitations & Future Work

  • Residual Performance Gap at Peak Budget: While M-MoE matches or outperforms specialists across \(k=1 \dots 4\), a fractional gap remains at \(k=6\) compared to dedicated Top-6 training (56.94 vs. 57.03 Avg), indicating that lower-ranked experts could benefit from more explicit residual supervision.
  • Distributed System Communication Imbalance: Layer-wise heterogeneous expert activations can introduce uneven All-to-All communication patterns in standard distributed environments unless combined with specialized dynamic schedulers.
  • vs Dynamic-MoE / Top-p Routing: Dynamic-MoE determines active experts dynamically based on input token hardness, making the total compute non-deterministic; M-MoE enables operator-controlled elastic budgets where resource limits are determined externally based on system load.
  • vs Matryoshka Representation Learning (MRL): MRL truncates representations in embedding space without reducing internal Transformer feed-forward FLOPs; M-MoE truncates active parameters directly in the compute-heavy expert layers.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneering integration of Matryoshka coarse-to-fine nesting into sparse MoE routing dynamics.]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive verification encompassing continual training, from-scratch training, layer-wise asymmetric inference, MODS specialization, and routing correlation.]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, rigorous mathematical grounding, and well-structured empirical narrative.]
  • Value: ⭐⭐⭐⭐⭐ [Crucial breakthrough for production LLM serving, allowing dynamic adaptation to fluctuating cloud workloads without maintaining multiple model instances.]