Skip to content

Complexity-Aware Routing: Detecting and Allocating Dynamic Compute for Efficient Mixture-of-Experts Large Language Models

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Acking646/car
Area: LLM (Other)
Keywords: Mixture of Experts, Dynamic Routing, Computational Efficiency, Shannon Surprisal, Predictive Entropy

TL;DR

CAR introduces an information-theoretic dual-signal framework combining Shannon Surprisal and Predictive Entropy within a decoupled perception-decision architecture, enabling dynamic expert allocation that cuts active parameters by ~10% while outperforming static Top-k and existing dynamic routing baselines.

Background & Motivation

Sparse Mixture-of-Experts (MoE) architectures have established themselves as an indispensable design paradigm for scaling the parameter capacity of large language models while keeping inference FLOPs manageable. By conditionally routing each token through a lightweight gating network to activate only a sparse subset of feed-forward network experts (e.g., Top-2 or Top-8), MoEs decouple parameter scale from single-token computation cost. However, prevailing state-of-the-art MoE architectures (such as Switch Transformer, GShard, Mixtral, DeepSeek-MoE, and Qwen3-MoE) universally rely on a static Top-k routing policy. This uniform compute allocation rigidly enforces an identical expert budget across all tokens, completely ignoring the vast heterogeneity in linguistic information density across natural language: common syntactic scaffolding, function words, and punctuation require negligible processing capacity, whereas semantically dense nouns, specialized technical terms, or pivotal reasoning transitions demand coordinated consultation across diverse expert specializations. Consequently, static Top-k routing incurs severe computational waste on trivial tokens while risking representational under-fitting on genuinely difficult ones.

To overcome the inherent rigidity of fixed computational budgets, recent literature has actively explored dynamic routing mechanisms. Representative heuristics include DA-MoE, which derives compute allocation from attention weight extrema, and MoE-D, which applies cumulative softmax probability thresholds (Top-p). Other paradigms explore structural sparsity, such as AdaMoE and MoE++ introducing zero-computation null experts, or Ada-K, which casts dynamic expert selection as a reinforcement learning problem driven by downstream task rewards. Nevertheless, these dynamic alternatives remain plagued by fundamental flaws: internal router confidences and self-attention heuristics are noisy proxies that conflate decision ambiguity with true linguistic difficulty; reward-driven policy optimizations suffer from acute training instability and task-specific overfitting; and end-to-end joint optimization across the router and backbone frequently triggers catastrophic routing collapse, starving unselected experts and degenerating back into dense or homogeneous processing.

Recognizing that internal model heuristics are fragile proxies for difficulty, CAR takes a fundamentally distinct path rooted in cognitive linguistics and classical information theory. Empirical psycholinguistics demonstrates that human cognitive processing effort correlates directly with conditional surprisal, while the dispersion of predictive probability distributions captures global uncertainty over forward continuations. CAR formalizes token processing difficulty through an objective dual-signal objective: localized Shannon Surprisal and global Predictive Entropy. To sidestep the optimization instability of discrete routing decisions, CAR adopts a strict Perception-Decision Decoupling strategy: the entire MoE backbone remains completely frozen to preserve acquired linguistic knowledge, while a lightweight Recurrent Complexity Memory (RCM) and Complexity Perception (CP) module are trained via supervised regression on teacher-derived signals, leaving a Dynamic Expert Allocator (DEA) to smoothly map continuous difficulty to a discrete expert count \(k\). The core idea is to decouple continuous difficulty perception from discrete expert allocation, supervising a lightweight cross-layer recurrent predictor with teacher-derived Shannon Surprisal and Predictive Entropy to dynamically modulate expert budgets via continuous power-law mapping according to intrinsic information density.

Method

Overall Architecture

The end-to-end CAR pipeline operates across three decoupled phases: offline dual-signal annotation and robust normalization, parameter-isolated predictor alignment, and online dynamic inference via the Dynamic Expert Allocator (DEA). In the offline phase, a frozen, high-capacity teacher model executes fast forward passes across an unannotated corpus to extract token-level Shannon Surprisal and Predictive Entropy, which are then mapped to \([0, 1]\) via corpus-level percentile clipping. During the alignment phase, the pre-trained MoE backbone (including self-attention layers, expert MLPs, and gate weights) is completely frozen; only the shared Recurrent Complexity Memory (RCM) and layer-specific Complexity Perception (CP) modules are trained under a joint Mean Squared Error (MSE) objective. At inference time, the CP module predicts continuous complexity scores on the fly, which the DEA aggregates via probabilistic union and converts into a discrete expert budget \(k_t \in [k_{\min}, k_{\max}]\) via power-law scaling, dispatching tokens to the top \(k_t\) experts selected by the native router.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Token Sequence<br/>Contextual Tokens"] --> B["Dual-Signal Extraction and Robust Normalization<br/>Teacher extracts Surprisal and Predictive Entropy with percentile clipping"]
    B --> C["Recurrent Complexity Memory and Lightweight CP Predictor<br/>Cross-layer recurrent propagation and two-layer MLP regression"]
    C --> D["Dynamic Expert Allocator with Continuous Power-Law Mapping<br/>Probabilistic aggregation and continuous k budget scaling"]
    D --> E["Parameter-Isolated Predictor Alignment Strategy<br/>Supervised MSE training on frozen backbone to prevent routing collapse"]
    E --> F["Dynamic Sparse Expert Execution<br/>Broad expert ensemble for difficult tokens, minimal budget for easy tokens"]

Key Designs

1. Dual-Signal Extraction and Robust Normalization: Grounding Token Difficulty in Information Theory

Departing from prior dynamic methods that rely on internal router confidence or attention scoresโ€”which conflate routing indecision with linguistic difficultyโ€”CAR establishes an objective difficulty metric grounded in information theory. For an input sequence \(x = (t_1, \ldots, t_L)\) processed by a frozen pre-trained teacher model \(M_{\text{teacher}}\) parameterized by \(\Theta_T\), the localized, post-hoc cognitive demand is captured by Shannon Surprisal: $\(S(t_i) = -\log_2 P(t_i \mid t_{<i}; \Theta_T)\)$ Simultaneously, the global, a priori uncertainty over the next-token vocabulary distribution \(V\) is captured by Predictive Entropy: $\(H(t_i) = -\sum_{v \in V} P(v \mid t_{<i}; \Theta_T) \log_2 P(v \mid t_{<i}; \Theta_T)\)$ While surprisal reflects the immediate processing load of the instantiated token given its preceding context, predictive entropy quantifies the divergence across candidate continuations. Because raw surprisal and entropy exhibit heavy-tailed distributions where standard Min-Max scaling compresses the vast majority of informative tokens into an indistinguishable narrow band, CAR introduces a global percentile-based robust normalization: $\(C_S(t_i) = \frac{\text{clip}(S(t_i), S_{\text{robust}}^{\min}, S_{\text{robust}}^{\max}) - S_{\text{robust}}^{\min}}{S_{\text{robust}}^{\max} - S_{\text{robust}}^{\min}}\)$ Clipping the 5th and 95th percentiles (\(S \in [0.05, 0.95]\)) removes extreme outlier noise, providing normalized targets \(C_S, C_H \in [0, 1]\) that offer superior dynamic range and gradient stability.

2. Recurrent Complexity Memory and Lightweight CP Predictor: Multi-Layer Contextual Perception

To model the progressive evolution of representational difficulty across Transformer depths rather than treating layers in isolation, CAR incorporates a shared Recurrent Complexity Memory (RCM). At layer \(l\), the student backbone's hidden state \(h_l^t\) is first projected into a compact latent space via a layer-specific linear transformation: $\(x'_l = \text{Proj}_l(h_l^t)\)$ The shared RCM unit updates its recurrent state using the projected hidden representation and the contextual memory from the previous layer: $\(h_{\text{rcm}}^l = \text{RCM}(x'_l, h_{\text{rcm}}^{l-1})\)$ A layer-specific two-layer MLP (the CP Predictor) then maps this recurrent state to estimated dual complexity scores, strictly bounded by a terminal Sigmoid function: $\([\hat{C}_S^l(t), \hat{C}_H^l(t)] = \sigma(W_{l2} \cdot \text{ReLU}(W_{l1} \cdot h_{\text{rcm}}^l))\)$ Ablation experiments confirm that a Standard MLP with hidden dimension 1024 outperforms heavier architectures incorporating RMSNorm or Dropout, as the ground-truth complexity manifold is relatively smooth and benefits from unencumbered regression fitting on modest adaptation data.

3. Dynamic Expert Allocator with Continuous Power-Law Mapping: Flexible Compute Scaling without Retraining

To map multidimensional continuous predictions into a discrete expert budget without sensitive manual thresholds, the Dynamic Expert Allocator (DEA) first computes an effective scalar complexity \(C_{\text{eff}}(t)\) via probabilistic union: $\(C_{\text{eff}}(t) = 1 - (1 - \hat{C}_S^l(t))(1 - \hat{C}_H^l(t))\)$ This formulation ensures that an elevated score in either surprisal or predictive entropy reliably elevates the overall compute demand. Next, the DEA translates this scalar into a discrete expert allocation using a controlled power-law mapping: $\(\tilde{k}_t = \text{round}\left( k_{\min} + (C_{\text{eff}}(t))^\gamma \cdot (k_{\max} - k_{\min}) \right)\)$ which is bounded within rigid capacity limits: $\(k_t = \text{clamp}(\tilde{k}_t, k_{\min}, k_{\max})\)$ Here, \(k_{\min}\) guarantees a baseline representation capacity, while \(k_{\max}\) matches the backbone's native Top-k budget. Crucially, the exponent \(\gamma\) allows post-hoc runtime trade-offs between performance and efficiency without retraining: Performance Mode (\(0 < \gamma < 1\), e.g., \(\gamma=0.5\)) generously allocates experts to moderately complex tokens; Balanced Mode (\(\gamma=1.0\)) applies linear scaling; and Efficiency Mode (\(\gamma > 1\), e.g., \(\gamma=1.2\)) aggressively restricts high expert counts to the most challenging tokens, driving average active \(k\) down to 4.49.

4. Parameter-Isolated Predictor Alignment Strategy: Eliminating Routing Collapse and Catastrophic Forgetting

Standard end-to-end dynamic MoE training intertwines discrete router policy gradients with expert parameter updates, often triggering winner-take-all feedback loops where routing collapses onto a tiny fraction of experts. CAR bypasses this failure mode entirely via a strict Parameter Isolation strategy. During the predictor alignment phase, the pre-trained MoE backbone \(\Theta_{\text{MoE}}\) remains frozen. Training exclusively updates the shared memory \(\Theta_{\text{RCM}}\) and layer-specific predictors \(\Theta_{\text{CP}}\) via supervised regression, minimizing the joint masked Mean Squared Error (MSE): $\(\mathcal{L}_{\text{complexity}} = \frac{1}{N_{\text{valid}}} \sum_{i=1}^{B \times L} M_i \left( \|\hat{C}_S^l(t_i) - C_S(t_i)\|^2 + \|\hat{C}_H^l(t_i) - C_H(t_i)\|^2 \right)\)$ where \(M_i\) masks out padding tokens. The trainable parameters constitute a negligible 0.52% of the model (approx. 160M parameters on a 30.6B backbone), requiring only 12K samples and under 3.0 GPU hours to converge. This decoupled alignment establishes a stable routing initialization that completely prevents expert starvation while leaving the pre-trained knowledge base intact.

Loss & Training

During predictor alignment, CAR uses a compact mixed corpus of ~12K samples (10K random sentences from RedPajama, 2K reasoning chains from GSM8K, and ~200 samples from MMLU; evaluation splits are strictly excluded). Training runs for a single epoch using the AdamW optimizer with learning rate \(1 \times 10^{-3}\), batch size 4, and gradient accumulation every 2 steps. The loss function is the masked joint MSE \(\mathcal{L}_{\text{complexity}}\) computed over valid non-padding tokens. Evaluation is conducted under zero-shot deterministic generation with evaluation seed fixed to 42.

Key Experimental Results

Main Results

CAR is evaluated on the Qwen3-30B-A3B backbone (30.6B total parameters, 128 routed experts, activating 8 experts for 3.00B active parameters in native static routing) across nine diverse benchmarks: MMLU (multitask comprehension), GSM8K (multi-step math reasoning), ARC-Challenge and ARC-Easy (scientific reasoning), Winogrande (commonsense coreference), TruthfulQA (factuality and hallucination resistance), CommonsenseQA (CSQA), WMT24++ (machine translation), and Big-Bench Hard (BBH).

Method Type Active Params MMLU GSM8K ARC-C ARC-E Wino Truth CSQA WMT24++ BBH
Top-k (k=7) Static Baseline 2.66B 85.53 94.54 91.89 94.43 67.29 74.42 81.84 79.62 58.13
Top-k (k=8) Static Baseline 3.00B 85.53 95.70 91.99 94.24 68.07 74.42 82.03 80.68 59.26
DA-MoE Dynamic Baseline 2.87B 58.33 60.13 72.66 85.94 64.60 14.84 81.81 79.42 42.34
AdaMoE Dynamic Baseline 2.68B 85.51 96.24 91.92 94.23 67.85 74.25 81.90 80.54 58.24
MoE-D (p=0.35) Dynamic Baseline 2.62B 85.75 94.32 91.70 94.34 68.85 73.93 81.84 80.48 57.21
MoE-D (p=0.5) Dynamic Baseline 2.84B 85.97 95.13 92.19 94.53 68.64 73.93 82.42 81.22 58.41
Ada-K Dynamic Baseline 2.67B 86.20 95.47 91.97 94.41 67.95 74.41 82.05 81.12 58.32
CAR (kmin=6) Ours 2.68B 86.41 97.27 91.89 94.14 68.97 74.71 82.32 81.21 57.41

Generalization is further verified on the DeepSeek-MoE-16B backbone (16.4B total parameters, 64 routed experts, native static Top-6 activating 2.80B parameters), supervised by DeepSeek-LLM-67B-Chat:

Method Active Params MMLU ARC-C HellaSwag GSM8K TruthfulQA Winogrande
Top-6 (Base) 2.80B 44.72 52.22 79.84 16.45 31.07 75.14
AdaMoE 2.72B 45.58 55.35 80.50 15.63 31.48 76.34
MoE-D (p=0.5) 2.64B 45.83 55.17 79.24 15.30 32.27 76.24
Ada-K 2.16B 44.98 53.44 80.24 16.86 31.83 75.92
CAR (Ours) 2.31B 46.20 58.50 81.78 16.58 32.08 76.58

Ablation Study

Systematic ablations on the Qwen3-30B-A3B backbone dissect the impact of perception signals, memory, and training configurations:

Model Configuration Variant Description MMLU GSM8K ARC-C TruthfulQA Active Params (\(N_a\))
CAR (Full) Default configuration (\(k_{\min}=6, \gamma=1\)) 86.41 97.27 91.89 74.71 2.68B
Random Signal Bypasses CP and assigns random difficulty 84.85 93.10 91.24 73.85 2.70B
w/o Surprisal Removes Surprisal, relying solely on Predictive Entropy 85.44 94.15 92.19 74.17 2.70B
w/o Entropy Removes Predictive Entropy, relying solely on Surprisal 85.92 96.12 91.95 74.52 2.69B
w/o RCM Removes cross-layer memory, predicting layer-wise 86.52 96.35 91.68 74.45 2.68B
w/o Norm. Standard Min-Max scaling instead of robust percentile clipping 86.38 96.84 91.97 74.65 2.64B
w/o DEA Discrete threshold bucket mapping instead of power-law DEA 85.11 94.85 92.28 74.31 2.71B
w/o Align. Skips supervised alignment (unaligned random initialization) 86.18 95.33 92.09 74.61 3.00B
E2E Joint End-to-end backprop across backbone, RCM, and CP simultaneously 85.09 96.48 91.31 73.32 2.70B

Key Findings

  • Superiority over Heavy Static Baselines: Operating with only 2.68B active parameters (a 10.7% parameter reduction versus static Top-8), CAR boosts GSM8K accuracy to 97.27 (+1.57 over static Top-8 and +2.73 over static Top-7) and MMLU to 86.41 (+0.88 over static Top-8), while dominating on Winogrande, TruthfulQA, CSQA, and WMT24++.
  • Catastrophic Failure of Attention Heuristics: The attention-driven DA-MoE baseline collapses dramatically on complex tasks (MMLU falls to 58.33, GSM8K to 60.13, and TruthfulQA to 14.84), showing that raw attention weights are too noisy for budget allocation. RL-based Ada-K achieves minor gains but remains bound to reward-specific policies, trailing CAR significantly.
  • Dual Signals Are Strictly Complementary: Dropping surprisal (w/o Surprisal) causes the sharpest drop on multi-step reasoning (GSM8K plunges from 97.27 to 94.15), proving that localized surprisal captures critical reasoning pivots. Removing predictive entropy also degrades performance across the board. Alternative signals like Gating Entropy or Attention Entropy (Table 4) underperform, confirming the validity of information-theoretic targets.
  • Detriment of End-to-End Joint Training: End-to-end joint optimization (E2E Joint) causes severe degradation across reasoning benchmarks (TruthfulQA drops from 74.71 to 73.32; MMLU drops to 85.09). This empirically validates CAR's core architectural thesis: freezing the backbone and isolating the complexity predictor is essential to prevent routing collapse and representational distortion.
  • Linguistic and Layer-wise Stratification: Part-of-Speech analysis shows that information-dense content words (Nouns and Verbs) sustain the highest expert allocations in deep layers (> 6.7 experts), whereas functional scaffolding (Conjunctions and Punctuation) requires < 6.5 experts. Across depths, bottom layers (0-5) universally engage broad expert ensembles for representation encoding, middle layers exhibit sharp divergence based on task complexity, and final layers (45-48) taper down compute as predictive uncertainty resolves.

Highlights & Insights

  • Objective Information-Theoretic Routing: Transcends heuristic thresholds and ungrounded policy rewards by anchoring compute allocation in Shannon Surprisal and Predictive Entropy, providing principled interpretability and cognitive plausibility.
  • Highly Efficient Parameter Isolation: By freezing 99.48% of the model and training only ~160M parameters on 12K samples, CAR achieves convergence in under 3 GPU hours without risking catastrophic forgetting or expert starvation.
  • Practical Real-Time Throughput: Micro-benchmarking on an NVIDIA A100 demonstrates that the CP predictor introduces negligible sequential overhead (16.63 ms, 0.5% of per-token latency). CAR achieves a generative throughput of 1.47 samples/s (over 2ร— faster than Ada-K's 0.65 and MoE-D's 0.60), and scales cleanly to 3.59 tokens/s under 4-GPU expert parallelism.

Limitations & Future Work

  • GPU Kernel Mismatch for Irregular Shapes: Because standard CUDA implementations are heavily optimized for static matrix shapes and fixed Top-k dispatch, variable-k expert activation introduces dispatch overhead (accounting for 97.3% of latency). Developing fused, variable-batch GEMM kernels is necessary to unlock maximum wall-clock speedups.
  • Offline Teacher Annotation Overhead: Extracting complexity targets requires an initial forward pass by a teacher model over the calibration corpus. Although experiments demonstrate graceful scaling down to a tiny 0.6B teacher, an online self-bootstrapping mechanism could further streamline the pipeline.
  • Post-Training Adaptation Setting: CAR currently operates as a post-training adaptation framework on pre-trained MoEs. Extending this information-theoretic dynamic budget mechanism into large-scale pre-training from scratch remains an open avenue.
  • vs DA-MoE (Aghdam et al., 2024): DA-MoE maps maximum self-attention weights to expert counts via heuristic ceiling functions, which suffers from severe contextual noise and causes catastrophic failure on multi-step reasoning; CAR grounds allocation in objective information density, ensuring rock-solid stability.
  • vs MoE-D (Huang et al., 2024): MoE-D uses dynamic cumulative probability thresholds (Top-p) directly from the gating network, which conflates internal routing uncertainty with actual linguistic processing load; CAR decouples difficulty perception from router state.
  • vs Ada-K Routing (Zhao et al., 2025): Ada-K trains an RL policy via PPO against downstream rewards to select \(k\), which incurs unstable training and task-specific overfitting; CAR uses supervised regression on information-theoretic signals, yielding superior reasoning performance at a fraction of the adaptation cost.

Rating

  • Novelty: โญโญโญโญโญ Groundbreaking formulation of MoE compute allocation via information-theoretic dual signals with decoupled perception-decision architecture.
  • Experimental Thoroughness: โญโญโญโญโญ Rigorously tested across 30B and 16B backbones, nine standard benchmarks, extensive ablations, linguistic POS analysis, inference profiling, and multi-GPU expert parallelism.
  • Writing Quality: โญโญโญโญโญ Exceptionally clear narrative, disciplined mathematical formulation, and transparent architectural design.
  • Value: โญโญโญโญโญ Delivers an actionable, highly data-efficient blueprint for deploying adaptive, high-throughput MoE models in compute-constrained production environments.