Skip to content

Complexity-Aware Routing: Detecting and Allocating Dynamic Compute for Efficient Mixture-of-Experts Large Language Models

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Acking646/car
Area: Model Compression
Keywords: Mixture-of-Experts, Dynamic Routing, Complexity-Aware Computing, Shannon Surprisal, Predictive Entropy

TL;DR

To resolve the computational waste and under-fitting in conventional static Top-k MoE models, CAR decouples offline dual-signal complexity perception (Shannon Surprisal and Predictive Entropy) from discrete dynamic expert allocation. Operating under a frozen backbone, it reduces active parameters on Qwen3-30B-A3B by approximately 10% while lifting GSM8K accuracy by 1.57 points over static routing.

Background & Motivation

As Large Language Models scale up, Sparse Mixture-of-Experts (MoE) architectures have emerged as an indispensable paradigm for decoupling total parameter capacity from per-token inference cost. Modern architectures such as Mixtral and DeepSeek-MoE conditionally activate a tiny fraction of total expert sub-networks per token (e.g., standard Top-2 or Top-8 selection), attaining model quality comparable to dense counter-parts with drastically fewer floating-point operations. Nonetheless, standard MoE models persistently rely on a rigid static Top-k routing policy, assigning an identical compute budget to every token regardless of its linguistic information density. Under this rigid uniform policy, simple functional stopwords (such as "the" or "is") receive superfluous computational capacity, whereas information-dense content words and critical reasoning steps remain throttled by an insufficient expert ensemble, suffering from severe under-fitting.

To overcome the inefficiency of static computation, various dynamic routing approaches have been explored recently. Common paradigms involve setting cumulative softmax probability thresholds (such as Top-p routing and MoE-D), introducing learnable null experts to skip computation (AdaMoE), leveraging attention weights as heuristic importance indicators (DA-MoE), or formulating expert selection as a reinforcement learning problem driven by downstream task rewards (Ada-K). However, these methods suffer from intrinsic vulnerabilities: they derive dynamic routing decisions implicitly from noisy internal router confidences, surface-level attention patterns, or brittle task-specific reward engineering. Such routing signals do not reflect the objective processing complexity of the input token, and end-to-end learnable gates or RL policies frequently suffer from routing collapse, optimization instability, and poor cross-task generalization.

To resolve these dilemmas, this paper posits that dynamic expert allocation must be grounded directly in objective, information-theoretic signals of cognitive processing load. By pairing Shannon Surprisal—which measures the localized, post-hoc processing difficulty of an observed token given prior context—with Predictive Entropy—which quantifies the global, a priori uncertainty across the candidate vocabulary space—one can objectively capture token processing demands without relying on subjective internal heuristics. Core idea: introduce the Complexity-Aware Routing (CAR) framework, which completely decouples dual-signal complexity perception (combining Shannon Surprisal and Predictive Entropy) from discrete expert allocation, utilizing a cross-layer recurrent memory module under a frozen backbone to stably supervise adaptive compute budgeting.

Method

Overall Architecture

CAR adopts a Perception-Decision Decoupling strategy to avoid the training instability and non-differentiable gradient hurdles of discrete routing optimization. The entire pipeline proceeds through three distinct phases: first, an offline labeling stage runs a high-capacity frozen teacher model in batch inference to compute token-level Shannon Surprisal and Predictive Entropy, followed by global robust percentile normalization to clip long-tail outliers; second, under a frozen MoE backbone, a lightweight Complexity Perception (CP) module equipped with shared Recurrent Complexity Memory (RCM) is trained via mean squared error regression to accurately predict the dual complexity labels; third, during dynamic inference, the Dynamic Expert Allocator (DEA) aggregates predicted complexity scores into a unified scalar and maps it through a power-law transformation into an adaptive discrete budget \(k_t \in [k_{\min}, k_{\max}]\), directing the native router to activate variable expert ensembles on the fly.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Sequence Tokens"] --> B["Dual-Signal Offline Extraction<br/>Shannon Surprisal S + Predictive Entropy H"]
    B --> C["Robust Percentile Normalization<br/>Clip long-tail extremes into [0, 1]"]
    C --> D["Recurrent Complexity Memory & Perception<br/>RCM state propagation + CP module regression"]
    D --> E["Dynamic Expert Allocator DEA<br/>Probabilistic union aggregation + Power-law mapping"]
    E --> F["Dynamic Expert Routing Execution<br/>Dispatch k_t experts based on token complexity"]

Key Designs

1. Dual-Signal Complexity Extraction and Robust Normalization: Eliminating Heuristic Noise and Outlier Distortion

Existing dynamic routers depend on internal gating logits or attention weights that conflate model uncertainty with token difficulty. CAR bypasses these heuristics by extracting two objective information-theoretic metrics from a frozen teacher model \(M_{\text{teacher}}\): Shannon Surprisal \(S(t_i) = -\log_2 P(t_i | t_{<i})\) to gauge contextual unpredictability, and Predictive Entropy \(H(t_i) = -\sum_{v} P(v | t_{<i}) \log_2 P(v | t_{<i})\) to quantify global next-token ambiguity. Because natural language surprisal and entropy follow heavy-tailed distributions, standard Min-Max scaling compresses informative tokens into a very narrow band. CAR establishes robust empirical clipping percentiles \([S_{\text{robust}}^{\min}, S_{\text{robust}}^{\max}]\) at 5% and 95% over the training corpus:

\[C_S(t_i) = \frac{\text{clip}(S(t_i), S_{\text{robust}}^{\min}, S_{\text{robust}}^{\max}) - S_{\text{robust}}^{\min}}{S_{\text{robust}}^{\max} - S_{\text{robust}}^{\min}}\]

Applying the identical normalization to entropy yields \(C_H(t_i) \in [0, 1]\). This step eliminates extreme outlier distortion, provides stable bounded targets for the downstream regressor, and preserves meaningful variance across medium-to-high difficulty tokens.

2. Recurrent Complexity Memory and Perception: Bridging Cross-Layer Dependencies for Coherent Reasoning

In deep Transformer architectures, shallow layers encode lexical primitives while intermediate and deeper layers resolve semantic synthesis and output generation. A purely isolated, layer-local hidden state lacks the multi-step context necessary to anticipate downstream reasoning difficulty. To model depth-wise computational dynamics, CAR incorporates a cross-layer shared Recurrent Complexity Memory (RCM) across all layers. At each layer \(l\), the hidden representation \(h_t^l\) is projected via a layer-specific linear transformation \(x_l' = \text{Proj}_l(h_t^l)\) and merged with the previous layer's recurrent memory state \(h_{\text{rcm}}^{l-1}\):

\[h_{\text{rcm}}^l = \text{RCM}(x_l', h_{\text{rcm}}^{l-1})\]

A layer-dedicated two-layer MLP with ReLU activation and a Sigmoid output projection then maps \(h_{\text{rcm}}^l\) to the dual complexity estimates \([\hat{C}_l^S(t), \hat{C}_l^H(t)] \in [0, 1]^2\). By propagating memory states across network depth, RCM preserves multi-token reasoning trajectories and prevents volatile per-layer compute fluctuations on intricate problem-solving tasks.

3. Dynamic Expert Allocator: Probabilistic Aggregation and Controllable Power-Law Budgeting

To translate multi-dimensional continuous complexity estimates into an integer expert count without introducing tuning overhead, the Dynamic Expert Allocator (DEA) combines the predicted surprisal and entropy via a probabilistic union:

\[C_{\text{eff}}(t) = 1 - (1 - \hat{C}_l^S(t))(1 - \hat{C}_l^H(t))\]

This formulation guarantees that if a token exhibits elevated difficulty in either localized unexpectedness or global predictive dispersion, its effective complexity score \(C_{\text{eff}}(t)\) increases accordingly. Next, DEA applies a power-law mapping parameterized by curvature exponent \(\gamma\) to determine a tentative budget, followed by bounding within predetermined hardware bounds:

\[\tilde{k}_t = \text{round}\left(k_{\min} + (C_{\text{eff}}(t))^\gamma \cdot (k_{\max} - k_{\min})\right), \quad k_t = \text{clamp}(\tilde{k}_t, k_{\min}, k_{\max})\]

Here, \(k_{\min}\) guarantees baseline model sanity, while \(k_{\max}\) matches the backbone's native capacity. The parameter \(\gamma\) serves as a runtime control lever: Performance Mode (\(0 < \gamma < 1\)) allocates more experts to moderate-complexity tokens; Balanced Mode (\(\gamma = 1\), default) enforces linear proportional scaling; and Efficiency Mode (\(\gamma > 1\)) reserves peak capacity strictly for the most complex tokens, enabling compute adjustments without retraining.

Loss & Training

CAR adopts a parameter-isolated predictor alignment strategy. During alignment, the entire MoE backbone \(\Theta_{\text{MoE}}\) (including multi-head attention weights, feed-forward experts, and pre-trained router gates) remains completely frozen. Optimization updates solely the shared RCM weights \(\Theta_{\text{RCM}}\) and layer-specific CP MLPs \(\Theta_{\text{CP}}\) via supervised regression, minimizing the joint Mean Squared Error (MSE) over valid tokens:

\[\mathcal{L}_{\text{complexity}} = \frac{1}{N_{\text{valid}}} \sum_{i=1}^{B \times L} M_i \cdot \left(\|\hat{C}_l^S(t_i) - C_S(t_i)\|^2 + \|\hat{C}_l^H(t_i) - C_H(t_i)\|^2\right)\]

Training is performed on a compact mixture of approximately 12K samples (10K sentences from RedPajama, 2K reasoning paths from GSM8K, and 200 samples from MMLU). Using the AdamW optimizer (learning rate 1e-3, batch size 4, gradient accumulation of 2 steps), alignment finishes in a single epoch. On Qwen3-30B-A3B, trainable parameters comprise only 158.7M (0.52% of total parameters) and converge in under 3.0 GPU hours, avoiding the representational collapse common in end-to-end joint training.

Key Experimental Results

Main Results

The primary evaluation is conducted on the Qwen3-30B-A3B backbone (30.6B total parameters, 3.0B active parameters in native Top-8 mode, 128 routed experts), measuring zero-shot accuracy across 9 benchmarks alongside average activated parameter counts.

Methods Active Params MMLU GSM8K ARC-C ARC-E Wino TruthfulQA CSQA WMT24++ BBH
Top-k (k=7) (Baseline) 2.66B 85.53 94.54 91.89 94.43 67.29 74.42 81.84 79.62 58.13
Top-k (k=8) (Baseline) 3.00B 85.53 95.70 91.99 94.24 68.07 74.42 82.03 80.68 59.26
DA-MoE 2.87B 58.33 60.13 72.66 85.94 64.60 14.84 81.81 79.42 42.34
AdaMoE 2.68B 85.51 96.24 91.92 94.23 67.85 74.25 81.90 80.54 58.24
MoE-D (\(p=0.35\)) 2.62B 85.75 94.32 91.70 94.34 68.85 73.93 81.84 80.48 57.21
MoE-D (\(p=0.5\)) 2.84B 85.97 95.13 92.19 94.53 68.64 73.93 82.42 81.22 58.41
Ada-K 2.67B 86.20 95.47 91.97 94.41 67.95 74.41 82.05 81.12 58.32
CAR (\(k_{\min}=6\)) (Ours) 2.68B 86.41 97.27 91.89 94.14 68.97 74.71 82.32 81.21 57.41

Evaluation on DeepSeek-MoE-16B (16.4B total parameters, 64 routed experts, baseline Top-6 with 2.80B active parameters) guided by DeepSeek-LLM-67B-Chat: - Static Top-6 baseline: 2.80B active parameters, 44.72 MMLU, 52.22 ARC-C, 79.84 HellaSwag, 16.45 GSM8K, 31.07 TruthfulQA. - CAR: reduces active parameters to 2.31B (saving 17.5% compute) while achieving 46.20 MMLU (+1.48), 58.50 ARC-C (+6.28), 81.78 HellaSwag (+1.94), 16.58 GSM8K, and 32.08 TruthfulQA.

Ablation Study

Component ablations and complexity signal source comparisons on the Qwen3-30B-A3B backbone:

Config MMLU GSM8K ARC-C TruthfulQA Active Params Note
CAR (Full Model) 86.41 97.27 91.89 74.71 2.68B Full dual-signal decoupled architecture
Random Signal 84.85 93.10 91.24 73.85 2.70B Random assignment confirms necessity of difficulty perception
w/o Surprisal 85.44 94.15 92.19 74.17 2.70B Dropping surprisal hurts GSM8K by -3.12, proving local grounding role
w/o Entropy 85.92 96.12 91.95 74.52 2.69B Removing entropy degrades global uncertainty guidance
w/o RCM 86.52 96.35 91.68 74.45 2.68B Layer-wise isolation harms multi-step reasoning continuity (GSM8K -0.92)
w/o Norm. 86.38 96.84 91.97 74.65 2.64B Standard Min-Max causes performance degradation on outliers
w/o DEA 85.11 94.85 92.28 74.31 2.71B Fixed threshold binning damages continuous budget flexibility
w/o Align. 86.18 95.33 92.09 74.61 3.00B Unaligned random weights fail to restrict active capacity
E2E Joint 85.09 96.48 91.31 73.32 2.70B Joint backbone tuning triggers internal representation degradation
Gating Entropy 85.18 95.20 91.12 74.81 2.55B Router ambiguity proxy causes under-fitting on reasoning
Attention Entropy 85.64 96.82 91.47 73.52 2.78B Noisy attention heuristic inflates compute with degraded accuracy

Key Findings

  • Shannon Surprisal is the Primary Driver for Complex Reasoning: Ablation reveals that removing Shannon Surprisal triggers a severe drop on GSM8K (97.27 down to 94.15), validating that abrupt contextual unpredictability pinpoints pivotal mathematical derivation and reasoning steps where additional expert capacity is required.
  • Linguistic and Part-of-Speech Stratification: Layer-wise analysis across Part-of-Speech tags indicates that Nouns and Verbs (semantic carriers) consistently activate over 6.7 experts in deep layers, whereas Conjunctions and Punctuation require fewer than 6.5 experts. CAR acts as an automated semantic allocator, economizing compute on predictable syntactic scaffolding.
  • Decoupled Alignment Preserves Expert Balance Without Collapse: While end-to-end joint training severely damages reasoning (TruthfulQA plummets to 73.32), CAR's isolated parameter alignment maintains a healthy, uniform expert load profile identical to the pre-trained base model, avoiding expert starvation or monopolization.

Highlights & Insights

  • Objective Information-Theoretic Signals Over Internal Heuristics: While previous methods rely on gating probabilities or attention entropy—which primarily measure router ambiguity or noisy attention dispersion—CAR anchors compute allocation on Shannon Surprisal and Predictive Entropy, directly modeling true linguistic processing complexity.
  • Perception-Decision Decoupling for Fast, Stable Adaptation: By formulating complexity estimation as a continuous supervised regression problem and mapping to expert counts via a non-linear allocator, CAR sidesteps non-differentiable discrete routing gradients. It converges in a single epoch (< 3.0 GPU hours) with only 0.52% trainable parameters.
  • Zero-Retraining Dynamic Efficiency Tuning: The DEA power-law exponent \(\gamma\) functions as an agile runtime knob. Practitioners can dynamically toggle between Performance Mode (\(\gamma < 1\)), Balanced Mode (\(\gamma = 1\)), and extreme Efficiency Mode (\(\gamma > 1\)) across diverse edge-to-cloud deployment targets without modifying network weights.

Limitations & Future Work

  • Irregular Dispatch Latency Overhead in Practical Hardware: Fine-grained micro-benchmarking on an NVIDIA A100 shows that the CP predictor introduces negligible overhead (16.63 ms, 0.5% of latency). However, variable per-token expert activation disrupts static fixed-shape batching and mature CUDA dispatch kernels, leading to an inference throughput of 1.47 samples/s versus 2.32 samples/s for static Top-8.
  • Validation Confined to Post-Training Alignment: CAR has currently been verified as an isolated post-training alignment stage on MoE models up to 30B parameters, rather than incorporated into full-parameter pre-training from scratch.
  • Promising Research Directions: Developing ragged-batching CUDA kernels and communication-fused dispatch primitives tailored for variable-k expert routing, alongside exploring distillation techniques to internalize complexity perception directly into the backbone representations.
  • vs DA-MoE / MoE-D: DA-MoE and MoE-D utilize attention heuristics or cumulative probability cutoffs, which often lead to erratic token allocation or catastrophic accuracy drops on complex reasoning tasks; CAR anchors routing on external information theory, simultaneously improving reasoning performance while trimming redundant compute.
  • vs Ada-K Routing: Ada-K employs reinforcement learning with task-dependent downstream rewards, which is susceptible to training collapse and reward overfitting; CAR employs task-agnostic cognitive signals, demonstrating superior out-of-domain transfer and cross-benchmark robustness.
  • vs AdaMoE / MoE++: Rather than inserting static dummy computation through null experts, CAR adaptively resizes the active expert pool \(k_t \in [k_{\min}, k_{\max}]\), maintaining full expert capacity for challenging tokens while sparsely budgeting predictable ones.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Grounding dynamic MoE compute in Shannon Surprisal and Predictive Entropy via a decoupled architecture is conceptually elegant and principled.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation spanning 9 diverse benchmarks, teacher scaling studies (0.6B to 30B), latency profiling, and layer/POS activations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Rigorously motivated, theoretically coherent, and accompanied by transparent architectural explanations.
  • Value: ⭐⭐⭐⭐⭐ Provides an efficient, data-frugal, and highly deployable blueprint for mitigating the pervasive compute waste of static Top-k MoE inference.