Skip to content

Routing-Free Mixture of Experts

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://liuyilun2000.github.io/RoutingFreeMoE/
Area: Model Compression
Keywords: Mixture of Experts, Routing-Free Architecture, Adaptive Load Balancing, Activation Sparsity, Expert Parallelism

TL;DR

Routing-Free MoE completely eliminates centralized routers, Softmax, and TopK selection by allowing each expert to autonomously determine its activation via internal low-rank projections and learnable thresholds, combined with a unified dual-axis soft load-balancing framework that unlocks emergent layer-heterogeneous sparsity and barrier-free expert parallelism.

Background & Motivation

In the ongoing scaling of Transformer-based Large Language Models (LLMs), Mixture-of-Experts (MoE) architectures have emerged as an indispensable paradigm for expanding parameter capacity without incurring a proportional surge in floating-point operations. Conventional MoE designs universally depend on centralized external routers that orchestrate token-to-expert dispatch. However, these routing modules suffer from a critical structural information bottleneck: parameterized by only small, low-dimensional linear projections, the external router has several orders of magnitude fewer parameters than the experts it coordinates. Deprived of direct feedback regarding internal expert representations, the router is forced to learn assignment policies via indirect, trial-and-error dot-product scoring against static expert vectors, routinely leading to early training instability, optimization bottlenecks, and suboptimal capacity allocation.

Compounding this structural bottleneck are the rigid inductive biases enforced by centralized routing components. Standard TopK selection imposes a uniform, input-invariant sparsity ratio across every token and layer, completely ignoring the reality that tokens exhibit vast differences in semantic complexity and that experts differ substantially in specialization depth. Simultaneously, Softmax normalization coerces expert scores into a competitive, sum-to-one probability distribution that discards the absolute magnitude of expert responses, artificially suppressing the residual contributions of suitable experts whenever another happens to score slightly higher. In parallel, conventional load-balancing strategies enforce a false dichotomy between Token-Choice (which guarantees per-token compute budgets but leaves expert load unbalanced) and Expert-Choice (which guarantees uniform expert utilization at the cost of variable per-token computational capacity), stifling the model's ability to discover richer, input-dependent compute allocation schemes.

To transcend these structural constraints, MoE architectures require a decentralized, bottom-up formulation where expert activation decisions are internalized directly within the experts themselves rather than dictated by an external arbiter. Core idea: completely eliminate centralized routers, Softmax, TopK, and layer-wise static load balancing, enabling each individual expert to decide its own activation independently via internal low-rank representations and a learnable bias, while optimizing a unified, adaptively scheduled load-balancing loss that continuously interpolates between token- and expert-balancing objectives.

Method

Overall Architecture

Routing-Free MoE (RFMoE) replaces the centralized arbitration pipeline of standard MoE with a bottom-up, autonomous expert execution workflow. When a sequence of hidden states enters an MoE layer, the tokens are directly broadcast to all local expert modules without passing through an external gating network. Each expert independently processes the hidden states using its internal low-rank down-projection matrix, computing an intrinsic activation score based on the projected vector's Euclidean norm and a learnable scalar threshold. A global post-activation threshold then determines discrete execution: tokens that pass the threshold continue through the full SwiGLU feed-forward network, while non-activated tokens terminate immediately with zero redundant computation. During backward optimization, a unified dual-axis auxiliary loss dynamically steers the model toward the target global computational budget while preventing expert collapse.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Input Hidden Sequence<br/>(x in R^{T x D})"] --> D1["Autonomous Low-Rank Scoring and Thresholding<br/>Each expert independently projects xA_gate with bias b_i"]
    D1 --> D2["Dual-Axis Adaptive Load Balancing<br/>Jointly optimizes expert and token distributions with dynamic lambda_t"]
    D2 --> D3["Global Cross-Layer Sparsity and Parallel Execution<br/>Relaxing layer-wise sparsity with non-blocking All-Gather and P2P aggregation"]
    D3 --> Out["Weighted Expert Output Aggregation and Residual Stream<br/>(h = sum G_i(x) E_i(x) + x)"]

Key Designs

1. Autonomous Low-Rank Scoring and Thresholding: Router Capacity Bottleneck and Uniform TopK -> Internal Low-Rank Norm with Learnable Bias -> Decentralized Activation and Zero-Marginal FFN Reuse

To overcome the router's severe capacity deficit and the inflexibility of fixed TopK selection, RFMoE decentralizes activation decisions into the individual experts. Adapting a low-rank gated linear unit FFN structure, each expert \(E_i\) incorporates an internal low-rank projection matrix \(A_{gate,i} \in \mathbb{R}^{D \times r}\) with rank \(r \ll D\). Instead of relying on an external gating vector, the expert computes the Euclidean norm \(\|x A_{gate,i}\|_2\) of its own internal projection as a grounded, self-generated preference score reflecting its compatibility with the input token \(x\). Because the norm is strictly non-negative, each expert is equipped with a learnable scalar bias \(b_i\), producing a rectified activation weight via:

\[G_i(x) = \text{ReLU}\left(\|x A_{gate,i}\|_2 - b_i\right)\]

When the internal projected norm fails to exceed the expert's bias threshold, \(G_i(x)\) evaluates to exactly zero, effectively deactivating the expert for that token. A configurable global post-activation threshold \(\theta\) is then applied to yield the final discrete activation decision \(f_i(x) = \mathbf{1}\{G_i(x) - \theta \ge 0\}\). For activated token-expert pairs, the intermediate projection \(x A_{gate,i}\) is directly reused in the subsequent SwiGLU forward pass, incurring zero marginal overhead for the activation decision. Inactive experts terminate immediately after the lightweight rank-\(r\) projection, saving the dominant portion of FFN computation without external routing intervention.

2. Dual-Axis Adaptive Load Balancing: Mutually Exclusive TC/EC Paradigms and Manual Hyperparameter Tuning -> Unified Differentiable Auxiliary Loss with Dynamic Schedule -> Emergent Joint Resource Allocation

Eliminating TopK removes the hard-coded mechanism that guarantees uniform computational budgets, necessitating a principled objective to maintain activation sparsity and prevent expert collapse. Rather than forcing a rigid choice between Token Choice and Expert Choice, RFMoE introduces a unified auxiliary loss framework that simultaneously optimizes expert balancing and token balancing. Using the pre-threshold score \(G_i(x)\) as a smooth, differentiable proxy \(\tilde{\rho}(E, B)\) for the non-differentiable binary activation \(f_i(x)\), the framework defines two complementary objectives:

\[L_{EB} = \frac{1}{|E|} \sum_{e_i \in E} \left( \frac{1}{|B|} \sum_{x \in B} f_i(x) \right) \left( \frac{1}{|B|} \sum_{x \in B} G_i(x) \right)\]
\[L_{TB} = \frac{1}{|B|} \sum_{x \in B} \left( \frac{1}{|E|} \sum_{e_i \in E} f_i(x) \right) \left( \frac{1}{|E|} \sum_{e_j \in E} G_j(x) \right)\]

Here, \(L_{EB}\) penalizes experts that receive disproportionate token shares, promoting uniform expert utilization across the batch, while \(L_{TB}\) penalizes tokens that activate an excessive or insufficient number of experts, encouraging balanced per-token compute. The two objectives are seamlessly combined via an interpolation coefficient \(\mu \in [0, 1]\) into \(L_{LB} = \mu L_{EB} + (1 - \mu) L_{TB}\). To circumvent brittle manual tuning of the regularization weight, the loss coefficient \(\lambda_t\) is updated dynamically at each step via a sign-based feedback controller: \(\lambda_{t+1} = \lambda_t (1 + \eta)^{\text{sign}(\rho_t(E, B) - \rho_\infty)}\). When empirical activation density exceeds target \(\rho_\infty\), \(\lambda_t\) scales up exponentially to enforce sparsity; when below, it relaxes to allow richer expert participation.

3. Global Cross-Layer Sparsity and Parallel Execution: Rigid Per-Layer Uniformity and Blocking All-to-All Barriers -> Global Aggregated Sparsity Target and Asynchronous P2P Communication -> Natural Representational Specialization and Scalable EP Throughput

Standard MoE enforces identical TopK sparsity independently at every transformer layer, imposing a rigid inductive bias that ignores the distinct representational dynamics of deep networks. RFMoE removes per-layer constraints, enforcing the target density \(\rho_\infty\) only across the global aggregate of all layers. This allows depth-dependent activation patterns to emerge naturally during training: early layers display high activation density and variance (converting discrete tokens to continuous concepts), middle layers stabilize at moderate, low-variance density, and final layers ramp up activation density to synthesize output distributions. Relaxing this per-layer constraint directly improves validation perplexity by over 10 points on scale S.

At deployment under Expert Parallelism (EP), conventional MoE relies on sequential router matmuls, Softmax, and TopK index generation, followed by blocking All-to-All collective communication that forces high synchronization barriers. In contrast, RFMoE begins with a non-blocking All-Gather to broadcast token batches, enabling each device to immediately score its local \(N/M\) experts on incoming chunks. Because expert scoring and execution are decentralized, activated token outputs are dispatched back to originating devices via asynchronous point-to-point messages that accumulate partial sums on the fly, eliminating the centralized barrier and delivering substantial throughput improvements as device counts scale.

Loss & Training

The overall training objective combines the standard causal language modeling cross-entropy loss \(L_{LM}\) with the dynamically weighted auxiliary load-balancing loss:

\[L = L_{LM} + \lambda_t L_{LB}\]

The adaptive regularizer is initialized with \(\lambda_0 = 10^{-10}\) and a feedback step size \(\eta = 0.02\). During initialization, each expert's internal bias is set to a tiny positive constant \(b_i = 10^{-6}\). This gentle warm-up ensures that all experts are active in early training steps, encouraging thorough, collective exploration of the latent representation space. As training proceeds and \(\lambda_t\) scales upward in response to density feedback, sparsity pressure gradually increases, orchestrating a stable phase transition where experts differentiate into specialized roles without suffering from representation collapse, finally settling at the target global density \(\rho_\infty = 1/4\).

Key Experimental Results

Main Results

Routing-Free MoE was implemented on top of the Mixtral architecture (incorporating GQA, RoPE, SwiGLU, and RMSNorm) and trained from scratch on OpenWebText for one epoch across three model scales: S (approx. 92M), M (approx. 290M), and L (approx. 808M). Comparisons were conducted under iso-compute conditions matching total training FLOPs within \(\sim 1\%\). Zero-shot accuracy was evaluated across 9 standard NLP benchmarks: PIQA, HellaSwag, WinoGrande, ARC-Easy, ARC-Challenge, OpenBookQA, QQP, QNLI, and SST-2.

Scale Architecture Params FLOPs Perplexity PPL↓ PIQA HellaS WinoG ARCe ARCc OBQA QQP QNLI SST-2 Avg.↑
S Standard MoE 92.44M 90.93M 31.22 57.56 27.19 51.22 33.42 21.33 24.60 36.82 49.46 49.08 38.96
S AoE (Lv et al.) 93.85M 88.57M 30.00 56.09 27.01 50.20 33.84 21.93 25.00 36.82 49.46 49.08 38.82
S ReMoE (Wang et al.) 92.44M 90.93M 29.60 56.58 26.69 51.62 33.38 22.27 26.00 36.83 49.48 49.08 39.10
S RFMoE (Ours) 95.32M 91.08M 27.42 58.49 27.09 50.59 35.77 21.59 24.40 36.98 49.44 53.56 39.77
M Standard MoE 289.9M 248.0M 25.00 58.32 28.26 52.33 36.20 21.42 24.60 36.85 49.46 49.31 39.64
M RFMoE (Ours) 307.3M 249.2M 22.08 58.92 27.85 49.72 35.27 21.50 26.40 37.06 49.51 57.34 40.40
L Standard MoE 808.4M 608.4M 24.58 59.19 29.37 50.83 36.41 23.29 25.00 37.57 49.28 49.08 40.00
L RFMoE (Ours) 870.6M 613.2M 19.97 58.87 28.27 50.51 37.46 21.67 26.60 39.93 49.86 53.67 40.76

Ablation Study

The following table decomposes the architectural components at scale S, analyzes the impact of the load-balancing interpolation coefficient \(\mu\), and explores the inference-time sensitivity of the global post-activation threshold \(\theta\):

Study Category Configuration Key Settings Params FLOPs Perplexity PPL↓ Throughput / Accuracy Core Characteristic
Architectural Standard MoE Top3/12 baseline 92.44M 90.93M 31.22 38.96% (Avg) Centralized router with TopK/Softmax
Ablation w/o router (AoE) \(r=16\), TopK/Softmax 93.85M 88.57M 30.00 38.82% (Avg) Internal scoring fed to centralized TopK
(Scale S) w/o router (AoE) \(r=32\), TopK/Softmax 95.32M 91.08M 30.31 38.80% (Avg) Larger internal rank, centralized bottleneck persists
w/o TopK/Softmax (ReMoE) Central router, ReLU 92.44M 90.93M 29.60 39.10% (Avg) ReLU gating without TopK, router persists
RFMoE (Ours) \(r=16, \rho_\infty=1/4\) 93.85M 88.57M 28.73 39.42% (Avg) Fully routing-free, autonomous expert gating
RFMoE (Ours) \(r=32, \rho_\infty=1/4\) 95.32M 91.08M 28.33 39.77% (Avg) Optimal quality and compute efficiency
Load Balancing Token-only balance \(\mu = 0.0\) 95.32M 91.08M 28.41 645.7 samples/s Lacks expert balance; activation imbalance
Interpolation Token-leaning \(\mu = 0.2\) 95.32M 91.08M 28.35 648.3 samples/s Mitigates expert load disparities smoothly
(Scale S) Balanced (Default) \(\mu = 0.5\) 95.32M 91.08M 28.34 662.3 samples/s Lowest perplexity and highest throughput
Expert-leaning \(\mu = 0.8\) 95.32M 91.08M 28.38 643.9 samples/s Uniform expert usage, higher input sensitivity
Expert-only balance \(\mu = 1.0\) 95.32M 91.08M 28.43 648.8 samples/s Lacks token compute regularization
Inference Dense execution \(\theta = 0.1, \rho_{eff}=100.0\%\) 95.32M 120.46M — 39.41% (Avg) All experts activated; no numerical instability
Threshold \(\theta\) Moderate sparsity \(\theta = 1.1, \rho_{eff}=29.3\%\) 95.32M 92.97M — 39.98% (Avg) Robust accuracy with reduced activation
(Scale S) Aggressive pruning \(\theta = 1.9, \rho_{eff}=4.3\%\) 95.32M 83.14M — 40.23% (Avg) FLOPs reduced by 31%, accuracy intact

Key Findings

  • Compounding Language Modeling Gains with Scale: RFMoE consistently outperforms standard MoE across all scales, delivering validation perplexity reductions of \(-12.2\%\) at S (31.22 → 27.42), \(-11.7\%\) at M (25.00 → 22.08), and \(-18.8\%\) at L (24.58 → 19.97). A paired statistical test over 27 matched benchmark-scale pairs yields a statistically significant positive gain of \(+0.77\) percentage points (\(t=1.858, p=0.037\), Cohen's \(d=0.358\)).
  • Orthogonal Complementarity of Token and Expert Balancing: The balanced interpolation (\(\mu=0.5\)) achieves the lowest perplexity (28.34) and the highest evaluation throughput (662.3 samples/s). Disabling expert balancing (\(\mu=0.0\)) results in significant expert activation disparities, while disabling token balancing (\(\mu=1.0\)) renders expert activations excessively brittle to input domain shifts, demonstrating that the two objectives address orthogonal failure modes.
  • Emergent Layer-Wise Functional Specialization: Lifting the rigid per-layer sparsity constraint and optimizing toward a global density \(\rho_\infty\) drops perplexity from 39.44 to 28.74 (a massive 10.7-point gain). The model spontaneously develops a tripartite organization: high early activation for token-to-concept translation, moderate middle activation for conceptual processing, and dense late activation for token emission.
  • Superior Training Optimization Stability: Under an aggressive learning rate of \(\alpha = 2 \times 10^{-3}\) at scale S, standard MoE completely diverges (PPL exploding to 214.0), whereas RFMoE trains smoothly to an optimal perplexity of 27.42, underscoring the stabilizing effect of continuous, differentiable activation weights.
  • Barrier-Free Expert Parallel Scaling: On 4-device expert parallelism (\(M=4\)), standard MoE throughput degrades by \(\sim 19\%\) in prefill and \(\sim 21\%\) in decode due to blocking All-to-All dispatches. RFMoE retains \(\ge 97.8\%\) of single-device prefill and \(\ge 95.6\%\) of decode throughput, delivering \(1.19\times\) prefill and \(1.12\times\) decode throughput over standard MoE at \(M=4\).

Highlights & Insights

  • Decentralized Expert Autonomy: RFMoE fundamentally reframes sparse computation: by letting experts score their own affinity using internal low-rank representations, it removes the external router's capacity bottleneck and trial-and-error optimization dynamics.
  • Compute-Free Scoring Reuse: The low-rank gate projection \(x A_{gate,i}\) serves a dual purpose as both the activation affinity indicator and the primary projection for the SwiGLU FFN pass, ensuring that scoring introduces virtually zero redundant arithmetic overhead.
  • Decisive Activation and Smooth Inference-Time Throttling: Because RFMoE trains experts to push affinity scores far from the decision boundary, the global threshold \(\theta\) acts as a stable runtime knob. Scaling activation density from \(100\%\) down to \(4.3\%\) degrades benchmark accuracy by less than 2 points, offering unprecedented elasticity for real-world serving under varying hardware loads.

Limitations & Future Work

  • Scale and Pretraining Budget Scope: Current evaluations are conducted on models trained from scratch up to 0.8B parameters on single-epoch OpenWebText. Scaling behaviors at the 7B–70B parameter frontier and complex domains (e.g., mathematics, coding, and multi-turn reasoning) warrant further empirical validation.
  • Absence of Dense-to-MoE or Pretrained MoE Adaptation: The study exclusively explores training from scratch; converting existing pretrained MoE checkpoints (such as Mixtral or DeepSeekMoE) into RFMoE architectures via continued pretraining remains an open engineering challenge.
  • Future Directions: Investigating the synergy between routing-free architectures and shared expert designs, along with hardware-level asynchronous communication kernel optimizations for non-blocking point-to-point aggregation.
  • vs Standard MoE (Fedus et al., 2022; Shazeer et al., 2017): Standard MoE relies on a centralized linear router with discrete TopK gating and competitive Softmax probabilities, creating capacity bottlenecks and rigid uniform sparsity; RFMoE removes all centralized components, enabling input-adaptive, autonomous expert activations.
  • vs Autonomy-of-Experts (AoE, Lv et al., 2025): While AoE introduces expert-internal low-rank representations for scoring, it feeds the resulting scores back into centralized TopK and Softmax operators; RFMoE achieves true decentralization by replacing TopK/Softmax with per-expert learnable thresholds.
  • vs ReMoE (Wang et al., 2024b): ReMoE replaces TopK and Softmax with ReLU gating but preserves an external centralized router; RFMoE removes the external router entirely, uniting internal representation scoring with rectified activation.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Decisively departs from the decade-long centralized routing convention in MoE, establishing the first fully autonomous, bottom-up expert execution framework.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 3 scales, 9 benchmarks, complete architectural ablations, training stability sweeps, and distributed expert-parallel critical path profiling.
  • Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative, elegant mathematical formulations, and thorough systems-level communication analysis.
  • Value: ⭐⭐⭐⭐⭐ Delivers compelling solutions to the longstanding MoE challenges of routing capacity bottlenecks, discrete optimization instability, and distributed barrier synchronization overheads.