Skip to content

Routing-Aware Safety Alignment for Mixture-of-Experts Models

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/JACKPURCELL/RASAMoE-public
Area: LLM Safety
Keywords: Mixture-of-Experts (MoE), Safety Alignment, Alignment Shortcut, Routing Consistency, Safety-Critical Experts

TL;DR

Reveals that standard full-parameter fine-tuning on MoE models induces brittle "alignment shortcuts" via routing evasion rather than repairing unsafe experts, and proposes RASAโ€”an alternating bi-level framework combining selective expert fine-tuning and routing consistency optimization to achieve robust defense without sacrificing utility.

Background & Motivation

Mixture-of-Experts (MoE) architectures have emerged as the foundational design paradigm for frontier language models, scaling parameter capacity while preserving inference efficiency via conditional sparse routing. However, this architectural flexibility introduces an insidious, MoE-specific vulnerability during safety alignment. When standard full-parameter safety fine-tuning (SFT, RLHF, or DPO) is applied to MoE models, optimization algorithms consistently exhibit degenerate shortcut-seeking behaviors: empirical training loss drops not because the parameters responsible for generating harmful content are genuinely repaired, but because the router learns to redirect unsafe inputs toward already-safe benign experts or amplify safe expert activation weights, leaving latent safety-critical experts uncorrected.

The core tension lies in the structural coupling between parameter updates and dynamic routing decisions, which induces severe "alignment shortcuts." Because sparse gating introduces an extra degree of freedom, optimization converges to path-level evasion rather than representation-level remediation. While this yields an illusion of safety on in-distribution test prompts, the defense collapses catastrophically when facing unseen, complex jailbreak prompts (e.g., character flipping, narrative nesting, or persuasive framing) or adaptive white-box perturbations designed to manipulate router logits. Causal routing restoration experiments demonstrate that forcing prefill routing back to the original model's decisions collapses the harmlessness of full-parameter fine-tuned models by 38% to 46%. Moreover, unconstrained full-parameter updates severely distort the routing distribution on benign queries, inducing rampant over-refusal (XsTest over-refusal surges to 50%โ€“97%) and systematically degrading core reasoning benchmarks.

To resolve this dilemma, safety supervision in MoE models must explicitly disentangle expert parameter correction from routing controlโ€”ensuring that alignment gradients directly penetrate the neural substrates of unsafe behavior while routing distributions remain topologically stable against adversarial perturbations. Core idea: propose RASA, a routing-aware expert-level alignment framework that models MoE safety alignment as a bi-level alternating optimization problemโ€”identifying Safety-Critical Experts (SCEs) via Adversarial Activation Discrepancy (AAD) under fixed routing to inject refusal representations, and subsequently regularizing router parameters via forward KL divergence toward safe anchor distributions to eliminate routing-based shortcut solutions.

Method

Overall Architecture

RASA structures safety alignment as a bi-level alternating optimization solved via block-coordinate descent across two complementary phases. The input consists of paired mixed batches comprising benign anchor prompts and their corresponding adversarial jailbreak variants sharing identical underlying malicious intent. In the inner optimization phase, the framework dynamically identifies Safety-Critical Experts by measuring activation frequency discrepancies between safe and adversarial contexts, freezes all routing gates and non-critical experts, and selectively fine-tunes the identified experts to internalize diverse refusal representations. In the outer optimization phase, all expert parameters are frozen, and a double-forward routing strategy optimizes the router parameters via a forward KL divergence loss, aligning the adversarial routing distribution with the safe anchor routing reference.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Mixed Batch Input<br/>Paired safe anchor and adversarial prompts"] --> B["Adversarial Activation Discrepancy Identification<br/>Gating activation difference isolates Top 10-15% safety-critical experts"]
    B --> C["Selective Expert Fine-tuning<br/>Freeze router parameters to inject diverse refusal representations into SCEs"]
    C --> D["Double-Forward Routing Consistency Alignment<br/>Freeze expert parameters to align adversarial routing distribution via KL divergence"]
    D --> E["Aligned MoE Model Output<br/>Robust jailbreak defense with minimal over-refusal"]

Key Designs

1. Adversarial Activation Discrepancy Identification: Pinpointing Safety-Critical Experts

Standard full-parameter fine-tuning cannot differentiate which experts are causally driving harmful outputs, while traditional gradient attribution techniques (such as Integrated Gradients, IG) require 32 forward-backward passes per sample (~1.5 minutes on an H100 GPU), making online identification computationally prohibitive. To locate jailbreak-sensitive experts with negligible overhead, RASA constructs paired batches \(B = B_{anchor} \cup B_{adv}\), where \(B_{anchor}\) denotes original malicious prompts for which the model currently outputs a safe refusal, and \(B_{adv}\) denotes jailbreak-modified variants of the same intents that successfully bypass safety filters. For each expert \(e \in \{1, \dots, E\}\) at layer \(l \in \{1, \dots, L\}\), RASA computes its Adversarial Activation Discrepancy (AAD):

\[\Delta_{l,e} = P(\text{active}_{l,e} \mid B_{adv}) - P(\text{active}_{l,e} \mid B_{anchor})\]

where the empirical activation probability is defined as \(P(\text{active}_{l,e} \mid X) = \frac{1}{|X|} \sum_{x \in X} H_{l,e}(x)\), with \(H_{l,e}(x) \in \{0, 1\}\) being the binary indicator of whether expert \((l, e)\) is selected by top-\(k\) routing for input \(x\). Experts exceeding a discrepancy threshold \(\tau\) (filtering the top 10%โ€“15% per layer) form the operational Safety-Critical Expert set \(E_{crit}^{(b)} = \{(l, e) \mid \Delta_{l,e} > \tau\}\). AAD requires only a single forward pass (~2.4 seconds, a 40x speedup over IG) yet exhibits a 0.58โ€“0.66 Jaccard overlap with IG-ranked experts (9x higher than random selection). Mechanistic analysis reveals that 86%โ€“96% of selected SCEs concentrate in early layers (L0โ€“L3), directly reflecting that adversarial prompt perturbations alter routing flow at the very bottom of the Transformer stack.

2. Selective Expert Fine-Tuning: Decoupling Routing to Directly Repair Expert Weights

Jointly updating router and expert parameters allows the optimizer to minimize training loss by deflecting tokens away from toxic experts rather than repairing them. RASA resolves this in the inner optimization by freezing all router parameters \(\Phi\) and non-critical experts \(\Theta \setminus \Theta_{crit}\), restricting gradient updates strictly to the identified subset \(\Theta_{crit} = \{\theta_{l,e} \mid (l, e) \in E_{crit}^{(b)}\}\). To avoid overfitting to rigid, repetitive refusal strings (such as "I am sorry, but I cannot..."), an external model \(G_{ref}\) generates semantically rich, diverse safety-aligned refusal targets \(y_{ref}\). The inner objective optimizes standard token cross-entropy loss over adversarial prompts:

\[\mathcal{L}_{\text{SCE-FT}}(x, y_{ref}; \Theta_{crit}, \Phi) = - \sum_{t} \log P(y_t \mid y_{<t}, x; \Theta_{crit}, \Phi)\]

Directly injecting refusal knowledge into expert representations prevents the model from relying on path evasion. Causal necessity experiments demonstrate that restoring only the AAD-selected SCE parameters to their unaligned states causes model harmlessness to crash from 1.00 to 0.31; conversely, causal sufficiency tests show that transplanting repaired SCE parameters into an unaligned base model increases harmlessness from 0.10 to 0.74, verifying that the selected experts causally govern the safety capability.

3. Double-Forward Routing Consistency Alignment: Suppressing Evasion and Preserving Utility

Repairing expert parameters alone is insufficient for two reasons: residual, weakly implicated experts remain uncorrected and vulnerable to adaptive routing perturbations, and parameter changes in repaired experts distort gating logit statistics, inadvertently driving benign inputs toward refusal-heavy experts. In the outer optimization phase, all expert parameters \(\Theta\) are frozen, and a double-forward routing strategy optimizes the router parameters \(\Phi\). An anchor forward pass without gradients computes the mean routing reference induced by safe anchor prompts:

\[\bar{p}_{anchor}^{(l)} = \text{sg}\left(\frac{1}{|B_{anchor}|}\sum_{x \in B_{anchor}} \text{softmax}(z_l(x))\right)\]

where \(\text{sg}[\cdot]\) denotes the stop-gradient operator and \(z_l(x)\) are router logits at layer \(l\). A second forward pass computes the routing distribution for adversarial prompts with gradients enabled: \(\bar{p}_{adv}^{(l)}(\Phi) = \frac{1}{|B_{adv}|}\sum_{x \in B_{adv}} \text{softmax}(z_l(x; \Phi))\). The router parameters are then optimized using the forward KL divergence:

\[\mathcal{L}_{router}(\Phi) = \frac{1}{L} \sum_{l=1}^L D_{\text{KL}}\left(\bar{p}_{anchor}^{(l)} \parallel \bar{p}_{adv}^{(l)}(\Phi)\right)\]

Crucially, this objective does not prevent adversarial tokens from routing to repaired SCEs (their activation rate remains virtually identical to the original model); rather, it penalizes anomalous routing shifts, anchoring adversarial routing to the safe topological manifold. This simultaneously seals adaptive routing evasion channels and stabilizes gating behavior for benign queries.

Loss & Training

RASA follows an alternating optimization schedule of \(N=3\) training rounds. Each round adopts a balanced \(1:1\) ratio between expert fine-tuning and router consistency epochs (\(E_{sce-ft} = 1, E_{rt} = 1\)). The learning rate is set to \(5 \times 10^{-5}\) for expert parameters and \(1 \times 10^{-4}\) for router parameters, with a batch size of 16 and a maximum generation length of 128 tokens. Ablation studies confirm that the \(1:1\) alternating schedule strictly outperforms joint simultaneous optimization (which suffers from conflicting gradient updates) as well as unbalanced schedules, achieving the optimal Pareto trade-off between robustness, capability retention, and over-refusal control.

Key Experimental Results

Main Results

Evaluation spans three representative MoE architectures from 1B to 47B parameters: OLMoE-1B-7B (64 experts/layer, top-8 routing), Qwen3-30B-A3B (128 experts/layer, top-8 routing), and Mixtral-8x7B (8 experts/layer, top-2 routing). Benchmarks evaluate harmlessness rate against three primary jailbreak vectors (FlipAttack, DeepInception, Persuasion) and a mixed setting (judged by GPT-4o-mini and verified by Claude 3.5 Sonnet), general capabilities (MMLU, GSM8K, TruthfulQA, MATH-500, GPQA), and over-refusal on XsTest.

Model Alignment / Attack Method Flip Harm. (โ†‘) Deepin Harm. (โ†‘) Pers Harm. (โ†‘) MMLU GSM8K TruthfulQA MATH GPQA XsTest Over-Refusal (โ†“)
OLMoE Original Baseline (Origin) 0.47 0.10 0.24 0.38 0.62 0.73 0.27 0.25 0.17
OLMoE RASA (FlipAttack aligned) 1.00 0.18 0.20 0.39 0.66 0.74 0.29 0.28 0.12
OLMoE RASA (DeepInception aligned) 0.38 1.00 0.16 0.42 0.64 0.73 0.26 0.24 0.12
OLMoE RASA (Persuasion aligned) 0.39 0.58 1.00 0.38 0.64 0.71 0.25 0.26 0.24
OLMoE RASA (Mixed-Attack aligned) 1.00 1.00 0.98 0.42 0.71 0.72 0.28 0.27 0.24
OLMoE SteerMoE (Inference steering) 0.50 0.13 0.27 0.38 0.62 0.73 0.27 0.25 0.17
OLMoE SafeMoE-FT (Mixed routing regularized) 0.53 0.18 0.30 0.34 0.68 0.75 0.27 0.25 0.22
Qwen Original Baseline (Origin) 0.04 0.04 0.09 0.74 0.94 0.80 0.76 0.49 0.05
Qwen RASA (FlipAttack aligned) 1.00 0.50 0.18 0.79 0.94 0.86 0.79 0.51 0.12
Qwen RASA (DeepInception aligned) 0.02 1.00 0.19 0.80 0.92 0.87 0.75 0.47 0.09
Qwen RASA (Persuasion aligned) 0.00 0.06 1.00 0.80 0.91 0.84 0.76 0.49 0.16
Qwen RASA (Mixed-Attack aligned) 1.00 1.00 1.00 0.81 0.92 0.85 0.77 0.50 0.24
Qwen SteerMoE (Inference steering) 0.07 0.08 0.11 0.74 0.94 0.80 0.76 0.49 0.06
Qwen SafeMoE-FT (Mixed routing regularized) 0.12 0.08 0.13 0.75 0.92 0.85 0.77 0.48 0.23

Ablation Study

The causal underpinnings of the alignment shortcut and the individual necessity of RASA's modules are validated via two core experimental protocols: causal routing restoration and component ablations.

Dimension / Setting Configuration / Attack Vector Normal Routing Harm. Restored Routing Harm. Drop \(\Delta\) XsTest Over-Refusal Performance Notes
Routing Restoration (OLMoE) Full-Param FT (FlipAttack) 0.99 0.61 -0.38 0.77 Safety collapsed when original routing restored
Routing Restoration (OLMoE) RASA (FlipAttack) 1.00 0.94 -0.06 0.12 Safety robustly preserved at parameter level
Routing Restoration (OLMoE) Full-Param FT (DeepInception) 1.00 0.54 -0.46 0.98 Over half of safety gains lost under original routing
Routing Restoration (OLMoE) RASA (DeepInception) 1.00 0.96 -0.04 0.12 Immune to prefill routing reset
Routing Restoration (OLMoE) Full-Param FT (Persuasion) 1.00 0.55 -0.45 0.67 Severe shortcut reliance across attack vectors
Routing Restoration (OLMoE) RASA (Persuasion) 1.00 0.96 -0.04 0.24 True internal repair of unsafe parameters
Component Ablation (OLMoE) Expert-Only (Skip-Router) 1.00 (Flip) โ€” โ€” +0.12 Benign inputs drift to refusal SCEs; higher over-refusal
Component Ablation (OLMoE) Router-Only (Skip-Expert) 0.18 (Flip) โ€” โ€” +0.05 -0.82 harmlessness drop; fails under alternate routing
Component Ablation (OLMoE) Joint Optimization (Joint-Opt) 0.94 (Flip) โ€” โ€” +0.11 Gradient interference degrades cross-attack transfer
Component Ablation (OLMoE) Random Expert Selection 1.00 (Flip) โ€” โ€” +0.22 Shortcut illusion: TQA drops -0.14, XsTest surges
Adaptive Defense (OLMoE) Routing-Evasion Attack (REA) ASR 0.85 (Base) 0.60 (Full-FT) 0.12 (RASA) โ€” REA explicitly circumvents SCEs; RASA cuts evasion

Key Findings

  • Causal Proof of the Alignment Shortcut: Forcing prefill routing decisions back to the original model's routing patterns collapses the harmlessness of full-parameter fine-tuned models by 38%โ€“46% across all attacks, whereas RASA drops by merely 4%โ€“6%. This confirms that conventional fine-tuning safety relies causally on routing shortcuts, while RASA repairs underlying expert parameters.
  • Cross-Attack SCE Overlap Drives Generalization: Single-attack RASA alignment yields noticeable cross-attack transfer. Jaccard similarity across SCE sets identified by different attacks is 0.31โ€“0.38 (~5x the 0.07 random baseline), proving that distinct jailbreaks disproportionately converge on shared safety-critical experts. Mixed-attack training unites these sets to achieve 0.98โ€“1.00 harmlessness.
  • Early-Layer Concentration Driven by Variance: In OLMoE (16 layers, 64 experts/layer), 86%โ€“96% of per-batch SCE selections concentrate in layers 0โ€“3, dropping to zero beyond layer 5. While mean AAD is uniform across layers, batch-to-batch variance in L0โ€“L3 is massively elevated, indicating that adversarial surface perturbations originate routing distortions immediately at the bottom of the Transformer stack.
  • Resilience Against Adaptive Routing Evasion: Under the white-box Routing-Evasion Attack (REA), which uses GCG-style token search to maximize jailbreak compliance while explicitly minimizing SCE activation, full-parameter fine-tuning suffers a 60% Attack Success Rate (ASR). RASA's router consistency loss successfully counteracts evasion, keeping ASR at an exceptionally low 12%.

Highlights & Insights

  • First Systematic Characterization of MoE Alignment Shortcuts: Discovers and causally verifies that sparse routing provides an escape hatch during standard safety training, allowing models to look safe by redirecting inputs rather than repairing defective representations.
  • Ultra-Efficient AAD Attribution: Bypasses costly 32-step Integrated Gradients backpropagation by utilizing paired batch activation frequency discrepancies, identifying causal safety experts with a 40x speedup (single-forward in ~2.4s).
  • Architecture-Preserving Defense-in-Depth: Avoids introducing auxiliary LoRA modules, specialized safety experts, or inference-time routing overrides, preserving original MoE parameter topology, inference latency, and general reasoning capability.

Limitations & Future Work

  • Multi-Turn Adaptive Attack Resilience: While generalizing well to static multi-turn jailbreaks (MHJ: 0.68โ€“0.76 harmlessness), conversational extension RASA-MT achieves only 0.45โ€“0.55 harmlessness against adaptive multi-agent frameworks (X-Teaming), leaving multi-turn cumulative context drift an open challenge.
  • Dependency on Base Anchor Behavior: The framework assumes the base model can output a safe refusal for the benign anchor variant of a harmful intent; bootstrapping RASA on completely unaligned or raw base MoE models remains unexamined.
  • Frontier Extreme-Scale Verification: Although tested up to Mixtral-8x7B (47B parameters) across 8-way, 64-way, and 128-way routing regimes, computational budgets prevented verification on frontier hundred-billion-scale models with fine-grained shared experts like DeepSeek-V2/V3.
  • vs SteerMoE (Fayyaz et al., 2025): SteerMoE relies entirely on inference-time routing logit suppression without parameter updates, remaining vulnerable to surface jailbreak prompts; RASA achieves permanent training-time parameter and routing repair with zero inference latency.
  • vs SafeMoE-Fine-tuning (Kim et al., 2026): SafeMoE regularizes routing distributions but does not repair internal expert weights, leaving latent safety-critical experts unaligned; RASA addresses both expert remediation and routing stability.
  • vs SafeX / MidPO (Lai et al., 2025; Qi et al., 2025b): These methods attach auxiliary safety experts or alter the MoE topology; RASA operates strictly within the native parameter footprint without modifying architecture or routing semantics.

Rating

  • Novelty: โญโญโญโญโญ Formulates and causally validates the MoE alignment shortcut hypothesis; designs an elegant bi-level alternating repair framework.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive coverage of 3 distinct MoE architectures, causal restoration experiments, necessity/sufficiency validations, and white-box adaptive attacks.
  • Writing Quality: โญโญโญโญโญ Highly structured, rigorous mathematical formulation, and exceptionally clear mechanistic explanations.
  • Value: โญโญโญโญโญ Provides an indispensable blueprint for aligning sparse architectures in production without degrading general capabilities or inducing excessive over-refusals.