Expert-Aware Causal Tracing of Factual Recall in Sparse MoE Language Models¶
Conference: EMNLP 2026
Paper: https://aclanthology.org/2026.emnlp-main.1422/
Area: LLM Reasoning
Keywords: Sparse Mixture-of-Experts, Causal Tracing, Activation Patching, Factual Recall, Mechanistic Interpretability
TL;DR¶
Using a two-stage activation patching protocol under conditional fixed routing, this paper reveals that successful MoE-block factual recall restoration exhibits striking architectural asymmetry: Qwen3 displays a partial, nonunique recurrent singleton contribution with positive specificity, whereas Mixtral exhibits routed-set specificity without singleton specificity.
Background & Motivation¶
In mechanistic interpretability of large language models, factual recall serves as a foundational benchmark for causal mediation analysis. Under the conventional causal tracing and activation patching paradigm, researchers disrupt factual predictions by injecting Gaussian noise into the subject-token input embeddings of cloze-style prompts, and subsequently patch clean intermediate activations into the corrupted execution stream to isolate the internal representations that mediate the retrieval. In standard dense Transformer architectures, this framework has firmly established that mid-layer feed-forward network (FFN) modules function as localized key-value memories that write factual attributes into the residual stream.
However, the rapid transition toward sparse Mixture-of-Experts (MoE) architectures, exemplified by Mixtral and Qwen MoE variants, introduces a critical resolution gap into this block-level attribution mechanism. In sparse MoE layers, token processing is no longer handled by a single dense feed-forward block, but rather by an input-dependent subset of routed experts whose sparse outputs are dynamically scaled and aggregated by a routing gate. Consequently, standard MoE-block output patching creates fundamental interpretive ambiguity: while patching an entire clean MoE-block output can demonstrate that a specific layer causally rescues a corrupted factual preference, the block output is an aggregate over routed expert updates, leaving it completely unresolved whether the recovery is driven by an individual key expert or inherently distributed across the co-activated routed coalition.
Prior interpretability efforts on MoEs have predominantly centered around routing statistics, domain specialization heuristics, or parameter editing, lacking a controlled, counterfactual intervention framework targeted directly at expert-level updates. To bridge this gap, this paper formulates an expert-aware causal tracing methodology designed to deconstruct MoE block-level factual rescue down to individual routed components. Core idea: establish a two-stage discovery-validation activation patching protocol under conditional fixed routing that isolates routed expert updates via suppression ablation differences and patches clean-minus-noised updates, demonstrating that MoE-block factual recovery can either localize to a specific recurrent singleton or resolve strictly at the routed-set level.
Method¶
Overall Architecture¶
The paper designs a rigorous, two-stage causal tracing and activation patching protocol to systematically deconstruct block-level factual rescue across individual routed experts. The complete pipeline unfolds across two distinct operational stages: the first stage conducts a layer-level discovery sweep across all MoE layers on a discovery set (128 cases), replacing corrupted final-position MoE-block outputs with clean counterparts to identify candidate layers that maximize the restoration of the true-vs-foil logit difference; the second stage performs fine-grained expert-level causal patching at the pinned target layer, utilizing suppression ablation differences under conditional fixed routing to inject clean-minus-noised expert updates into the corrupted run, evaluating whether candidate recurrent experts exhibit genuine specificity over same-layer active controls on a strictly held-out validation set (128 cases).
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Factual Prefix Input<br/>Subject embeddings corrupted with Gaussian noise"] --> B["Stage 1: Two-Stage Sweep & Recurrence-First Selection<br/>Discovery split sweeps MoE layers and isolates frequent clean-active candidates"]
B --> C["Stage 2: Conditional Fixed-Routing Ablation Patching<br/>Suppression difference computes clean-minus-noised expert delta for rescue"]
C --> D["Stage 3: Same-Layer Active & Equal-Norm Specificity Controls<br/>Benchmarked against same-source active controls to rule out gate weight and norm confounds"]
D --> E["Causal Granularity Attribution Conclusion<br/>Qwen3 shows partial singleton specificity / Mixtral shows routed-set coalition specificity"]
Key Designs¶
1. Two-Stage Sweep & Recurrence-First Selection: Decoupling Accidental Single-Case Spikes from True Replicability Unconstrained parameter-space sweeping in sparse MoE models risks severe multiple testing artifacts, particularly in architectures with 128 experts per layer where an expert might achieve inflated average rescue scores by sheer coincidence on just one or two instances. To prevent statistical fishing, the authors institute a strict separation between discovery (128 cases) and validation (128 cases) splits. During the discovery sweep, candidate MoE layers are evaluated for aggregate block rescue, identifying the peak layer \(\ell\). Within this layer, candidate experts must satisfy a strict recurrence-first criterion: an expert must be clean-active in at least half of the discovery cases (\(\ge 64\) out of 128). Among all eligible recurrent experts, the candidate with the highest mean rescue score is selected as \(e^\star\). Crucially, this selected expert and its layer are subsequently fixed as unalterable hypotheses and evaluated exclusively on the held-out validation set.
2. Conditional Fixed-Routing Ablation Patching: Isolating Expert Updates without Dynamic Rerouting Confounds Directly zeroing out an expert's parameters or intervening on router logits in an MoE model typically triggers dynamic weight renormalization or rerouting among remaining experts, entangling the expert's intrinsic update vector with complex routing perturbation dynamics. To eliminate this confound, the framework defines expert updates under conditional fixed routing. Letting \(\text{MoEOut}(x)\) denote the final-position MoE-block output vector for input \(x\), the marginal update contributed by expert \(e\) under the observed routing decision is extracted via the ablation difference: $\(c_e(x) = \text{MoEOut}(x) - \text{MoEOut}_{\setminus e}(x)\)$ where \(\text{MoEOut}_{\setminus e}(x)\) suppresses expert \(e\)'s contribution while keeping all remaining expert outputs and their unrenormalized routing weights strictly unchanged. If expert \(e\) was not chosen by the router for input \(x\), \(c_e(x) = 0\). The clean-minus-noised expert patch vector is then defined as: $\(\delta_e = c_e(x_{\text{clean}}) - c_e(x_{\text{noised}})\)$ Injecting \(\delta_e\) into the corrupted MoE-block output directly isolates the causal restoration effect \(\text{Rescue}(e) = \Delta_{\text{patched}, e} - \Delta_{\text{noised}}\) attributable to expert \(e\) strictly within the context of the observed routing state.
3. Same-Layer Active & Equal-Norm Specificity Controls: Ruling Out Gate Weight and Vector Magnitude Artifacts Observing a positive rescue score for a selected expert is insufficient to claim causal localization, as injecting any coherent representation vector into mid-to-late transformer layers could generically elevate next-token confidence. The protocol introduces an active specificity contrast \(\text{Spec}\), which benchmarks the selected expert \(e^\star\) against the average rescue achieved by alternative clean-active experts \(R\) routed at the same layer for the identical prompt: $\(\text{Spec} = \text{Rescue}(e^\star) - \frac{1}{|R|}\sum_{r \in R}\text{Rescue}(r)\)$ To further eliminate the structural possibility that \(e^\star\) only outperforms controls due to larger routing gate weights or disproportionately large patch vector norms, the authors introduce gate-weight-matched controls (selecting the active control \(r^\star\) closest in clean router weight) and equal-norm scaled controls. Under equal-norm scaling, both selected and control patch vectors are rescaled to match the smaller original vector norm \(\tilde{\delta}_e = \delta_e \frac{\min(\|\delta_{e^\star}\|, \|\delta_{r^\star}\|)}{\|\delta_e\|}\), proving whether the causal advantage stems from semantic directional alignment rather than raw perturbation energy.
Benchmark & Evaluation Protocol¶
The diagnostic setup leverages single-token factual contrasts curated from the COUNTERFACT benchmark (Meng et al., 2022). Each instance comprises a relation prefix \(x\), a true target object \(a_t\), and a counterfactual foil object \(a_f\), evaluated through the true-vs-foil logit difference metric \(\Delta(x; a_t, a_f) = \ell_x(a_t) - \ell_x(a_f)\). Corrupted runs are generated by adding zero-mean Gaussian noise \(\epsilon \sim \mathcal{N}(0, \sigma^2 I)\) to the subject token embeddings, where \(\sigma\) is calibrated to \(3.0\times\) the standard deviation of the embedding matrix. Strict filtering retains cases satisfying single-token targets/foils, clean margins \(\Delta_{\text{clean}} \ge 1.0\), and subject-noise drops \(\Delta_{\text{clean}} - \Delta_{\text{noised}} \ge 0.5\). Scanning a seed-0 shuffled pool of 21,919 records yielded 256 usable cases each for Qwen3-30B-A3B-Base (48 MoE layers, 128 experts/layer, top-8 routing) and Mixtral-8x7B-v0.1 (32 MoE layers, 8 experts/layer, top-2 routing), split into 128 discovery and 128 validation cases. Uncertainty is quantified via 5,000 case-bootstrap resamples and 10,000 two-sided sign-flip significance tests.
Key Experimental Results¶
Main Results¶
On the 128 held-out validation cases per model, block-level MoE rescue and expert-level causal tracing results are detailed below:
| Model | Target Layer | Layer Rescue (95% CI) | Selected Expert | Expert Rescue (95% CI) | Specificity Spec (95% CI) | Expert/Layer Ratio |
|---|---|---|---|---|---|---|
| Qwen3-30B-A3B | L44 | +0.901 [+0.752, +1.053] | L44E069 | +0.463 [+0.344, +0.590] | +0.400 [+0.276, +0.533] | 51.4% |
| Mixtral-8x7B-v0.1 | L19 | +0.457 [+0.331, +0.579] | L19E006 | +0.099 [+0.018, +0.175] | -0.175 [-0.284, -0.072] | 21.6% |
The primary empirical finding demonstrates an unmistakable cross-model structural divergence: 1. In Qwen3-30B-A3B, layer sweep pinpoints L44, and expert tracing identifies L44E069 as a recurrent contributor with robust, positive specificity (\(\text{Spec} = +0.400\), 95% CI \([+0.276, +0.533]\)), accounting for 51.4% of the total block rescue. 2. In Mixtral-8x7B-v0.1, while L19 restores block-level factual preference (+0.457), its most recurrent active expert L19E006 yields minimal individual rescue (+0.099) and significantly negative specificity (\(\text{Spec} = -0.175\), 95% CI \([-0.284, -0.072]\)), underperforming randomly chosen active peer experts.
Ablation Study & In-Depth Controls¶
1. Qwen3 Control and Decomposition Analysis¶
Rigorous validation controls evaluated on the Qwen3 validation set across gate-matched subsets (116 cases) and relation-clustered cross-fact interventions (128 cases):
| Evaluation Setting / Control | Sample Size \(n\) | Selected Rescue | Control Rescue | Specificity / Contrast (95% CI) | Significance \(p\) |
|---|---|---|---|---|---|
| Raw Gate-Matched Control | 116 | +0.513 | +0.054 | +0.459 [+0.322, +0.607] | \(p < .001\) |
| Equal-Norm Rescaled Control | 116 | +0.239 | +0.051 | +0.188 [+0.119, +0.259] | \(p < .001\) |
| All-Active Peers Mean (\(\text{Spec}_{\text{all}}\)) | 115 | +0.513 | +0.072 | +0.441 [+0.309, +0.584] | \(p < .001\) |
| Cross-Fact Shuffled Patch | 128 | +0.463 (Fact-matched) | -0.032 (Shuffled) | +0.491 [+0.256, +0.753] | \(p < .001\) |
| Full L44 Block minus Expert-Only | 128 | +0.901 (Full block) | +0.463 (Expert) | +0.458 [+0.309, +0.619] | \(p < .001\) |
Even after strictly controlling for gate routing probabilities and matching perturbation vector norms, L44E069 retains a substantial specificity margin (+0.188). Cross-fact patch shuffling destroys the rescue (-0.032), confirming semantic fact-matching rather than generic representation recovery. Meanwhile, the full block patch delivers +0.458 additional rescue over L44E069 alone, proving that L44E069 is a partial and non-exclusive mediator.
2. Mixtral Routed-Set Reconstruction and Matched-Size Coalition Controls¶
To determine whether Mixtral's block effect resolves at the coalition level, routed-set patches were benchmarked against matched-size alternatives (10,000 relation-clustered bootstrap resamples, 20,000 sign-flip tests):
| Hypothesis / Coalition Contrast | Cases | Relations | Mean Gain | 95% Confidence Interval (95% CI) | Statistical \(p\)-value |
|---|---|---|---|---|---|
| Clean top-2 vs. Mean of 8 all-expert pairs | 128 | 26 | +0.303 | [+0.217, +0.385] | \(p < .001\) |
| Clean top-2 vs. Mean of within-union pairs | 70 | 23 | +0.176 | [+0.068, +0.281] | \(p = .0048\) |
| Routing union vs. 8 same-size coalitions | 128 | 26 | +0.275 | [+0.188, +0.360] | \(p < .001\) |
| Routing union vs. Clean top-2 coalition | 128 | 26 | +0.002 | [-0.043, +0.041] | \(p = .929\) |
The results confirm additive sufficiency and routed-set specificity: the actual clean-routed top-2 pair significantly outperforms random all-expert pairs (+0.303) and alternative combinations within the routing union (+0.176). Expanding the patch to include all experts in the clean/noised routing union offers no statistically significant advantage over the clean top-2 alone (\(p = 0.929\)).
Key Findings¶
- Singleton specificity does not track sparsity cardinality: Contrary to the naive intuition that sparser routing (top-2 in Mixtral) would concentrate functional responsibility into distinct individual experts, Mixtral exhibits negative singleton specificity. Conversely, Qwen3 with top-8 routing displays a recurrent, highly specific singleton contributor (L44E069, active in 112/128 discovery and 116/128 validation cases).
- Expert rescue improves logits and ranks without flipping greedy top-1: In Qwen3, the +0.463 rescue of L44E069 decomposes into a +0.394 boost in true-token logit and a -0.069 suppression of the foil logit. True-token probability and full-vocabulary rank improve in 67.2% and 65.6% of cases respectively, yet 0% of cases flip to top-1, demonstrating that factual recall requires multi-component convergence across the broader transformer circuit.
- Robustness across relation folds and noise regimes: In 5-fold cross-relation validation where entire relation categories were held out, L44E069 was universally selected across all 5 folds, retaining an average held-out specificity of +0.388. Across noise scales \(\sigma \in [1.0, 4.0]\), L44E069 rescue monotonically scales from +0.098 to +0.478 while preserving statistical significance.
Highlights & Insights¶
- Formulation of conditional fixed-routing interventions: By isolating expert updates via suppression difference vectors without triggering dynamic renormalization or network-wide rerouting, the study delivers an elegant, non-invasive protocol for MoE causal attribution.
- Dismantling the block-to-expert localization assumption: The paper provides definitive counter-evidence against the assumption that a causally validated MoE block implies an underlying single-expert memory, demonstrating that block recovery can reflect either localized singletons or distributed coalitions.
- Architectural granularity hypothesis: The architectural contrast suggests that fine-grained parameter partitioning (128 experts in Qwen3) enables modular specialization into recurrent roles, whereas coarse partitioning (8 experts in Mixtral) necessitates synergistic co-representation across the routed set.
Limitations & Future Work¶
- Restriction to single-token contrasts and designated foils: The evaluation is confined to single-token cloze instances with a single designated foil from COUNTERFACT, leaving multi-token entities, open-ended factual generation, and complex reasoning chains unexamined.
- Local intervention versus global circuit tracing: Causal restoration at the final prediction position under conditional fixed routing measures local information mediation, which does not constitute full physical localization of stored knowledge or trace the full multi-layer propagation circuit.
- Confounded cross-architecture comparison: Because Qwen3 and Mixtral diverge simultaneously across expert counts, routing top-\(k\), pretraining corpora, tokenizers, and auxiliary losses, the observed divergence cannot be conclusively ascribed to a single architectural hyperparameter.
Related Work & Insights¶
- vs Causal Tracing in Dense Models (Meng et al., 2022; Geva et al., 2023): While ROME pioneered activation patching at mid-layer FFN blocks in dense GPT architectures, this work addresses the structural aggregation of MoEs, refining the causal target from layer-level representations to individual expert updates.
- vs MoE Routing Geometry & Specialization (Wang et al., 2026; Fayyaz et al., 2026): While recent analyses argue that MoE routing reflects input geometric clustering rather than semantic specialization, this study demonstrates via controlled counterfactual interventions that specific expert updates nonetheless exert statistically significant, fact-matched semantic causal effects.
- vs MoE Knowledge Editing (Gu et al., 2026; Li et al., 2025): Where knowledge editing works (e.g., MoEEdit) seek to steer or patch model parameters while preserving routing stability, this work probes the intrinsic causal properties of unedited pretrained representations, establishing a benchmark for targeted expert editing.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ (Pioneering expert-level causal activation patching protocol resolving the MoE block-aggregation ambiguity)
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Exemplary design featuring discovery/validation splits, equal-norm and gate-matched controls, cross-relation folds, and matched-size coalition baselines)
- Writing Quality: ⭐⭐⭐⭐⭐ (Impeccable formal clarity, rigorous statistical framing, and careful disambiguation between fixed-routing and natural rerouting effects)
- Value: ⭐⭐⭐⭐☆ (Crucial foundational insights for mechanistic interpretability and localized knowledge editing in modern sparse MoE architectures)