Skip to content

Mechanism Shift During Post-training from Autoregressive to Masked Diffusion Language Models

Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning
Keywords: Masked Diffusion Models, Autoregressive Post-training, Mechanistic Interpretability, Edge Attribution Patching, Circuit Reorganization

TL;DR

By employing Edge Attribution Patching with Integrated Gradients (EAP-IG) and logit-lens diagnostic probes across paired 7B architectures, this paper demonstrates that post-training autoregressive models into masked diffusion models induces a task-dependent mechanism shift: prefix-dominant tasks largely preserve inherited causal circuits, while globally constrained tasks systematically reorganize computation toward earlier transformer layers.

Background & Motivation

Autoregressive language models (ARMs) have established near-human fluency across varied language tasks, yet their causal masking enforces a unidirectional decoding bottleneck where early token errors propagate irreversibly throughout the sequence. Furthermore, tasks requiring multi-step planning, global consistency, or bidirectional infilling impose boundary constraints across the entire context that causal generation struggles to resolve natively. Masked diffusion models (MDMs) have recently surfaced as a promising non-autoregressive alternative capable of iterative refinement and simultaneous bidirectional attention. However, training large discrete diffusion models from scratch incurs staggering computational costs due to sluggish empirical convergence. To circumvent this hurdle, prominent post-training approaches such as DiffuLLaMA and Dream initialize directly from pretrained ARMs and adapt them to iterative discrete denoising, achieving strong performance at a fraction of the compute required for greenfield training.

Despite the notable empirical success of diffusion adaptation, the internal algorithmic consequences of post-training remain an uninspected black box. Imposing a discrete denoising objective onto a pretrained causal backbone does not guarantee an intrinsic transformation of computational mechanisms. It has remained fundamentally unresolved whether diffusion adaptation uniformly preserves inherited autoregressive pathways, wholesale overwrites the underlying computational graph, or selectively rewires circuits in response to the architectural demands of specific generation regimes. Understanding this transitional dynamic is crucial for diagnosing representational bottlenecks and designing principled training paradigms for non-autoregressive foundation models.

To resolve this question, this work investigates paired ARM–MDM families derived from identical pretrained backbones under rigorous mechanistic scrutiny. The authors formalize an entropy-based boundary distinguishing prefix-bound causal tasks from globally constrained planning tasks, applying gradient-based edge attribution and output-space projections across controlled counterfactual distributions. Core idea: diffusion post-training does not uniformly overwrite inherited autoregressive mechanisms, but selectively preserves or reorganizes computation according to task structure—retaining mid-to-late causal circuits on prefix-dominant tasks while front-loading task-critical computation into earlier layers and dispersing semantic specialization across broader component ensembles on globally constrained tasks.

Method

Overall Architecture

The authors develop a rigorous comparative circuit-analysis pipeline designed to map computational reorganization across paired autoregressive and masked diffusion architectures while controlling for model scale, initialization, and base tokenization. To evaluate whether qualitative shifts reproduce across independent model families, the study benchmarks Qwen2.5-7B paired with Dream-Base-7B, alongside LLaMA-2-7B paired with DiffuLLaMA-7B. The pipeline proceeds across three controlled stages: first, counterfactual clean–corrupted prompt pairs are constructed under identical semantic perturbations, from which sparse subgraphs of equal edge budget (Top-1,000 edges) are extracted using Edge Attribution Patching with Integrated Gradients (EAP-IG); second, structural shifts are quantified across edge Jaccard similarity, node attribution scores, and a layer-wise Center of Gravity (CoG); third, component-level logit-lens projections and residual-stream neuron activations probe the semantic specialization and representation density of task-critical components.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Paired ARM–MDM Model Inputs<br/>Qwen/Dream & LLaMA/DiffuLLaMA"] --> B["Counterfactual Interventions & EAP-IG Subgraph Extraction<br/>Top-1000 attribution edges under matched sparsity budget"]
    B --> C["Multi-Level Attribution Subgraph Comparison<br/>Edge Jaccard overlap & depth-wise Center of Gravity"]
    C --> D["Component-Wise Logit-Lens Probing<br/>Selective amplification via LogitGap & Delta-LME"]
    D --> E["Early-Layer Neuron Activation Probing<br/>Semantic category variance & interventional zero-ablation"]

Key Designs

1. Counterfactual Interventions & EAP-IG Subgraph Extraction: Isolating Task Circuits Across Incompatible Decoding Interfaces
To discover sparse attribution subgraphs across fundamentally different inference interfaces (unidirectional autoregression versus iterative discrete denoising), this design applies identical semantic interventions to counterfactual clean and corrupted input pairs without using the [MASK] token as the semantic perturbation. For ARMs, the intervention perturbs the prefix context directly and scores logits at the target token position; for MDMs, the target position, mask schedule, and timestep aggregation remain strictly invariant between clean and corrupted passes, changing only the unmasked semantic context. Using Integrated Gradients applied to edge attribution patching, each directed connection \(u \to v\) receives an attribution score \(a_{u \to v}\) reflecting its marginal contribution to the clean–corrupted task contrast. Retaining an exact matched-sparsity operating budget of 1,000 top edges (\(S^{(1000)}\)) ensures that overlap metrics are not confounded by divergent subgraph sizes.

2. Multi-Level Attribution Subgraph Comparison: Quantifying Pathway Reuse and Depth-Wise Relocation
To capture computational changes across complementary structural scales, this design establishes three quantitative scalar metrics. At the edge level, the Jaccard overlap \(J(E_{\text{top}}^{\text{ARM}}, E_{\text{top}}^{\text{MDM}})\) measures exact connection reuse between architectures. At the node level, incoming and outgoing attribution scores incident to node \(v\) are summed into an importance score \(s(v) = \sum_{(u,v) \in E_{\text{top}}} a_{u \to v} + \sum_{(v,u) \in E_{\text{top}}} a_{v \to u}\), identifying the Top-100 influential source/sink components. At the architectural depth level, the Center of Gravity (CoG) summarizes the vertical concentration of attribution mass: $$ \text{CoG} = \frac{\sum_l l \cdot A_l}{\sum_l A_l} $$ where \(l\) denotes transformer layer index and \(A_l = \sum_{v:\ell(v)=l} s(v)\) is the aggregated attribution mass at layer \(l\). A substantial drop in CoG serves as the mathematical marker of computational front-loading in non-autoregressive reasoning.

3. Component-Wise Logit-Lens Probing: Measuring Semantic Alignment and Specialization Dispersion
To uncover how task-critical components project information into the output space, intermediate activations from the Top-100 attribution nodes are mapped to vocabulary logits via each model's native unembedding matrix \(W_U\). Probing traces ARM alignments across token positions and MDM alignments across denoising timesteps. The analysis formulates selective amplification via the top-two logit gap \(\text{LogitGap}(c) = l_{(1)}(c) - l_{(2)}(c)\), entity fraction \(\text{NameFrac@}K(c) = \frac{|T_K(c) \cap \mathcal{N}|}{|T_K(c)|}\), and the Log-Mean-Exp dominance gap: $$ \Delta\text{LME}(c) = \text{LME}(T_K(c) \cap \mathcal{N}) - \text{LME}(T_K(c) \setminus \mathcal{N}) $$ evaluated alongside normalized entropy \(H(c) = -\sum_{i \in T_K(c)} p_i \log p_i\). These metrics reveal that while ARMs depend on a handful of sharply tuned components with massive logit gaps, MDMs distribute evidence across diffuse component ensembles characterized by higher entropy and subdued selective amplification.

4. Early-Layer Neuron Activation Probing: Characterizing Bottom-Network Reorganization Under Global Constraints
To explain how early layers absorb relocated computation when CoG drops sharply, this design inspects scalar residual-stream neuron coordinates across layers 0–4 and aggregates automated natural-language explanations over activating contexts. In ARMs, early-layer activations exhibit high semantic concentration within narrow categories (e.g., LLaMA allocating 2,821 neurons to descriptive adjectives in IOI, yielding high across-category variance \(\sigma=369.78\)). In contrast, MDMs display a flat, task-agnostic profile with active neurons distributed evenly (50–200 per category, \(\sigma=202.52\)) across 426 interpretable units. This establishes that MDM early layers accommodate relocated global constraints through dense, distributed ensemble routing rather than sharp single-neuron tuning.

Benchmark & Evaluation Protocol

The empirical evaluation grounds mechanistic probes across four diagnostic benchmarks categorized by an MDM conditional-entropy gap \(\Delta H := H_{\text{MDM}}(x_t \mid x_{<t}) - H_{\text{MDM}}(x_t \mid x_{\setminus t})\). Non-positive \(\Delta H\) designates the causal regime: Indirect Object Identification (IOI) and Greater-Than (GT). Positive \(\Delta H\) designates the global regime: Countdown (arithmetic target planning) and Semantic Infilling (SI, bidirectional context completion). To prevent behavioral inaccuracies from distorting internal circuit comparisons, all EAP-IG extractions, CoG metrics, and logit-lens probes are conducted strictly on pair-specific common-correct subsets (500 instances sampled per task). Finally, targeted zero-ablation of identified early-layer MDM components (layers 2–8) is benchmarked against random and middle-to-late layer controls to verify functional necessity.

Key Experimental Results

Main Results

Under a matched sparsity budget of Top-1,000 attribution edges, the table below details edge Jaccard similarity, Top-100 component overlap, and depth-wise Center of Gravity across both model families (95% bootstrap confidence intervals in brackets):

Task Regime Benchmark Task Model Pair (ARM / MDM) Edge Jaccard Overlap Top-100 Component Overlap ARM Center of Gravity (CoG) MDM Center of Gravity (CoG)
Causal / Prefix-Dominant IOI Qwen2.5 / Dream 0.193 [0.174, 0.214] 0.105 [0.091, 0.121] 17.5 [17.2, 17.9] 20.4 [20.0, 20.9]
Causal / Prefix-Dominant IOI LLaMA-2 / DiffuLLaMA 0.088 [0.074, 0.104] 0.124 [0.108, 0.143] 18.2 [17.8, 18.6] 19.8 [19.3, 20.3]
Causal / Prefix-Dominant GT Qwen2.5 / Dream 0.290 [0.268, 0.313] 0.515 [0.489, 0.542] 18.7 [18.3, 19.1] 14.4 [14.0, 14.9]
Causal / Prefix-Dominant GT LLaMA-2 / DiffuLLaMA 0.054 [0.044, 0.066] 0.117 [0.101, 0.135] 17.3 [16.9, 17.8] 11.7 [11.2, 12.1]
Global Constraint Countdown Qwen2.5 / Dream 0.008 [0.005, 0.012] 0.093 [0.079, 0.109] 16.5 [16.1, 16.9] 4.8 [4.5, 5.1]
Global Constraint Countdown LLaMA-2 / DiffuLLaMA 0.032 [0.025, 0.041] 0.081 [0.067, 0.098] 17.1 [16.6, 17.5] 5.3 [4.9, 5.7]
Global Constraint SI Qwen2.5 / Dream 0.225 [0.191, 0.263] 0.429 [0.386, 0.471] 13.7 [13.0, 14.4] 3.8 [3.4, 4.3]
Global Constraint SI LLaMA-2 / DiffuLLaMA 0.018 [0.010, 0.031] 0.124 [0.096, 0.158] 13.1 [12.4, 13.8] 3.9 [3.4, 4.5]

Ablation Study & Mechanistic Probes

Targeted zero-ablation on Countdown and logit-lens semantic dominance metrics on IOI quantify the causal necessity and representational dispersion of reorganized circuits:

Experimental Probe Model / Evaluated Configuration Measured Metric Metric Value Mechanistic Interpretation & Role
Causal Zero-Ablation Unablated MDM Baseline Task Accuracy (%) 91.2% Unintervened shared-success baseline on Countdown
Causal Zero-Ablation Ablate Identified Early Layers (L2–L8) Task Accuracy (%) 35.2% Severe 56.0% accuracy drop confirms early-layer functional necessity
Causal Zero-Ablation Ablate Middle-to-Late Layers (L16–L24) Task Accuracy (%) 89.4% Negligible performance degradation under matched control ablation
Logit-Lens Probing Qwen2.5-7B (ARM) Median \(\Delta\text{LME}\) / LogitGap 0.90 / 2.12 Sharp single-head amplification (Entropy 0.97); localized pointer circuit
Logit-Lens Probing LLaMA-2-7B (ARM) Median \(\Delta\text{LME}\) / LogitGap 0.89 / 0.70 Substantial name-token dominance in early-to-mid heads; Entropy 1.81
Logit-Lens Probing Dream-Base-7B (MDM) Median \(\Delta\text{LME}\) / LogitGap 0.31 / Diffuse Dominance gap contracts significantly; proper-noun alignment is dispersed
Logit-Lens Probing DiffuLLaMA-7B (MDM) Median \(\Delta\text{LME}\) / LogitGap 0.05 / 0.49 Near-zero dominance gap and highest entropy (2.01); diffuse ensemble alignment

Key Findings

  • Task-Dependent Depth Relocation: For global planning and bidirectional integration (Countdown and SI), MDMs shift attribution mass massively toward early transformer layers (Dream CoG drops by 11.7 layers from 16.5 to 4.8). In contrast, prefix-dominant tasks (IOI) maintain attribution in mid-to-late layers (CoG 20.4 vs 17.5), showing no early-layer relocation.
  • Structural Decoupling on Global Tasks: Edge Jaccard similarity drops to near zero on Countdown (0.008 for Qwen/Dream; 0.032 for LLaMA/DiffuLLaMA), proving that diffusion post-training abandons inherited autoregressive pathways when generation requires global constraint satisfaction.
  • Collapse of Single-Component Specialization: While ARMs depend on individual heads exerting decisive control (e.g., Qwen head a27.h5 achieving a mean logit of 21.14 on Dan), MDMs exhibit lower LogitGaps and higher distributional entropy, reflecting evidence pooling across broader component ensembles.

Highlights & Insights

  • The Task-Compatibility Hypothesis: Post-training does not indiscriminately rewrite the base network or rigidly inherit causal execution. Instead, the optimization process preserves inherited circuitry when task demands align with prefix structure, but selectively constructs new front-loaded pathways when bidirectional dependencies arise.
  • Front-Loading Global Planning: The radical shift of attribution to layers 0–4 on Countdown suggests that non-autoregressive diffusion models establish global planning constraints at the very bottom of the residual stream, freeing later layers for parallel token denoising and fine-grained lexical recovery.
  • Methodological Implications for Circuit Discovery: The findings warn against assuming that circuits in non-autoregressive models mirror the sparse, head-centric motifs found in ARMs. Mechanistic interpretability for discrete diffusion must transition from searching for isolated "hero heads" to characterizing distributed ensemble routing.

Limitations & Future Work

  • Scope Confined to Controlled Diagnostic Tasks: The empirical findings are demonstrated on four structured diagnostic benchmarks (IOI, GT, Countdown, SI) where clean–corrupted counterfactuals are tractable. Generalization to open-ended conversational generation and open-domain text modeling remains unproven.
  • Confounded Post-Training Variables: The holistic comparative framework captures the net outcome of diffusion adaptation without isolating the separate causal impacts of the diffusion loss objective, discrete noise schedules, dataset distribution shifts, and optimization trajectories.
  • Scaling to Long CoT and Verb-Final Typologies: Future work should scale attribution patching to extended multi-step reasoning traces and evaluate natural languages with strong right-to-left dependencies (e.g., verb-final syntax in Japanese, Korean, Basque, or German subordinate clauses).
  • vs Autoregressive Circuit Discovery (Bhaskar et al., 2024; Wang et al., 2023): Prior mechanistic interpretability established canonical circuits (e.g., induction heads and name movers) within causal transformers; this work extends EAP-IG across architectural paradigms, discovering how those circuits mutate during diffusion post-training.
  • vs Empirical Diffusion Adaptation (Gong et al., 2025; Ye et al., 2025b): While foundational papers for DiffuLLaMA and Dream benchmarked generation speed and downstream task accuracy, this work provides the first circuit-level explanation of how pre-trained autoregressive representations are rewired internally.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering circuit-level analysis bridging causal and discrete masked diffusion language model paradigms.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous controls across two 7B families, four tasks, bootstrap confidence intervals, and interventional zero-ablation.
  • Writing Quality: ⭐⭐⭐⭐⭐ Flawless conceptual clarity, precise mathematical definitions, and disciplined interpretation of mechanistic probes.
  • Value: ⭐⭐⭐⭐⭐ Provides profound foundational insights into non-autoregressive model adaptation and the internal mechanics of discrete diffusion.