Knowledge Editing for Masked Diffusion Language Models¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/holi-lab/MDM-KE
Area: LLM Reasoning
Keywords: Masked Diffusion Models, Knowledge Editing, Locate-then-Edit, Multi-token Generation, Intermediate State Optimization
TL;DR¶
This work provides the first systematic study of locate-then-edit knowledge editing in Masked Diffusion Language Models (MDMs), demonstrating that factual recall causally localizes to the early-to-mid MLP of the last subject token across both ARMs and MDMs, identifying a severe multi-token editing breakdown during iterative denoising, and proposing an intermediate-state optimization correction that substantially restores multi-token editing efficacy.
Background & Motivation¶
Large language models acquire immense amounts of factual associations during self-supervised pretraining, yet they frequently internalize erroneous information and inevitably become obsolete as real-world facts change over time. While full-parameter fine-tuning can inject new facts, it is computationally prohibitive and prone to catastrophic forgetting and collateral damage to general knowledge. Consequently, locate-then-edit methods have emerged as a dominant paradigm. By conceptualizing the down-projection layer of Transformer feed-forward networks (MLPs) as linear associative memory, these methods first localize the causal substrate of a fact via internal mechanistic tracing and subsequently compute closed-form rank-one parameter updates at that exact site, thereby preserving unrelated knowledge without auxiliary networks.
Crucially, the entire theoretical and empirical foundation of locate-then-edit methods (e.g., ROME, MEMIT, PMET, AlphaEdit) was established exclusively on causal autoregressive models (ARMs). Recently, masked diffusion models (MDMs) have surfaced as a competitive generative paradigm, leveraging non-causal bidirectional attention and iterative parallel denoising. Unlike ARMs, which predict the next token conditioned strictly on a causal left prefix, MDMs recover randomly masked absorbing tokens from bidirectional context. This profound architectural and objective divergence raises fundamental mechanistic questions: does factual recall remain localized to the same token-layer coordinates when attention flows globally in both directions? And can an edit optimized for a single static conditioning state successfully steer dynamic, multi-step iterative denoising?
To resolve these questions, this paper establishes a rigorous, controlled comparative framework across multiple paired ARM and MDM architectures. The investigation unveils a striking dual finding: on one hand, factual recall exhibits identical causal localization—concentrating in the early-to-mid MLP layers at the final subject token across both paradigms despite the MDMs' bidirectional connectivity. On the other hand, while single-token edits transfer seamlessly, multi-token edits suffer a catastrophic breakdown in MDMs. This failure is traced directly to the fact that standard edits are optimized solely on a fully masked initial state, whereas iterative multi-token denoising traverses dynamically revealed intermediate states for which the edit was never optimized. Core idea: expand the target-value optimization of locate-then-edit methods from a single fully masked input to uniformly sampled subsets of partially unmasked intermediate states, rendering parameter updates robust throughout iterative denoising trajectories and restoring multi-token editing performance.
Method¶
Overall Architecture¶
The methodology encompasses three core stages: causal localization verification, mask-augmented locate-then-edit adaptation (MEMIT†), and intermediate-state optimization correction (+Corr.). Given an edit request modifying a fact triplet \((s, r, o \to o^*)\), the pipeline first verifies the intervention site at the last subject token, optimizes the target value update over diverse masked and partially unmasked conditioning states, and solves for the closed-form weight update across early-to-mid MLP layers.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Fact Triplet & Prompt<br/>(s, r, o*) with Appended Mask Slot"] --> B["Bidirectional Causal Tracing<br/>Gaussian Noise & Layerwise Restoration"]
B --> C["Mask-Augmented Association MEMIT†<br/>Bidirectional Mask Context & Residual Solver"]
C --> D["Intermediate State Subset Sampling<br/>Uniform Revealed Count & Random Position Masking"]
D --> E["Multi-State Joint Value Optimization<br/>Robust Residual Optimization & Closed-Form Update"]
E --> F["Output: Edited MDM Weights<br/>Iterative Denoising Sustains Multi-Token Fact"]
Key Designs¶
1. Bidirectional Causal Tracing: Verifying Localized Factual Recall under Global Attention To determine where factual information is processed in an MDM without causal left-to-right masks, the authors adapt the causal tracing framework to masked inputs. Specifically, an 8-token mask block is appended to the prompt \((s, r)\), and output probabilities \(P(o)\) are measured at the first mask position. The subject entity tokens are corrupted with calibrated Gaussian noise \(\epsilon \sim \mathcal{N}(0, \nu^2)\) with \(\nu = 3\sigma\) (scaled to the model's subject embedding standard deviation). During corrupted runs, individual hidden states, attention contributions, or MLP outputs across token positions \(i\) and layers \(l\) are restored to their clean uncorrupted values to calculate the normalized Average Indirect Effect (normAIE):
Complementary validations via gradient-based attribution (\(\alpha_i^{(\ell)} = \sum_h m_{i,h}^{(\ell)} \frac{\partial \log P(o)}{\partial m_{i,h}^{(\ell)}}\)) and 10-layer MLP knockout independently confirm that factual recall in MDMs concentrates sharply at the final subject token in early-to-mid MLP layers (e.g., layers 4–7 in LLaDA and layers 4–7 in Dream). This rigorously refutes the hypothesis that bidirectional pretraining disperses factual recall uniformly across the token sequence.
2. Mask-Augmented Association MEMIT†: Unified Conditioning for Key-Value Extraction In ARMs, causal attention guarantees that the hidden state at the last subject token is mathematically invariant whether the subsequent prompt is bare or followed by target tokens. In MDMs, however, bidirectional attention allows appended tokens to retroactively alter activations at the subject tokens. To eliminate representation mismatch, MEMIT† unifies all four input distributions involved in locate-then-edit (the rewriting prompt, the KL-divergence anchor prompt, key extraction, and residual baseline computation) by appending an identical block of \(L\) mask tokens matching the length of \(o^*\). The target residual \(\delta\) is obtained by gradient descent to minimize negative log-likelihood on mask positions alongside a KL constraint:
The weight update \(\Delta = R K_1^\top (C + K_1 K_1^\top)^{-1}\) is then solved in closed form across the chosen layer range, using empirical covariance matrix \(C \approx \lambda \mathbb{E}[k k^\top]\) to preserve existing associations.
3. Intermediate State Subset Sampling: Eliminating Denoising Trajectory Mismatch While MEMIT† succeeds on single-token targets, it fails severely on multi-token targets because \(\delta\) is fitted solely on the fully masked state \(S_0\). During iterative generation, MDMs evaluate partially unmasked states \(S_j\) where a subset of target tokens has already been revealed with high confidence while others remain masked. Controlled probing reveals that revealing even a single correct gold token (\(S_1\)) flips top-1 predictions of subsequent masked tokens from correct to incorrect in 33.1%–53.4% of cases.
To resolve this trajectory mismatch, the intermediate-state correction (+Corr.) optimizes the residual \(\delta\) across a diverse spectrum of unmasked configurations. For an \(N\)-token target \((t_1, \dots, t_N)\), the initial optimization step maintains the fully masked state. In each subsequent step, the algorithm uniformly samples the number of revealed tokens \(k \sim \mathcal{U}\{0, \dots, N-1\}\), uniformly selects a subset of positions \(R \subseteq \{1, \dots, N\}\) of cardinality \(k\), injects gold tokens \(t_i\) into \(R\), and optimizes \(\delta\) strictly on the remaining \(N-k\) masked positions:
By decoupling optimization from any fixed reveal order or decoding heuristic, \(\delta\) becomes inherently robust to dynamic confidence-based unmasking schedules during inference.
Loss & Training¶
Because the optimization target is distributed across \(N\) distinct partial-mask configurations rather than concentrated on a single state, the gradient optimization budget for fitting \(\delta\) is expanded (4-fold for LLaDA, 2-fold for Dream), and the norm clamp for Dream is adjusted proportionally. For multi-layer editing across range \(R_{\text{layers}}\), residuals are distributed evenly and solved sequentially layer-by-layer: at each layer \(l\), the remaining residual \(\delta^{(l)}\) and key matrix \(K_l^1\) are dynamically recomputed after earlier layers are updated, preserving strict algebraic consistency.
Key Experimental Results¶
Main Results¶
Evaluation is conducted on KAMEL, recast into the COUNTERFACT schema, where facts are strictly filtered to ensure that targets have identical token lengths \(N \in \{1, 2, 3, 4\}\) across both model tokenizers in each pair (\(n=200\) facts per length). The table below compares editing efficacy (Efficacy, %) across autoregressive models (ARM), standard masked diffusion adaptation (MEMIT†), and the corrected formulation (+Corr.).
| Target Length \(N\) | LLaMA-3 (ARM) | LLaDA (MEMIT†) | LLaDA (+Corr.) | Qwen2.5 (ARM) | Dream (MEMIT†) | Dream (+Corr.) |
|---|---|---|---|---|---|---|
| \(N=1\) | 98.0 | 92.0 | — | 100.0 | 97.0 | — |
| \(N=2\) | 95.0 | 60.0 | 87.0 (+27.0) | 99.0 | 88.0 | 99.0 (+11.0) |
| \(N=3\) | 92.0 | 33.0 | 76.0 (+43.0) | 97.0 | 47.0 | 90.0 (+43.0) |
| \(N=4\) | 89.0 | 27.0 | 73.0 (+46.0) | 95.0 | 34.0 | 80.0 (+46.0) |
To demonstrate algorithmic generality, the correction was evaluated on two additional locate-then-edit frameworks, AlphaEdit and PMET:
| Model | Method | \(N=1\) | \(N=2\) | \(N=3\) | \(N=4\) |
|---|---|---|---|---|---|
| LLaDA-8B | AlphaEdit† | 93.0 | 61.0 | 35.0 | 34.0 |
| LLaDA-8B | AlphaEdit† + Corr. | — | 95.0 (+34.0) | 91.0 (+56.0) | 91.0 (+57.0) |
| LLaDA-8B | PMET† | 95.0 | 73.0 | 58.0 | 57.0 |
| LLaDA-8B | PMET† + Corr. | — | 75.0 (+2.0) | 73.0 (+15.0) | 67.0 (+10.0) |
| Dream-v0-7B | AlphaEdit† | 100.0 | 99.0 | 56.0 | 46.0 |
| Dream-v0-7B | AlphaEdit† + Corr. | — | 98.0 (-1.0) | 97.0 (+41.0) | 84.0 (+38.0) |
| Dream-v0-7B | PMET† | 96.0 | 81.0 | 67.0 | 43.0 |
| Dream-v0-7B | PMET† + Corr. | — | 90.0 (+9.0) | 78.0 (+11.0) | 61.0 (+18.0) |
Ablation Study¶
Ablations on intermediate-state sampling examine how positions are chosen during optimization (evaluated on KAMEL at \(N=3\)) and how reveal-count distributions impact efficacy across lengths.
| Ablation Dimension / Policy | LLaDA Efficacy | LLaDA Generalization | Dream Efficacy | Dream Generalization |
|---|---|---|---|---|
| Reveal Position Policy (\(N=3\)) | ||||
| Left-to-Right | 58.5 | 27.5 | 84.5 | 56.5 |
| Confidence-Based (Pre-edit model) | 61.5 | 34.0 | 81.5 | 59.5 |
| Random Uniform Resampling (Ours) | 76.0 | 36.0 | 89.5 | 68.0 |
| Reveal-Count Distribution (LLaDA) | \(N=2\) | \(N=3\) | \(N=4\) | Notes |
| (a) Fully masked only | 64.0 | 44.0 | 39.0 | Expanded budget without partial masks |
| (b) Biased toward fewer revealed | 88.0 | 73.0 | 69.0 | Favors small reveal steps |
| (c) Biased toward more revealed | 85.0 | 65.0 | 55.0 | Favors prefix completion |
| (e) Deterministic cycle | 87.0 | 76.0 | 73.0 | Fixed cyclic order |
| (f) Uniform random (Ours) | 87.0 | 76.0 | 73.0 | Robust without scheduling assumptions |
Key Findings¶
- Multi-token editing breakdown is driven by intermediate conditioning shifts: Probing \(S_1\) (one gold token revealed) on facts that succeeded perfectly at \(S_0\) causes 33.1%–53.4% of remaining suffixes to turn incorrect. Furthermore, 59.9%–74.1% of failed generations contain partial target tokens (e.g., generating "dance" or "trit" for target "trance"), proving that the failure originates from trajectory instability rather than an inability to recall target components.
- Random subset sampling substantially outperforms autoregressive left-to-right schedules: Enforcing left-to-right unmasking overfits \(\delta\) to sequential generation, dropping efficacy by 17.5 percentage points on LLaDA. Random resampling provides robustness across diverse decoding heuristics (Top-1 probability, margin, entropy, random) and multi-token reveal counts per step.
- Zero degradation in general capabilities and ripple effects: Across SST-2, MRPC, CoLA, RTE, NLI, and MMLU, the six-benchmark average shifts by less than 0.8 percentage points after editing. In RippleEdits (Popular split), +Corr. boosts multi-token edit verification from 18%–20% to 58%–70% while preserving related factual associations (Preservation = 1.00).
Highlights & Insights¶
- Invariance of the Associative Memory Substrate: The finding that facts remain causally localized in the early-to-mid MLP at the last subject token in MDMs provides powerful mechanistic evidence that Transformers inherently organize associative memories at entity boundaries regardless of whether attention is causal or bidirectional.
- Optimization vs. Generation Mismatch: The study highlights an essential lesson for model manipulation: transferring techniques across paradigms requires matching not only parameter locations but also the dynamic generation trajectory. Standard benchmarks like COUNTERFACT (where over 98% of targets are single tokens) severely mask this failure mode.
- Trajectory-Agnostic Regularization: Rather than tailoring optimization to a specific decoding heuristic, sampling unmasked positions uniformly at random makes the edit robust across arbitrary unmasking orders and variable stride lengths.
Limitations & Future Work¶
- Performance Gap at Longer Target Sequences: While the correction boosts 4-token editing from 27% to 73% on LLaDA, a residual performance gap remains compared to the 89% efficacy of LLaMA-3, suggesting that discrete subset sampling does not fully resolve high-dimensional error compounding across many denoising steps.
- Evaluation Limited to Dense Architectures: Experiments focus on dense models ranging from 355M to 8B parameters (LLaDA, Dream, DiffuLLaMA, DiffuGPT); extending the analysis to sparse Mixture-of-Experts architectures (e.g., LLaDA-MoE) remains an open challenge.
- Integration with Continuous Denoising Timesteps: The current correction operates on discrete token masks. Incorporating continuous diffusion time representations \(\tau\) directly into key-value associative optimization represents a promising avenue for future exploration.
Related Work & Insights¶
- vs. ROME / MEMIT (Meng et al., 2023a,b): Pioneer locate-then-edit by treating MLPs as linear associative memories under autoregressive causal masking; this work adapts the formulation to bidirectional MDMs and repairs the multi-token trajectory failure.
- vs. AlphaEdit (Fang et al., 2025) & PMET (Li et al., 2024): Advanced locate-then-edit methods that constrain parameter updates to the null space or incorporate attention updates; this paper shows that both methods suffer the same multi-token degradation in MDMs and benefit equally from intermediate-state correction.
- vs. MDM Mechanistic Studies (Piskorz et al., 2025; Wang et al., 2026): While prior work analyzes context distortion from mask tokens or inspects representations via sparse autoencoders, this study provides the first causal investigation into explicit factual recall and parameterized model editing in diffusion language models.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering study of locate-then-edit knowledge editing in masked diffusion models, exposing and solving the critical multi-token generation mismatch.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluation across 5 ARM–MDM matched pairs (355M–8B), three localization methods (causal tracing, gradient attribution, knockout), multiple editing frameworks, and comprehensive multi-token datasets.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear narrative structure, transparent diagnosis of failure mechanisms, and well-designed empirical validations.
- Value: ⭐⭐⭐⭐⭐ Establishes a foundational benchmark and methodology for knowledge editing in non-autoregressive and diffusion-based language models.