The Labyrinth and the Thread: Rethinking Regularizations in Sequential Knowledge Editing for Large Language Models

Conference: ICML 2026
arXiv: 2605.26670
Code: https://github.com/Wangzzzzzzzz/OTE-SE-Alignment (Available)
Area: Knowledge Editing / LLM
Keywords: Sequential Knowledge Editing, AlphaEdit, Null-space Projection, OTE-SE Equivalence, Necessity of Regularization

TL;DR

This paper proves from an optimization perspective that the stability of sequential editing (SE) stems from "cumulative updates being equivalent to the solution of one-time editing (OTE)." Fancy mechanisms like AlphaEdit's null-space projection or post-processing regularizations in PRUNE/RECT are not the critical factors—as long as OTE-SE alignment is ensured, 2000 steps of sequential editing can be stably completed across four mainstream LLMs even after removing these regularizations.

Background & Motivation

Background: Structured knowledge editing aims to precisely modify facts stored in LLMs as \((s,r,o)\) triplets without retraining. Mainstream approaches follow the locate-and-edit route: first locate the FFN output projection \(\mathbf{W}\) where the fact is stored, then perform a constrained least-squares update on that layer. To support "continuous/sequential editing," several mechanisms have emerged recently, including AlphaEdit (null-space projection), PRUNE (limiting update spectrum), RECT (restricting relative weight changes), and MEMIT (constrained optimization), becoming increasingly complex.

Limitations of Prior Work: Each method introduces its own set of regularizations/constraints and claims them to be the key to stability, yet a unified theory is lacking. Specifically, AlphaEdit attributes "thousands of stable editing steps" almost entirely to null-space projection \(\mathbf{P}\) (satisfying \(\mathbf{K}_0^\top \mathbf{P}=\mathbf{0}\)), an explanation that has never been rigorously tested. Consequently, new methods continue to stack constraints, making the design space as cluttered as a labyrinth.

Key Challenge: Existing explanations tie "stability" to a "specific regularization mechanism," but no one has answered the more fundamental question: What is the true necessary and sufficient condition for stable SE? If null-space projection is indeed the core, why does the model collapse immediately when the historical knowledge constraints are removed (leaving only the current one) despite keeping \(\mathbf{P}\)?

Goal: (1) Falsify the claim that "null-space projection is the core of AlphaEdit's success"; (2) Establish a unified design criterion for stable SE that is independent of specific regularizations; (3) Evaluate the true necessity of various regularizations under this criterion.

Key Insight: By formulating all locate-and-edit methods as the same ordinary least squares (OLS) problem and comparing the "cumulative over \(T\) steps" solution with the "one-time batch" solution, one can decouple "stability" from "the use of specific regularizations" if they are mathematically identical.

Core Idea: Stability = SE cumulative updates strictly reconstruct the OTE closed-form solution. Null-space projection is merely an instance that happens to satisfy this equivalence, not a necessary requirement.

Method

Overall Architecture

The authors rewrite all locate-and-edit methods as a single regularized least-squares problem, fitting the preservation set \((\mathbf{K}_0,\mathbf{V}_0)\) and the current edit set \((\mathbf{K}_t,\mathbf{V}_t)\) simultaneously to find a closed-form solution for \(\mathbf{W}+\boldsymbol{\Delta}\). The key transformation is that instead of arbitrarily adding regularizations at each step, they define the one-time editing (OTE) target first and then define the increment for step \(t\) as the difference between two adjacent OTE closed-form solutions: \(\tilde{\boldsymbol{\Delta}}_t := \boldsymbol{\Delta}^*_{\text{total},t} - \boldsymbol{\Delta}^*_{\text{total},t-1}\). This differential definition decouples "stability" from "regularization type" and rephrases it as whether the sequential accumulation strictly reconstructs the OTE solution. Based on this, the paper answers three progressive questions: falsifying the core role of null-space projection and establishing OTE-SE equivalence (RQ1); providing a differential mapping to construct stable SE from any regularized OTE target (RQ2); and providing analytical error compensation for post-processing regularizations like PRUNE/RECT (RQ3).

Key Designs

1. Null-space Falsification + OTE-SE Equivalence Theorem

To address "why AlphaEdit is stable," the authors designed a minimal falsifiable scenario called "Memorize-the-Latest": each step only requires remembering the current fact and drops historical preservation constraints, while still retaining null-space projection \(\mathbf{P}\) (where \(\mathbf{K}_0^\top\mathbf{P}=\mathbf{0}\)). On LLaMA-3 + CounterFact, the model's linguistic capability vanished immediately (GLUE all 0.000), proving \(\mathbf{P}\) alone cannot sustain stability. Investigating the root cause: the original AlphaEdit normal equation contains terms \(\mathbf{K}_0\mathbf{K}_0^\top\mathbf{P}\) and \((\mathbf{V}_0-\mathbf{W}_{t-1}\mathbf{K}_0)\mathbf{K}_0^\top\mathbf{P}\), which result in a cumulative bias of \(-\sum_{\tau=1}^{t}\boldsymbol{\Delta}_\tau^*\mathbf{K}_0\mathbf{K}_0^\top\mathbf{P}\). This bias amplifies linearly with steps unless history constraints are explicitly absorbed. Lemma 3.1 proves AlphaEdit strictly satisfies \(\sum_{\tau=1}^t \boldsymbol{\Delta}_\tau^* = \boldsymbol{\Delta}^*_{\text{total},t}\), and Proposition 3.2 shows this generalized form holds for any \(\mathbf{P}\) and \(\lambda\geq 0\) (MEMIT being a special case where \(\mathbf{P}=\mathbf{I},\lambda=0\)). Stability is thus redefined as a physical criterion: the cumulative update reconstructs OTE.

2. OTE→SE Constructive Mapping

With the equivalence criterion established, the authors provide a pipeline to construct stable SE algorithms from any OTE target with a convex regularization \(\mathcal{R}\). The core is the differential update \(\tilde{\boldsymbol{\Delta}}_t := \boldsymbol{\Delta}^*_{\text{total},t} - \boldsymbol{\Delta}^*_{\text{total},t-1}\). Proposition 3.4 proves that as long as the loss \(\mathcal{L}_t\) satisfies "shifted quadratic representability" (naturally met by least-squares), this difference is the unique solution to the convex subproblem \(\arg\min_{\boldsymbol{\Delta}} \ell_t(\boldsymbol{\Delta}) + \langle \nabla\mathcal{L}_t(\boldsymbol{\Delta}^*_{\text{total},t-1}),\boldsymbol{\Delta}\rangle + \mathcal{R}(\boldsymbol{\Delta}^*_{\text{total},t-1}+\boldsymbol{\Delta})\). This shifts SE design from heuristic constraint stacking to mechanical differentiation of an OTE target, eliminating cumulative drift.

3. Analytical Error Compensation for Post-processing Regularization (Algorithm 1)

Algorithms like PRUNE and RECT are problematic because they solve for \(\boldsymbol{\Delta}_t\) first and then apply a post-processing regularization \(\mathcal{R}_p(\boldsymbol{\Delta}_t)\), causing bias to accumulate over steps. The authors propose to explicitly maintain the cumulative error \(\mathbf{E}_t \leftarrow (\mathcal{R}_p(\boldsymbol{\Delta}_t)-\boldsymbol{\Delta}_t)\mathbf{C}_t\) and subtract it from the residual in the following step: \(\boldsymbol{\Delta}_t = (\mathbf{R}_t\mathbf{K}_t^\top - \mathbf{E}_{t-1})\mathbf{C}_t^{-1}\), where \(\mathbf{C}_t = \mathbf{C}_{t-1} + \mathbf{K}_t\mathbf{K}_t^\top\). This purely analytical term ensures \(\sum_\tau \mathcal{R}_p(\boldsymbol{\Delta}_\tau)\) still reconstructs the corresponding OTE solution.

Loss & Training

All methods use the Frobenius norm least squares loss \(\|(\mathbf{W}+\boldsymbol{\Delta})[\mathbf{K}_0\mid\mathbf{K}_\cdot] - [\mathbf{V}_0\mid\mathbf{V}_\cdot]\|_F^2\) with \(\lambda \mathbf{I}\) ridge regularization. The preservation set \((\mathbf{K}_0, \mathbf{V}_0)\) is estimated from 100,000 triplets sampled from Wikitext. Conflicting edits are handled per Proposition 3.5 by merging \(\mathbf{V}\) on the overlapping key set \(\mathcal{K}_o\), using the closed-form: \(\boldsymbol{\Delta}_t^* = (\mathbf{R}_t\mathbf{K}_t^\top - (\mathbf{V}_{\mathcal{B}_o^{(t)}}-\mathbf{W}_{t-1}\mathbf{K}_{\mathcal{B}_o^{(t)}})\mathbf{K}_{\mathcal{B}_o^{(t)}}^\top)(\mathbf{K}_{\mathcal{P}_{t-1}}\mathbf{K}_{\mathcal{P}_{t-1}}^\top + \mathbf{K}_t\mathbf{K}_t^\top - \mathbf{K}_{\mathcal{B}_o^{(t)}}\mathbf{K}_{\mathcal{B}_o^{(t)}}^\top)^{-1}\).

Key Experimental Results

Main Results

OTE-SE equivalence was verified across GPT-2 XL (1.5B), GPT-J (6B), LLaMA-3 (8B), and Qwen-2.5 (7B) on CounterFact and ZsRE. Setup: 100 edits per step, 20 steps = 2000 total edits.

Setting	Method	Eff.↑	Gen.↑	Spe.↑	Remarks
Fully Aligned	PRUNE (aligned)	99.87±0.03	94.91±0.22	79.90±0.20	OTE-SE Aligned + Compensation
Fully Aligned	RECT (aligned)	99.88±0.08	94.34±0.09	81.56±0.22	Same as above
Not OTE Aligned	PRUNE (Naive)	56.30±1.25	53.90±0.75	48.18±0.21	Naive repeated OTE
Not OTE Aligned	RECT (Naive)	60.35±1.12	58.35±1.25	46.80±0.20	Same as above

GLUE General Capability Test (LLaMA-3, Falsifying Null-Space):

Setting	SST	MMLU	MRPC	CoLA	RTE	NLI
Pre-edit	0.831	0.562	0.658	0.761	0.284	0.666
Full Normal Eq.	0.846	0.548	0.643	0.779	0.292	0.668
Null-Space Simplified	0.000	0.014	0.000	0.000	0.000	0.000

Ablation Study

Configuration	PRUNE (Eff./Gen./Spe.)	RECT (Eff./Gen./Spe.)	Description
Fully Aligned	99.87 / 94.91 / 79.90	99.88 / 94.34 / 81.56	Aligned + Comp.
No Err. Correction	99.82 / 95.22 / 80.19	96.98 / 83.60 / 84.86	Aligned only
Not OTE Aligned	56.30 / 53.90 / 48.18	60.35 / 58.35 / 46.80	Naive

Key Findings

• OTE alignment impacts editing success rates by \(\approx 40\) percentage points, identifying it as the true cause of stability; regularizations like null-space projection or spectral clipping become interference in long sequences.
• Error compensation is significantly more vital for RECT than for PRUNE. RECT informs information loss at every step by pruning relative weight ratios, necessitating explicit compensation via \(\mathbf{E}_t\).
• t-SNE visualization of the latent space indicates that while PRUNE (Naive)/RECT (Naive) suffer from significant distribution shifts after editing, these shifts nearly disappear after OTE alignment.

Highlights & Insights

• The "Memorize-the-Latest" counterexample effectively dismantles the "null-space narrative" of AlphaEdit: removing historical constraints while keeping \(\mathbf{P}\) causes immediate model collapse. This targeted falsification is more elegant than simply outperforming benchmarks on a dataset.
• Proposition 3.4 provides a mechanical pipeline to automatically upgrade any OTE target with convex regularization into a stable SE version, offering a clear path for future editor designs.
• The error compensation term \(\mathbf{E}_t\) in Algorithm 1 is conceptually similar to "deflation" in pseudo-inverses, linearizing non-convex biases introduced by post-processing into the residual.

Limitations & Future Work

• The theory assumes shifted quadratic representability, covering least-squares and ridge targets, but its applicability to non-quadratic losses (e.g., KL divergence, contrastive loss) remains unexplored.
• Editing is conducted only on one layer (FFN output projection). The equivalence for multi-layer joint editing needs further investigation.
• The "preservation set" \((\mathbf{K}_0,\mathbf{V}_0)\) is estimated from only 100,000 samples; the gap between the OTE solution and the "true distribution" alignment is not quantified.
• Future work: (1) Generalizing OTE-SE equivalence to dialogue/alignment scenarios with KL penalties; (2) Making error compensation learnable to handle non-linear biases from quantization or compression.

Related Work & Insights

• vs AlphaEdit (ICLR 2025): Both use null-space closed-form solutions, but AlphaEdit attributes stability to \(\mathbf{P}\). This paper proves OTE-SE alignment is the driver, making \(\mathbf{P}\) sufficient but not necessary; MEMIT (\(\mathbf{P}=\mathbf{I}\)) is equally stable if OTE-aligned.
• vs PRUNE / RECT: These rely on post-processing to limit update magnitudes. This paper reveals how this causes cumulative bias and provides Algorithm 1 to upgrade them, allowing PRUNE (aligned)/RECT (aligned) to reach AlphaEdit's performance.
• Approach: While others pursue "smarter constraints," this paper pursues "subtraction," proving which constraints are redundant and providing the simplest stable template.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Falsification via Memorize-the-Latest and OTE-SE equivalence proof provide an original theoretical perspective.
• Experimental Thoroughness: ⭐⭐⭐⭐ Four LLMs across two datasets plus GLUE tests; multi-layer joint editing and dialogue scenarios are missing.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear structure with the "labyrinth and thread" metaphor and consistent notation.
• Value: ⭐⭐⭐⭐⭐ Directly simplifies the design space for sequential editing and provides a mechanical construction process for stable SE.

Related Papers

• [ICML 2026] Revisiting Parameter-Based Knowledge Editing in Large Language Models: Theoretical Limits and Empirical Evidence
• [ACL 2025] Neuron-Level Sequential Editing for Large Language Models
• [AAAI 2026] Multiplicative Orthogonal Sequential Editing for Language Models (MOSE)
• [ICML 2026] Reverse-Engineering Model Editing on Language Models
• [ICML 2026] KORE: Enhancing Knowledge Injection for Large Multimodal Models via Knowledge-Oriented Controls