Generalizable Lifelong Model Editing via Preference Optimization¶
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/ekgus9/GLIME
Area: Knowledge Editing
Keywords: Knowledge Editing, Lifelong Learning, Preference Optimization, Orthogonal Gradient Projection, Autoregressive Decoding Evaluation
TL;DR¶
To tackle superficial prompt overfitting and catastrophic forgetting in lifelong knowledge editing, GLIME formulates continual editing as a joint optimization problem combining factual injection, edit-independent preference optimization for behavior-level regularization, experience replay, and randomized orthogonal gradient projection, achieving superior generalization and downstream portability under realistic autoregressive decoding.
Background & Motivation¶
Large language models (LLMs) acquire vast amounts of factual knowledge and strong reasoning capabilities during pretraining. However, real-world facts change constantly, necessitating targeted knowledge updates without prohibitive full retraining. While parameter-level knowledge editing has emerged as a cost-effective paradigm, real-world deployment requires handling continual streams of updates over time rather than isolated one-off modifications.
Under this lifelong editing scenario, existing methods encounter two severe failure modes. First, traditional locate-and-edit or optimization-based approaches optimize only the conditional likelihood of target tokens under the specific edit prompt; consequently, models overfit to surface lexical cues and generalize poorly to semantically paraphrased queries or downstream multi-hop reasoning. Second, as edits accumulate, consecutive parameter updates push the weights away from the pretrained distribution, leading to catastrophic forgetting of previously edited facts and severe collapse of general linguistic capabilities. Compounding these issues, most prior benchmarks rely on teacher forcing (TF) evaluation, where exposure to ground-truth token prefixes masks generation degradation and overestimates the actual usability of edited knowledge.
To resolve the tension between factual injection, memory retention, and generative stability, lifelong editing cannot rely solely on increasing target token probabilities. Core idea: Formulate lifelong knowledge editing as a multi-objective joint optimization problem that integrates factual injection with prompt-independent Continual Preference Optimization (CPO) to regularize generation behavior, while employing experience replay and an online randomized orthogonal gradient constraint (GC) to shield historical parameter subspaces from catastrophic interference.
Method¶
Overall Architecture¶
GLIME processes a stream of edit requests by jointly optimizing three objectives over selected MLP layers: the target factual edit loss, an experience replay loss sampled from past edits, and a logistic preference loss defined over an edit-independent alignment dataset. The combined gradient is then projected onto the orthogonal complement of the accumulated right-subspace of previous updates before applying the parameter update, ensuring non-interfering parameter trajectories.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Continual Edit Stream e_t<br/>(x_t, y_t^new)"] --> B["Multi-Objective Joint Optimization<br/>Edit Loss + Replay Loss + Preference Loss"]
D["Independent Preference Pair (u, y+, y-)<br/>Behavior-Level Regularization Anchor"] --> B
E["Replay Buffer B_{t-1}<br/>Random Historical Edit Retrieval"] --> B
B --> C["Joint Objective Gradient<br/>G_t = โ_W L(ฮธ_t)"]
C --> F["Orthogonal Gradient Projection<br/>G~_t = G_t(I - B B^T)"]
G["Historical Orthogonal Basis B_{t-1}<br/>Subspace of Prior Knowledge Directions"] --> F
F --> H["Parameter Update & Online Basis Update<br/>W โ W - ฮท G~_t"]
H --> I["Autoregressive Decoding<br/>Robust Paraphrase & Multi-hop Portability"]
Key Designs¶
1. Continual Preference Optimization (CPO): Behavior-Level Distribution Anchoring
Standard editing objectives optimize \(L_{\text{edit}}(\theta; e) = -\log p_\theta(y^{\text{new}} \mid x)\), which locally distorts token probabilities around the prompt \(x\) and degrades autoregressive generation under exposure bias. To prevent the model from collapsing into superficial prompt memorization, GLIME incorporates preference optimization as an external behavioral regularizer. Specifically, for an edit-independent context \(u\) sampled from general alignment datasets (e.g., OpenHermesPreferences), the model is optimized over preferred response \(y^+\) and dispreferred response \(y^-\) via a logistic preference loss:
Because \(u\) is independent of the edit trigger \(x\), it serves as a stabilizing anchor that constrains the output distribution to adhere to pre-edit conversational and reasoning norms, forcing parameter updates to discover shared generalizable manifolds rather than brittle overfitted solutions.
2. Experience Replay (Replay): Alleviating Catastrophic Forgetting
Continually adapting weights solely on the current instance \(e_t\) invariably overwrites previously edited knowledge \(\{e_i\}_{i<t}\). GLIME maintains a replay buffer \(\mathcal{B}_t\) storing a subset of past edit requests. At each update step \(t\), an auxiliary replay loss is computed over a mini-batch \(R \sim \mathcal{B}_t\):
The combined objective is \(L(\theta_t) = L_{\text{edit}}(\theta_t; e_t) + L_{\text{pref}}(\theta_t) + \lambda_{\text{rep}} L_{\text{replay}}(\theta_t)\), where \(\lambda_{\text{rep}}=0.1\) balances new fact absorption with historical retention.
3. Orthogonal Basis Gradient Constraint (GC): Structurally Eliminating Parameter Interference
Unconstrained updates for new edits interfere with parameter directions essential for earlier edits. GLIME maintains an orthonormal basis matrix \(B_{t-1} \in \mathbb{R}^{n \times r}\) (\(B_{t-1}^\top B_{t-1} = I\)) for editable MLP weight matrices \(W \in \mathbb{R}^{m \times n}\) across layers 4โ8. Before updating parameters, the raw gradient \(G_t = \nabla_W L(\theta_t)\) is projected onto the orthogonal complement of \(B_{t-1}\):
This guarantees \(\tilde{G}_t B_{t-1} = 0\), strictly confining parameter modifications to subspaces orthogonal to previous edits. To bypass computationally prohibitive full SVD on large layers, GLIME generates a low-rank sketch \(Y_t = \tilde{G}_t^\top \Omega_t \in \mathbb{R}^{n \times k}\) using a random Gaussian matrix \(\Omega_t \in \mathbb{R}^{m \times k}\) (\(k=256\)), performs thin-QR factorizations to extract new basis vectors, and maintains a compact rank \(r=4\) with a FIFO queue, cutting basis computation time by approximately \(6\times\) compared to exact SVD.
A Worked Example¶
Consider the 1,000th edit updating "Frank R. Strayer is a citizen of Canada" (originally USA): 1. Optimization Step: Model receives the context-free cloze prompt, samples an unrelated medical query preference pair \((u, y^+, y^-)\) from OpenHermesPreferences, and retrieves 3 historical edits from the replay buffer. 2. Gradient Projection: Joint gradients for MLP layers 4โ8 are projected via \(\tilde{G}_t = G_t(I - B_{t-1}B_{t-1}^\top)\) to strip out components aligned with previous 999 edits, followed by gradient descent update and basis registration. 3. Autoregressive Testing: When prompted with a paraphrased question ("What is the country of citizenship of Frank R. Strayer?"), GLIME fluently generates "Frank R. Strayer is a citizen of Canada." On multi-hop reasoning ("Who is the current head of the government in the country of citizenship of Frank R. Strayer?"), it correctly chains the new nationality to generate "Justin Trudeau", while retaining unrelated world knowledge ("Who founded Tudor dynasty?" \(\rightarrow\) "Henry VII") without repetitive stuttering or hallucination.
Loss & Training¶
Parameter updates are applied sequentially (batch size 1) using AdamW with learning rate \(5 \times 10^{-5}\) for 3 epochs per edit on a single RTX A6000 GPU. Replay buffer mini-batch size \(|R|=3\), replay loss weight \(\lambda_{\text{rep}}=0.1\), sketch dimension \(k=256\), and maximum basis rank \(r=4\). Edits are localized to MLP layers 4, 5, 6, 7, and 8 of LLaMA-3.1-8B-Instruct and Qwen2.5-7B-Instruct.
Key Experimental Results¶
Main Results¶
Evaluation was conducted on MQuAKE-Remastered (CF-3k split, 1,000 test samples) and ZSRE (743 test samples). Along with traditional Teacher Forcing (TF), performance is assessed under realistic Autoregressive Decoding (AD), alongside Multiple-Choice QA (MC) and Multi-hop Reasoning QA (MR) to evaluate Portability.
| Model & Method | MQuAKE Rel(AD) | MQuAKE Gen(AD) | MQuAKE Loc(AD) | Portability MC | Portability MR | MQuAKE AVG | ZSRE Rel(AD) | ZSRE Gen(AD) | ZSRE Loc(AD) | ZSRE AVG |
|---|---|---|---|---|---|---|---|---|---|---|
| LLaMA-3.1-8B | ||||||||||
| FT-L | 0.035 | 0.038 | 0.000 | 0.000 | 0.003 | 0.016 | 0.106 | 0.094 | 0.000 | 0.078 |
| R-ROME | 0.074 | 0.087 | 0.010 | 0.000 | 0.008 | 0.030 | 0.004 | 0.087 | 0.000 | 0.025 |
| GRACE | 0.035 | 0.046 | 0.714 | 0.010 | 0.060 | 0.270 | 0.021 | 0.014 | 0.203 | 0.220 |
| MEMIT | 0.014 | 0.014 | 0.010 | 0.000 | 0.020 | 0.022 | 0.000 | 0.000 | 0.000 | 0.000 |
| WISE | 0.193 | 0.147 | 0.121 | 0.000 | 0.028 | 0.243 | 0.073 | 0.055 | 0.012 | 0.201 |
| AlphaEdit | 0.604 | 0.514 | 0.209 | 0.434 | 0.079 | 0.499 | 0.538 | 0.451 | 0.187 | 0.544 |
| GLIME (Ours) | 0.830 | 0.800 | 0.379 | 0.633 | 0.184 | 0.700 | 0.670 | 0.598 | 0.133 | 0.589 |
| Qwen2.5-7B | ||||||||||
| FT-L | 0.140 | 0.124 | 0.079 | 0.000 | 0.033 | 0.070 | 0.006 | 0.003 | 0.003 | 0.037 |
| R-ROME | 0.001 | 0.001 | 0.001 | 0.000 | 0.006 | 0.000 | 0.054 | 0.050 | 0.003 | 0.101 |
| GRACE | 0.050 | 0.055 | 0.684 | 0.009 | 0.079 | 0.243 | 0.024 | 0.020 | 0.100 | 0.210 |
| MEMIT | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 |
| WISE | 0.274 | 0.202 | 0.124 | 0.008 | 0.063 | 0.264 | 0.231 | 0.197 | 0.040 | 0.269 |
| AlphaEdit | 0.201 | 0.389 | 0.513 | 0.192 | 0.081 | 0.430 | 0.354 | 0.277 | 0.113 | 0.497 |
| GLIME (Ours) | 0.772 | 0.812 | 0.364 | 0.475 | 0.168 | 0.649 | 0.417 | 0.373 | 0.093 | 0.518 |
Ablation Study¶
Full component ablation of GLIME (CPO: Continual Preference Optimization, Replay: Experience Replay, GC: Gradient-space Constraint) on LLaMA-3.1-8B-Instruct on MQuAKE:
| CPO | Replay | GC | Reliability (AD) | Generalization (AD) | Locality (AD) | Portability (MC) | Portability (MR) |
|---|---|---|---|---|---|---|---|
| โ | โ | โ | 0.830 | 0.800 | 0.379 | 0.633 | 0.184 |
| โ | โ | 0.826 (-0.004) | 0.745 (-0.055) | 0.187 (-0.192) | 0.014 (-0.619) | 0.112 (-0.072) | |
| โ | โ | 0.657 (-0.173) | 0.633 (-0.167) | 0.326 (-0.053) | 0.531 (-0.102) | 0.096 (-0.088) | |
| โ | โ | 0.800 (-0.030) | 0.786 (-0.014) | 0.385 (+0.006) | 0.412 (-0.221) | 0.098 (-0.086) | |
| โ | 0.597 (-0.233) | 0.601 (-0.199) | 0.391 (+0.012) | 0.530 (-0.103) | 0.168 (-0.016) | ||
| โ | 0.624 (-0.206) | 0.615 (-0.185) | 0.137 (-0.242) | 0.063 (-0.570) | 0.109 (-0.075) | ||
| โ | 0.528 (-0.302) | 0.533 (-0.267) | 0.259 (-0.120) | 0.313 (-0.320) | 0.120 (-0.064) | ||
| 0.158 (-0.672) | 0.120 (-0.680) | 0.141 (-0.238) | 0.098 (-0.535) | 0.087 (-0.097) |
Key Findings¶
- CPO is Essential for Downstream Portability: Removing CPO while keeping Replay and GC retains high Reliability (0.826), but causes Portability MC to plummet from 0.633 to 0.014 (-0.619). This proves continual learning primitives alone merely retain raw memorization, whereas preference optimization enables transferable application of edited facts.
- Teacher Forcing Masks Catastrophic Model Drift: Baselines exhibiting decent TF performance degrade sharply under autoregressive generation. On MQuAKE with LLaMA-3.1-8B, AlphaEdit scores 0.912 on TF Reliability but drops to 0.604 under AD (and from 0.986 to 0.201 on Qwen2.5-7B). GLIME retains 0.830 under AD, demonstrating genuine knowledge internalization.
- Robust Long-Horizon Scalability up to 5,000 Edits: Under extended sequential editing stress tests, AlphaEdit undergoes a steep performance collapse after ~3,000 edits, whereas GLIME maintains stable trajectories across all 5,000 steps due to non-interfering gradient projection.
- Preservation of General Capabilities: Across 5 diverse NLP benchmarks (Winogrande, ARC, MathQA, SQuADv2, IFEval), GLIME avoids the severe degradation seen in FT-L, R-ROME, and MEMIT, staying closely aligned with pre-edit Base performance (e.g., ARC 0.772 vs Base 0.802; IFEval 0.486 vs Base 0.532).
Highlights & Insights¶
- Preference Optimization as a Behavioral Regularizer: Repurposing alignment preference pairs (DPO/ORPO loss formulation) as an edit-independent anchor provides a novel mechanism to prevent the language generation manifold from warping during parameter manipulation.
- Randomized Sketching for Scalable Gradient Subspace Projection: Rather than computing computationally heavy full SVD across large weight matrices, applying randomized Gaussian sketching with thin-QR factorization enables rapid online orthogonal basis updates in 8.9 seconds per edit.
- Synergy Between Replay and Subspace Isolation: Empirical analysis shows experience replay alone fails to safeguard editing (applying replay to AlphaEdit even hurts accuracy); true stability requires the geometric protection of gradient projection combined with replay and behavioral regularization.
Limitations & Future Work¶
- Memory Footprint of Basis and Replay Storage: Storing the replay buffer and low-rank basis vectors incurs added memory overhead. More scalable compression and pruning schemes will be needed for lifelong deployment spanning tens of thousands of edits.
- Exploration on Larger and Diverse Architectures: Evaluations are constrained to 7Bโ8B dense Transformer architectures editing specific MLP blocks; efficacy on 70B+ scale models or sparse Mixture-of-Experts (MoE) networks remains unverified.
- Inter-dependent Knowledge Conflicts: Benchmarks focus on mutually disjoint factual relations, leaving the dynamics of hierarchical, mutually conflicting, or causal multi-step updates for future investigation.
Related Work & Insights¶
- vs AlphaEdit: While AlphaEdit enforces a null-space projection of existing knowledge to mitigate localization noise, it does not constrain generation behavior, leading to severe exposure bias under AD; GLIME integrates continual preference optimization, vastly outperforming AlphaEdit in portability and autoregressive generation.
- vs WISE / GRACE: WISE and GRACE use side memory networks or adapter caches, which suffer from knowledge bleeding (e.g., repetitively emitting target entities on unrelated prompts) or retrieval refusals; GLIME directly updates backbone MLP weights under behavior and subspace regularization.
- vs R-ROME / MEMIT: Sequential closed-form MLP updates in R-ROME and MEMIT cause rapid parameter divergence and complete gibberish generation under lifelong streams; GLIME provides smooth, non-collapsing updates through online projected gradient descent.
Rating¶
- Novelty: โญโญโญโญโญ Elegant conceptualization of preference optimization as a lifelong editing regularizer.
- Experimental Thoroughness: โญโญโญโญโญ Evaluates both TF and AD decoding, downstream QA, general benchmarks, 5,000-step scaling, and LLM-as-a-Judge.
- Writing Quality: โญโญโญโญโญ Rigorous mathematical framing, detailed pseudocode, and thorough qualitative and ablation analysis.
- Value: โญโญโญโญโญ Sets a new standard for evaluating and developing generalizable, non-collapsing lifelong model editors.