title: >- [Paper Note] On the Plasticity Collapse in Continual Machine Unlearning description: >- [ECCV 2026][AI Safety][Machine Unlearning] Identifies and theoretically characterizes plasticity collapse in continual machine unlearning, revealing forward failure (diminishing forgetting) and backward failure (spontaneous re-memorization) induced by geometric constraint accumulation in shared parameter subspaces. tags: - ECCV 2026 - AI Safety - Machine Unlearning - Continual Unlearning - Plasticity Collapse date: 2026-09-19 content_hash: 8d9e3fffb63e1d16
On the Plasticity Collapse in Continual Machine Unlearning¶
Conference: ECCV 2026
Paper: ECCV Official Link
Code: https://github.com/TIML-Group/Continual-Machine-Unlearning-Plasticity-Collapse
Area: AI Safety
Keywords: Machine Unlearning, Continual Unlearning, Plasticity Collapse, Geometric Constraints, Privacy Preservation
TL;DR¶
This paper uncovers and theoretically characterizes the fundamental phenomenon of "plasticity collapse" in continual machine unlearning, demonstrating that sequentially accumulating geometric constraints in parameter space causes operator-product expansion that generically forces forward failure (diminishing forgetting efficacy) and backward failure (spontaneous re-memorization).
Background & Motivation¶
With legislative frameworks such as the General Data Protection Regulation (GDPR) and California Consumer Privacy Act (CCPA) codifying the "right to be forgotten", selectively expunging the influence of specific training data or toxic/biased behaviors from trained deep neural networks has become a non-negotiable requirement for compliant AI deployment. The overwhelming majority of machine unlearning literature focuses almost exclusively on single-shot settings, under the idealized premise that a model only needs to service an isolated unlearning request during its lifecycle. In practical enterprise workloads—such as evolving content moderation policies or perpetual user deletion streams in recommender systems—models must process sequential unlearning requests continuously over time without resorting to prohibitively expensive retraining from scratch.
However, naive extensions of existing unlearning techniques to continual scenarios reveal severe, systemic failures. Unlike traditional continual learning, which deals with learning new knowledge under catastrophic forgetting and loss of plasticity, continual unlearning operates under an inverse optimization regime: the model must selectively excise knowledge through repeated gradient modifications, often without new informative training data to anchor representation spaces. Iteratively applying single-shot unlearning algorithms over time rapidly degrades the model's ability to purge newly specified targets, and worse, previously scrubbed information unexpectedly re-emerges in the network's predictive behaviors.
The authors approach this crisis by investigating the parameter-space optimization dynamics across sequential unlearning steps. They discover that sequential unlearning update operators induce geometric amplification along shared parameter subspaces, and the bounded optimization dynamics must neutralize this expansion via internal counteractions that inherently cripple future unlearning capacity or undo past forgetting. The core idea is to formally conceptualize this systemic breakdown as "plasticity collapse" in continual machine unlearning, proving theoretically via operator-product expansion that the accumulation of geometric constraints in saturated parameter subspaces inevitably manifests as two distinct failure modes: forward failure (progressive deterioration in forgetting quality for subsequent tasks) and backward failure (spontaneous re-memorization of previously forgotten data).
Method¶
Overall Architecture¶
In continual machine unlearning, an initial model parameterized by \(\theta_0 \in \mathbb{R}^d\) receives a sequence of \(T\) sequential unlearning requests. At each unlearning stage \(t \in \{1, \dots, T\}\), the system must remove the influence of a designated forget set \(F_t \subset \mathcal{D}\) while maintaining predictive accuracy on an optional retain set \(R_t \subset \mathcal{D} \setminus F_t\), updating parameter iterates from \(\theta_{t-1}\) to \(\theta_t\). Representative gradient-based unlearning algorithms—including Fine-tuning (FT), NegGrad+, RandomLabeling (RL), SalUn, and MUNBa—can be framed locally within a unified surrogate optimization objective \(J_t(\theta) := \lambda L_{R_t}(\theta) - L_{F_t}(\theta)\) with balance hyperparameter \(\lambda \ge 0\), executing gradient ascent on the forget data and gradient descent on the retain data.
As sequential unlearning tasks unfold, their input feature spaces induce overlapping forget spans in parameter space, spanning an active shared subspace \(W\). The composition of time-ordered unlearning update operators within this subspace triggers an exponential operator-product expansion. To prevent the parameter trajectory from diverging, the optimization dynamics are forced to neutralize this expansion—either by suppressing parameter motion along \(W\) or injecting opposing updates—directly inducing forward and backward failures, respectively.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Sequential Unlearning Requests<br/>F_1, F_2, ..., F_t arrive incrementally"] --> B["Unified Unlearning Objective<br/>J_t(θ) = λ L_R_t(θ) - L_F_t(θ)"]
B --> C["Operator-Product Expansion<br/>A_t:s compounds exponentially along W"]
C --> D["Parameter Boundedness Constraint<br/>Optimization must neutralize expansion"]
D -->|Mechanism 1: Suppress updates along W| E["Forward Failure<br/>Updates pushed away from W; forgetting decays"]
D -->|Mechanism 2: Inject cancelling updates| F["Backward Failure<br/>Historical updates undone; spontaneous re-memorization"]
E --> G["Diagnostic Quantities<br/>Energy Ratio ER_t & Projection Coeff CO_t"]
F --> G
Key Designs¶
1. Operator-Product Expansion in Shared Subspaces: The Core Source of Parameter Instability
To rigorously analyze continual unlearning dynamics, the authors establish a tractable theoretical foundation under a linear model and squared loss. A single gradient step on the unlearning objective yields an affine recursion: \(\theta_t = A_t \theta_{t-1} - c_t\), where \(A_t = I + M_{F_t} - \lambda M_{R_t}\), and \(M_{F_t}, M_{R_t} \succeq 0\) are the positive semidefinite Hessian/feature covariance matrices of the forget and retain sets, respectively. The time-ordered product of linear update operators is defined as \(A_{t:s} := A_t A_{t-1} \cdots A_s\), giving the closed-form iterate trajectory \(\theta_t = A_{t:s} \theta_{s-1} - \sum_{j=s}^t A_{t:j+1} c_j\). The authors prove that if consecutive unlearning tasks share a subspace \(W\) where the forget curvature dominates the scaled retain curvature (\(v^\top (M_{F_j} - \lambda M_{R_j}) v \ge \rho \|v\|^2\) for \(\rho > 0\)), the composite operator grows exponentially on \(W\): $\(\|A_{t:s} w\| \ge (1 + \rho)^{t - s + 1} \|w\|, \quad \forall w \in W\)$ Furthermore, using first-order linearization under the Neural Tangent Kernel (NTK) regime, this exponential expansion is extended to deep non-linear networks. Because the homogeneous component \(A_{t:s} \theta_{s-1}\) grows geometrically along \(W\), the unlearning trajectory can remain bounded only if the optimization dynamics counteract this explosive growth.
2. Decoupling the Dual Failure Modes: Dynamic Mechanisms of Forward Decay and Spontaneous Re-memorization
The paper reveals that neutralizing operator expansion generically forces the optimization process into two distinct, non-mutually-exclusive failure modes. The first stabilizing response is the progressive suppression of parameter updates along \(W\). Because the inputs of subsequent forget tasks \(F_t\) have feature projections within the shared span \(U_t \subset W\), reducing parameter displacement along \(W\) directly inhibits the model's capacity to adjust outputs on incoming forget requests, leading to Forward Failure (progressive degradation in unlearning efficacy). The second stabilizing response is internal cancellation: the accumulated historical forcing terms \(\sum_{j=s}^t A_{t:j+1} c_j\) must generate opposing components along \(W\) to offset the exponential expansion. However, these historical terms precisely encode earlier forgetting trajectories; injecting antagonistic updates along shared directions partially cancels earlier unlearning updates, causing model predictions on previously forgotten data \(F_s\) (\(s < t\)) to spontaneously drift back to their pre-unlearning states, leading to Backward Failure (spontaneous re-memorization).
3. Geometric Diagnostic Quantities: Online Probing via Energy Ratio (ER) and Projection Coefficient (CO)
To empirically observe and diagnose these subspace dynamics in deep networks, the authors construct an orthonormal basis \(Q \in \mathbb{R}^{d \times r}\) for the dominant shared subspace \(W\) using the top-\(r\) singular vectors of the stacked parameter update matrix \(U = [u_1 \mid \cdots \mid u_T] \in \mathbb{R}^{d \times T}\), where \(u_t = \theta_t - \theta_{t-1}\). They formulate two complementary geometric metrics: $\(\text{ER}_t := \frac{\|P_W u_t\|_2^2}{\|u_t\|_2^2} = \frac{\|Q Q^\top u_t\|_2^2}{\|u_t\|_2^2}, \qquad \text{CO}_t := Q^\top u_t\)$ The Energy Ratio (\(\text{ER}_t\)) quantifies the fraction of update energy concentrated inside the shared subspace \(W\). When forward failure occurs, the saturation of \(W\) forces updates into orthogonal directions, driving \(\text{ER}_t \to 0\) alongside a steep decline in the total update \(L_2\) norm \(\|u_t\|_2\), reflecting progressive parameter freezing. Conversely, the Projection Coefficient (\(\text{CO}_t\)) captures the signed magnitude of updates along the principal directions of \(W\). Tasks undergoing re-memorization display near-zero engagement (\(\text{CO}_t \approx 0\)), flanked by rapid, large-magnitude sign oscillations in \(\text{CO}_t\) across adjacent unlearning tasks, directly reflecting antagonistic interference and historical update cancellation.
Loss & Training¶
The framework investigates standard unlearning algorithms sequentially applied over \(T\) unlearning rounds, with each step governed by: $\(J_t(\theta) = \lambda L_{R_t}(\theta) - L_{F_t}(\theta)\)$ where losses denote empirical cross-entropy over the designated partitions. Specifically, Fine-tuning minimizes empirical risk on \(R_t\); NegGrad+ concurrently performs gradient ascent on \(F_t\) and descent on \(R_t\); RandomLabeling substitutes uniform random pseudo-labels on \(F_t\); SalUn restricts gradient updates to salient parameters identified via gradient masks; and MUNBa models utility preservation and forgetting as a cooperative Nash bargaining problem. Training follows standard optimizer conventions across consecutive tasks without interleaving new training tasks.
Key Experimental Results¶
Main Results¶
Empirical evaluations are conducted on Tiny-ImageNet and CIFAR-100 benchmarks using VGG-16-BN and PreResNet-110 architectures across random data forgetting (1% random subsets per task) and class-wise forgetting (entire classes purged per task). Unlearning efficacy is primarily evaluated via Forgetting Accuracy (accuracy on the forget set, where lower is better).
| Dataset / Architecture | Setting | Method | Single-Task Baseline Acc (%) | Continual Final Task Acc (%) | Degradation (Gain, %) |
|---|---|---|---|---|---|
| Tiny-ImageNet (VGG-16-BN) | Random Data (1%) | FT | 0.00 | ~32.5 | +32.5 (Forward Failure) |
| Tiny-ImageNet (VGG-16-BN) | Random Data (1%) | NegGrad+ | 0.00 | ~48.7 | +48.7 (Most Severe Collapse) |
| Tiny-ImageNet (VGG-16-BN) | Random Data (1%) | RL | 0.00 | ~28.0 | +28.0 (Forward Failure) |
| Tiny-ImageNet (VGG-16-BN) | Random Data (1%) | SalUn | 0.00 | ~26.5 | +26.5 (Forward Failure) |
| Tiny-ImageNet (VGG-16-BN) | Random Data (1%) | MUNBa | 0.00 | ~30.3 | +30.3 (Forward Failure) |
| Tiny-ImageNet (VGG-16-BN) | Class-wise | FT | ~0.00 | ~62.0 | +62.0 (Forward Failure) |
| Tiny-ImageNet (VGG-16-BN) | Class-wise | NegGrad+ | ~0.00 | ~78.0 | +78.0 (Forward Failure) |
| Tiny-ImageNet (VGG-16-BN) | Class-wise | MUNBa | ~0.00 | ~45.0 | +45.0 (Dual Failures) |
(Note: Under random data forgetting, the final task in the continual setting exhibits an average forgetting accuracy increase of approximately 33.2% compared to single-task unlearning across all methods. Under class-wise unlearning, RL and SalUn maintain low current-task accuracy but hide massive backward re-memorization).
Ablation Study¶
Ablation and diagnostic breakdown across forgetting modes and methods in class-wise forgetting on Tiny-ImageNet (VGG-16-BN):
| Unlearning Algorithm | Setting | Dominant Failure Mode | Diagnostic Indicator (\(\text{ER}_t\) / \(\text{CO}_t\)) | Parameter Update Norm \(\|u_t\|_2\) |
|---|---|---|---|---|
| FT / NegGrad+ | Random / Class-wise | Forward Failure | \(\text{ER}_t \to 0\) (steep monotonic decay) | Systematic decay; parameter space "freezes" |
| RL (RandomLabeling) | Class-wise | Backward Failure (Re-memorization) | Marked sign oscillations in \(\text{CO}_t\) | Dips noticeably during re-memorization stages |
| SalUn (Saliency-guided) | Class-wise (Tasks 35–60) | Backward Failure (Re-memorization) | Historical accuracy turns red; \(\text{CO}_t \approx 0\) | Confined parameter budget forces antagonism |
| MUNBa (Nash Bargaining) | Class-wise | Co-existing Forward & Backward | Decay in \(\text{ER}_t\) with off-diagonal accuracy rebound | Continuous drop with acute local cancellations |
Key Findings¶
- Ubiquity of Forward Failure: Under random data unlearning, every single tested baseline suffers from forward collapse; as unlearning requests accumulate, the final task unlearning accuracy is roughly 33.2% worse on average than single-shot unlearning, proving that existing unlearning paradigms cannot sustain long-term effectiveness.
- Stealth and Severity of Backward Re-memorization: In class-wise unlearning, RL and SalUn ostensibly retain near-zero forgetting accuracy on current tasks, creating an illusion of success. However, historical heatmaps reveal that earlier forgotten classes (e.g., tasks 35–60 for SalUn) spontaneously revert to high accuracy (red entries on off-diagonals), silently compromising user privacy guarantees.
- Empirical Validation of Theoretical Dynamics: As predicted by Theorems 1 and 2, the decay curve of the Energy Ratio (\(\text{ER}_t \to 0\)) tightly tracks the worsening unlearning accuracy, while pronounced sign alternations in \(\text{CO}_t\) align precisely with the occurrence of re-memorization events, confirming the operator-product saturation thesis.
Highlights & Insights¶
- Paradigm Shift in Problem Formulation: Breaks free from the artificial "single-shot unlearning" convention to formally conceptualize the realities of continual multi-step unlearning, pinpointing the critical vulnerability of "plasticity collapse".
- Rigorous Mathematical Grounding: Links empirical failures directly to affine dynamical systems, operator-product expansion, and NTK linearization, showing that forward degradation and backward re-memorization are complementary mathematical consequences of maintaining bounded parameters under expansive operators.
- Actionable Online Diagnostic Tooling: The formulated Energy Ratio (ER) and Projection Coefficient (CO) require no raw historical data access, offering lightweight, practical online geometric monitors and regularization signals for future plasticity-preserving algorithms.
Limitations & Future Work¶
- Architectural Scope: The experimental validation is established on convolutional vision benchmarks (VGG-16, PreResNet-110 on Tiny-ImageNet/CIFAR-100); empirical characterization in generative vision architectures (e.g., Diffusion Models) and Large Language Models (LLMs) remains to be demonstrated.
- Linearization Assumptions: The theoretical analysis relies on first-order Taylor expansion and NTK assumptions; while accurate locally, severe non-linear curvature shifts over extensive multi-epoch unlearning regimes may require broader non-convex optimization tools.
- Future Algorithmic Pathways: Designing continual unlearning algorithms with explicit subspace orthogonalization, adaptive rank expansions, or periodic parameter rejuvenation (e.g., shrink-and-perturb resets) to preserve model unlearning plasticity.
Related Work & Insights¶
- vs. Single-Shot Machine Unlearning (e.g., SISA, NegGrad+, SalUn): Prior works operate under isolated deletion requests; this paper demonstrates that sequentially compounding these methods induces catastrophic geometric saturation in parameter space.
- vs. Loss of Plasticity in Continual Learning (e.g., Dohare et al., Kumar et al.): Continual learning studies progressive capacity loss when acquiring new tasks (via dormant neurons or implicit rank collapse); this work identifies an entirely distinct mechanism arising from purely subtractive gradient ascent dynamics and operator-product divergence.
- vs. Incremental Unlearning (FG-OrIU): FG-OrIU explored incremental unlearning strictly under LoRA parameter-efficient fine-tuning on ViT models; this paper characterizes the universal, architecture-agnostic dynamical phenomenon occurring in full parameter spaces.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates and mathematically defines the fundamental phenomenon of plasticity collapse and its dual failure modes in continual unlearning.
- Experimental Thoroughness: ⭐⭐⭐⭐ Comprehensive evaluation on Tiny-ImageNet/CIFAR-100 with extensive diagnostic metrics and heatmap tracking; extension to LLMs would make it comprehensive.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear mathematical exposition, elegant geometric intuition, and tight cohesion between theoretical predictions and empirical verification.
- Value: ⭐⭐⭐⭐⭐ Exposes a critical vulnerability in current unlearning deployments and provides the diagnostic groundwork for building reliable, long-term privacy-preserving systems.