---

title: >- [Paper Note] Gradient sparsity regularization for training unlearning-compatible models description: >- [ECCV 2026][AI Safety][Machine Unlearning] Proposes gradient l1-norm regularization during training to minimize worst-case loss sharpness and cumulative curvature, producing unlearning-compatible models and mitigating backdoor gradient spikes. tags: - ECCV 2026 - AI Safety - Machine Unlearning - Gradient Sparsity - Loss Sharpness date: 2026-09-19 content_hash: cfd5ee74eff35d3e

Gradient sparsity regularization for training unlearning-compatible models

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Area: AI Safety
Keywords: machine unlearning, gradient sparsity, loss sharpness, unrolling SGD, backdoor defense

TL;DR

Addressing the severe performance gap in approximate machine unlearning and the theoretical barrier of cumulative curvature in unrolling SGD, this paper proposes adding an \(\ell_1\)-norm gradient regularizer during pre-training to minimize worst-case loss sharpness, producing flatter optimization trajectories and yielding models that are inherently unlearning-compatible while dampening backdoor gradient spikes.

Background & Motivation

With the enforcement of modern privacy legislation such as the General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA), users possess the legal "right to be forgotten", requiring service providers to purge individual data and any derived intelligence from deployed deep models. Beyond legal compliance, mitigating the impact of poisoned training instances in backdoor attacks similarly necessitates selectively removing a target forget set while preserving general performance on the retain set. This task is known as machine unlearning (MU). The gold-standard solution is exact unlearning—retraining the entire network from scratch on the retain set. However, exact retraining discards valuable representations acquired over large-scale retain data and imposes prohibitive computational costs. Consequently, numerous approximate MU methods have emerged, leveraging techniques like fine-tuning (FT), gradient ascent (GA), Fisher information dampening, weight pruning, and representation distillation. Nevertheless, a substantial performance gap in both unlearning efficacy and retain accuracy persists between these approximations and exact retraining.

Examining the root cause of this performance divide reveals an overlooked structural bottleneck: virtually all existing research treats unlearning as a post-hoc patching exercise applied to models already trained via standard empirical risk minimization. However, theoretical analysis of unrolling SGD shows that the unlearning error incurred when reversing SGD steps is strictly proportional to the accumulated loss curvature (the average maximum eigenvalue of the Hessian matrix along the training trajectory). Furthermore, PAC-Bayesian analysis establishes that empirical loss sharpness directly bounds the generalization error, quantifying the extent of sample-specific memorization. When a network is trained through standard optimization, it traverses jagged, sharp regions of the loss landscape, etching deep idiosyncratic details of individual samples into coupled weight matrices. Reversing these parameter updates post-hoc without destabilizing retain knowledge becomes fundamentally ill-posed.

To resolve this limitation, the authors invert the conventional paradigm and formulate a foundational question: can we design an optimization strategy during the initial training stage that flattens the learning trajectory and naturally produces unlearning-compatible models? By investigating how loss landscape sharpness is modulated under different perturbation norms, the authors prove that sharpness defined over \(\ell_\infty\) parameter perturbations represents the theoretical worst-case sharpness, and its first-order approximation corresponds directly to the \(\ell_1\) norm of the gradient. Core idea: regularize the training objective with the \(\ell_1\) norm of the gradient to encourage gradient sparsity, flattening the optimization trajectory to minimize cumulative curvature and sample-specific memorization for effortless post-hoc unlearning, while simultaneously dampening anomalous gradient spikes triggered by poisoned data.

Method

Overall Architecture

Rather than proposing an isolated unlearning post-processing algorithm, the proposed approach intervenes at the primary training phase. Given a labeled training set, model parameters are updated by minimizing a composite objective comprising the original task loss and an explicit \(\ell_1\)-norm gradient penalty. By enforcing gradient sparsity, the model is guided toward wide, flat minima characterized by minimal cumulative Hessian curvature. When the model subsequently encounters unlearning requests, mainstream approximate MU baselines (such as FT, GA, weight sparsity, or influence functions) achieve near-exact unlearning performance with minimal retain degradation. Concurrently, the \(\ell_1\) penalty acts as an automated regularizing filter against high-magnitude gradient spikes induced by backdoor triggers.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Training data input<br/>retain set + forget/poisoned set"] --> B["Forward-backward pass & gradient computation"]
    B --> C["Gradient sparsity regularization objective<br/>joint empirical loss & gradient L1 norm"]
    C --> D["Flat trajectory optimization<br/>gradient spike dampening & curvature suppression"]
    D --> E["Unlearning-compatible model output"]
    E --> F["Downstream rapid approximate unlearning<br/>FT / GA / WS / IU / FF execution"]

Key Designs

1. Worst-case loss sharpness under norm duality: deriving the optimal flatness regularizer

To ensure minimal error when reversing SGD and suppress sample-specific memorization, the learning process must minimize the most stringent proxy of loss sharpness. For a perturbation radius \(\rho > 0\), the \(p\)-sharpness is defined as \(\mathcal{S}_p(\theta) := \max_{\|\epsilon\|_p \le \rho} (\ell(\theta+\epsilon) - \ell(\theta))\). Utilizing a first-order Taylor expansion combined with Hölder's inequality, the worst-case local loss increase is tightly bounded by the dual norm of the gradient vector: $\(\max_{\|\epsilon\|_p \le \rho} \epsilon^\top \nabla\ell(\theta) = \rho \|\nabla\ell(\theta)\|_q\)$ where \(\frac{1}{p} + \frac{1}{q} = 1\). When examining canonical perturbation norms \(p \in \{1, 2, \infty\}\), their dual exponents correspond to \(q \in \{\infty, 2, 1\}\), respectively. Because standard vector norms satisfy \(\|\nabla\ell(\theta)\|_1 \ge \|\nabla\ell(\theta)\|_2 \ge \|\nabla\ell(\theta)\|_\infty\) for any gradient vector, the first-order sharpness approximations exhibit a definitive ordering: $\(\mathcal{S}_1(\theta) \le \mathcal{S}_2(\theta) \le \mathcal{S}_\infty(\theta) \approx \rho \|\nabla\ell(\theta)\|_1\)$ This mathematical duality confirms that perturbations bounded by the \(\ell_\infty\) norm produce the worst-case sharpness, and penalizing its first-order surrogate—the \(\ell_1\) norm of the gradient—enforces the strictest flatness constraint across the entire optimization trajectory, directly reducing the accumulated curvature required for SGD reversal.

2. Gradient sparsity objective and PAC-Bayes generalization bound minimization

Having established \(\ell_\infty\)-sharpness as the dominant metric, the authors examine its connection to generalization through PAC-Bayesian complexity. Under isotropic Gaussian parameter perturbations, the additive complexity term satisfies \(F_\infty(\cdot) \le F_2(\cdot) \le F_1(\cdot)\), proving that \(\ell_\infty\)-sharpness yields the tightest upper bound on population error. Minimizing this term prevents the model from over-indexing on idiosyncratic training noise. The unlearning-compatible training objective is thus formulated as: $\(\ell_{\mathcal{D}}^{\text{unc}}(\theta) = \ell_{\mathcal{D}}(\theta) + \rho \|\nabla\ell_{\mathcal{D}}(\theta)\|_1\)$ where the hyper-parameter \(\rho\) governs regularization intensity. Geometrically, the \(\ell_1\) norm acts as a sparsity-inducing penalty on gradient components, encouraging non-essential parameter derivatives to vanish at convergence. This sparse gradient allocation decouples feature dependencies across training samples, allowing individual subsets to be pruned or adapted during unlearning without triggering catastrophic disruption to retain representations.

3. Gradient spike dampening mechanism for backdoor robustness

The gradient sparsity regularizer provides substantial auxiliary benefits in security settings, specifically against data poisoning and backdoor attacks. Malicious triggers force networks to associate unrelated background patterns with target labels, distorting clean decision boundaries and producing high-magnitude gradient spikes during back-propagation. Under standard training, models greedily assimilate these malicious artifacts along steep gradient surges. In contrast, the \(\ell_1\) norm imposes a substantial penalty that scales linearly with gradient magnitude across all dimensions, acting as an automatic high-amplitude gradient filter. By dampening these anomalous surges, the model naturally absorbs significantly less toxic knowledge during training, simplifying post-hoc removal while preserving clean data performance.

4. Dual finite-difference approximation for computational efficiency

Directly optimizing \(\ell_{\mathcal{D}}^{\text{unc}}(\theta)\) requires computing Hessian-vector products (HVPs), demanding computation graph retention and a double back-propagation pass per iteration (one forward and two backward passes, \(F+2B\)), which increases peak memory by approximately \(1.2\times\)–\(2.0\times\). To mitigate this overhead, the authors devise an efficient finite-difference approximation along the maximizing dual direction \(v = \operatorname{sign}(g)\) where \(g = \nabla_\theta \ell_{\mathcal{D}}(\theta)\): $\(\|g\|_1 = \max_{\|v\|_\infty \le 1} v^\top g \approx \frac{\ell_{\mathcal{D}}(\theta + \varepsilon \operatorname{sign}(g)) - \ell_{\mathcal{D}}(\theta)}{\varepsilon}\)$ By evaluating loss gradients at perturbed parameters \(\theta' = \theta + \varepsilon \operatorname{sign}(g)\), the method eliminates the need to retain higher-order autograd graphs, completely removing peak memory bloat while preserving unlearning compatibility.

Key Experimental Results

Main Results

The framework was comprehensively evaluated on CIFAR-10, CIFAR-100, SVHN, and ImageNet using ResNet-18, VGG-16, ViT-Small, and MobileNetV2 across both random data forgetting (10% random subset) and class forgetting (entire class deletion). Evaluated MU algorithms include Weight Sparsity (WS), Gradient Ascent (GA), Influence Unlearning (IU), Fisher Forgetting (FF), and Fine-Tuning (FT). Metrics include Retain Accuracy (RA), Forget Accuracy (FA), Membership Inference Attack efficacy (MIA-Efficacy), Test Accuracy (TA), and Average Gap (Avg. Gap, absolute deviation from exact Retrain, lower is better). Representative results for ResNet-18 on CIFAR-10 are detailed below:

Forgetting Task	MU Method	RA (%) [Std / GS]	FA (%) [Std / GS]	MIA-Efficacy (%) [Std / GS]	TA (%) [Std / GS]	Avg. Gap [Std → GS]
Random Forgetting (10%)	Retrain (Gold Standard)	100.0 / 100.0	94.60 / 94.51	12.73 / 11.82	94.23 / 94.14	0.00 → 0.00
	Weight Sparsity (WS)	88.36 / 96.78	86.22 / 94.36	15.44 / 10.64	85.47 / 90.46	7.87 → 2.06
	Gradient Ascent (GA)	98.88 / 99.41	98.31 / 99.11	2.76 / 1.44	93.07 / 94.23	3.99 → 3.91
	Influence Unlearning (IU)	99.38 / 96.40	99.18 / 96.24	2.04 / 6.76	93.93 / 89.18	4.05 → 3.84
	Fisher Forgetting (FF)	86.42 / 86.63	86.02 / 86.02	20.64 / 18.69	80.98 / 80.74	10.83 → 10.59
	Fine-Tuning (FT)	99.91 / 88.50	98.96 / 86.80	2.91 / 20.07	94.25 / 83.81	3.57 → 3.37
Class Forgetting (Full Class)	Retrain (Gold Standard)	100.0 / 100.0	0.00 / 0.00	100.0 / 100.0	94.90 / 94.59	0.00 → 0.00
	Weight Sparsity (WS)	87.10 / 87.39	0.00 / 0.00	100.0 / 100.0	84.89 / 84.08	5.73 → 2.68
	Gradient Ascent (GA)	11.11 / 11.11	0.00 / 0.00	0.00 / 100.0	11.11 / 11.11	68.17 → 43.15
	Influence Unlearning (IU)	99.47 / 93.71	90.02 / 6.58	28.33 / 97.29	94.67 / 86.80	40.61 → 5.76
	Fisher Forgetting (FF)	99.51 / 99.34	11.04 / 0.02	100.0 / 100.0	94.87 / 93.70	2.89 → 0.23
	Fine-Tuning (FT)	99.92 / 91.61	71.84 / 0.13	82.04 / 100.0	94.34 / 87.46	22.61 → 0.81

Ablation Study

The paper includes systematic ablations on norm selections, approximation fidelity, generative modeling, and poisoned data defense:

1. Gradient norm formulation (\(\ell_1\) vs \(\ell_2\) sharpness): In CIFAR-10 class unlearning with FT across varying values of \(\rho\), regularizing with \(\|\nabla\ell\|_1\) consistently achieves a smaller Avg. Gap compared to the standard SAM-style \(\|\nabla\ell\|_2\) norm penalty, confirming that worst-case \(\ell_\infty\) perturbations produce flatter and more unlearning-compatible landscapes.
2. Exact vs finite-difference gradient sparsity on CIFAR-10 class forgetting: The finite-difference approximation eliminates computational graph retention while retaining the core unlearning gains:

Regularization Strength \(\rho\)	Exact GS Avg. Gap ↓	Finite-Diff GS Avg. Gap ↓	Note
\(\rho = 0\) (Standard Training)	22.61	22.61	Baseline without gradient regularization
\(\rho = 1 \times 10^{-4}\)	4.65	9.04	Finite difference delivers substantial reduction
\(\rho = 5 \times 10^{-4}\)	0.81	1.49	Near-optimal gap reduction without graph retention

1. Generative unlearning on diffusion models (DDPM, CIFAR-10 Class-0 / Airplane): Evaluating class unlearning on DDPM via SalUn, standard DDPM achieves 0.854 forget accuracy and 37.89 FID, whereas GS-trained DDPM drops forget accuracy to 0.706 (a 0.148 drop in forgotten-class generation) while improving retain-class sample fidelity with an FID of 35.21 (2.68 improvement).

2. Backdoor defense performance (MobileNetV2, 3% poisoned CIFAR-10): Under BadNets attacks, standard training yields a clean test accuracy of 68.19% but suffers an 86.62% accuracy on poisoned test samples (successful backdoor execution). With gradient sparsity, clean accuracy is preserved at 68.10%, while poisoned accuracy drops sharply to 24.63% (a 61.99 percentage point reduction), verifying the gradient spike dampening effect.

Key Findings

• Dramatic improvements in class forgetting: Standard fine-tuning fails to forget target classes under standard training (achieving an FA of 71.84% and an Avg. Gap of 22.61). With gradient sparsity training, FT achieves an FA of 0.13% and an Avg. Gap of 0.81, approaching exact retraining.
• Orthogonality to parameter regularization: Standard regularizers like weight decay and dropout were already active in all baselines; adding gradient \(\ell_1\) regularization yields dramatic orthogonal improvements, demonstrating that gradient-space sparsity governs optimization dynamics distinct from weight-space shrinkage.

Highlights & Insights

• Inverting the unlearning problem: Shifts the paradigm from ad-hoc post-processing toward proactive model design during initial optimization, proving that unlearning complexity is fundamentally governed by training trajectory curvature.
• Duality-grounded regularization: Formulates gradient sparsity not through heuristic assumptions, but as the rigorous first-order dual representation of worst-case \(\ell_\infty\) loss sharpness.
• Synergistic backdoor mitigation: Demonstrates that penalizing gradient \(\ell_1\) norms simultaneously dampens anomalous gradient spikes, neutralizing poisoned training data before it corrupts learned representations.

Limitations & Future Work

• Computational overhead of second-order back-propagation: Exact computation requires an \(F+2B\) pass (\(1.5\times\)–\(2.0\times\) step latency). Although finite-difference approximation eliminates peak memory overhead, further investigations into amortized sharpness proxies (such as trajectory-based SAM-for-free) remain warranted.
• Static regularization weighting: Hyper-parameter \(\rho\) remains static across all architectural layers, overlooking differential curvature dynamics between feature extractors and classification heads.
• Scalability to frontier foundation models: Empirical validation is focused on ResNet, VGG, MobileNet, and small ViTs; evaluation on multi-billion-parameter LLMs and multimodal foundation models remains an important next step.

Related Work & Insights

• vs Weight Sparsity (Jia et al., NeurIPS 2023): While weight sparsity prunes static network parameters to zero, this paper introduces gradient sparsity, which operates on the dynamic loss geometry to flatten optimization trajectories while allowing weights to remain dense.
• vs Sharpness-Aware Minimization (SAM, Foret et al., 2020): Conventional SAM relies on \(\ell_2\) parameter perturbations for generic generalization, whereas this work proves that \(\ell_\infty\) perturbations (minimized via gradient \(\ell_1\) norm) provide superior worst-case bounds for trajectory flattening and unlearning compatibility.
• vs Unrolling SGD (Thudi et al., 2021): Thudi et al. identified cumulative Hessian eigenvalues as the primary determinant of SGD reversal error; this work operationalizes that insight into a practical training objective that suppresses curvature during pre-training.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Pioneers unlearning-compatible model design via dual gradient sparsity regularization of worst-case sharpness]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive cross-architecture and cross-dataset benchmarks spanning classification, diffusion models, and backdoor defense]
• Writing Quality: ⭐⭐⭐⭐⭐ [Mathematically rigorous and clearly structured, linking unrolling SGD, Hölder bounds, and empirical evaluation]
• Value: ⭐⭐⭐⭐⭐ [Provides a foundational pre-training paradigm for privacy compliance, data deletion, and model hygiene]