Forgetting Across Modalities: Unlearning in Multimodal Large Language Models¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: LLM Reasoning / Multimodal VLM / Machine Unlearning
Keywords: Multimodal Large Language Models, Machine Unlearning, Gradient Conflict, Manifold-Constrained Optimization, Selective Erasing
TL;DR¶
Addressing the severe gradient conflicts between forgetting and retaining objectives caused by cross-modal representation coupling in MLLMs, this paper proposes EASE (Efficient Alignment-aware Selective Erasing), which dynamically identifies conflicting parameters via opposite-sign gradient detection and resolves interference via manifold-constrained optimization in a 2D task-gradient subspace.
Background & Motivation¶
Multimodal Large Language Models (MLLMs) pre-trained on massive web-scale corpora frequently memorize and reproduce unauthorized, copyrighted, or sensitive multimodal content, raising pressing privacy and safety concerns. Machine unlearning provides an appealing paradigm to erase the influence of specific training subsets without re-training models from scratch. In unimodal setups, gradient conflict between the forgetting objective (pushing predictions away from target data) and the retaining objective (preserving model utility on retained data) is already a known bottleneck where opposing gradient vectors on shared parameters cause destructive interference and slow convergence.
Extending unlearning to MLLMs exacerbates this conflict substantially. Due to cross-modal representation coupling, visual and textual features are deeply entangled within the very same model weights, with individual parameters simultaneously mediating both visual reasoning and linguistic generation. Consequently, the parameter overlap between the forgetting set and the retaining set is markedly larger than in unimodal LLMs, and opposing gradient updates on these shared weights become far more frequent and destructive. Prior methods, such as heuristic gradient ascent, KL divergence minimization, or static saliency-based parameter masking, either ignore gradient conflicts altogether or rely on fixed pre-computed masks, leading to severe collateral degradation of non-target knowledge and slow optimization.
To overcome these obstacles, this work approaches the conflict from PAC-Bayesian and Lipschitz continuous optimization perspectives: full-space alignment is unnecessary, and one only needs to intervene on the specific coordinate subspace where the two gradients actively push parameters in opposite directions. Core idea: propose EASE (Efficient Alignment-aware Selective Erasing), which dynamically detects conflicting parameters at each step via opposite-sign gradient indicators and conducts a manifold-constrained rotation optimization in the 2D subspace spanned by the task gradients to find the optimal joint descent direction without manual trade-off tuning.
Method¶
Overall Architecture¶
EASE operates through a principled four-stage pipeline: theoretical distance bound formulation, per-step dynamic conflict identification, manifold-constrained gradient alignment in the 2D task subspace, and piecewise parameter updating. Given a target forgetting set \(\mathcal{D}_f = \{(I_i^f, Q_i^f, A_i^f)\}_{i=1}^{N_f}\) and a retaining set \(\mathcal{D}_r = \{(I_i^r, Q_i^r, A_i^r)\}_{i=1}^{N_r}\), the system computes auto-regressive task gradients \(g_f\) and \(g_r\) via sequential backward passes. EASE inspects gradient signs elementwise, forms the conflicting coordinate set, solves a lightweight scalar rotation program to synthesize the aligned update vector, and directly updates the model parameters.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multimodal Inputs<br/>Forgetting Set D_f & Retaining Set D_r"] --> B["Dual Task Gradient Computation<br/>Sequential Backprop for g_f and g_r"]
B --> C["Dynamic Conflicting Parameter Detection<br/>Elementwise Sign Check C_i < 0"]
C --> D["Manifold-Constrained Conflict Mitigation<br/>Rotation Optimization in 2D Gradient Subspace"]
D --> E["Piecewise Adaptive Parameter Update<br/>Apply Aligned Direction to Overlap Only"]
E --> F["Unlearned MLLM<br/>Erased Target Entities & Preserved Retention"]
Key Designs¶
1. Dynamic Conflicting Parameter Detection: Coordinate-Level Sign Filtering
Targeting the limitation that static saliency maps fail to adapt as parameters evolve during unlearning, EASE dynamically isolates the conflicting coordinates at each step. Let \(\theta^t\) denote the model parameters at gradient step \(t\), with corresponding auto-regressive losses \(\mathcal{L}_f^t\) and \(\mathcal{L}_r^t\). EASE defines an elementwise conflict indicator for each scalar parameter \(\theta_i\):
where \(\odot\) denotes elementwise multiplication. Parameters with \(C_i^t < 0\) experience true gradient opposition, as the tasks pull the parameter in reverse directions. The set of conflicting parameters is formally isolated as:
with dimension \(d = |\theta_{f,r}^t|\). Grounded in Proposition 4.2, the functional divergence between unlearned and reference models decomposes along parameter coordinates under Lipschitz continuity. Non-conflicting coordinates (\(C_i^t \ge 0\)) already follow task-optimal trajectories, meaning selective intervention solely on \(\theta_{f,r}^t\) guarantees minimal disturbance to retained multimodal capabilities while maintaining maximum computational efficiency.
2. Manifold-Constrained Conflict Mitigation: 2D Subspace Rotation Optimization
Targeting the drawbacks of manual balance hyper-parameters and lossy orthogonal gradient projection (e.g., PCGrad), EASE formulates conflict resolution as a geometric optimization on a 2-dimensional manifold. Restricting the task gradients to the identified conflicting coordinates yields \(g_f^t, g_r^t \in \mathbb{R}^d\), forming the gradient matrix \(G_t = [g_f^t, g_r^t] \in \mathbb{R}^{d \times 2}\). The feasible set of first-order directions capable of decreasing both task-parameter distances simultaneously is precisely the 2D subspace spanned by \(g_f^t\) and \(g_r^t\). EASE seeks a linear combination \(\tilde{g}^t = w_f g_f^t + w_r g_r^t\) minimizing destructive interference across all conflicting coordinates:
To optimize this objective tractably and avoid unconstrained scaling artifacts, EASE introduces a rotation parameterization with angle \(\alpha \in [0, 2\pi)\) and rotation matrix \(M(\alpha) = \begin{bmatrix} \cos(\alpha) & -\sin(\alpha) \\ \sin(\alpha) & \cos(\alpha) \end{bmatrix}\). The scalar \(\alpha\) is optimized directly via SGD for 10 iterations without requiring any additional forward pass through the backbone, yielding the optimal aligned gradient:
This formulation automatically derives the optimal trade-off entirely from the local gradient geometry without heuristic tuning.
3. Piecewise Adaptive Parameter Update: Selective Gradient Modulation
Targeting excessive parameter distortion and training instability, EASE applies a piecewise update policy across the parameter space. The optimized aligned descent direction \(\tilde{g}^t\) is applied strictly to the conflicting coordinates \(\theta_{f,r}^t\), while all remaining non-conflicting parameters retain their standard unperturbed sum:
The model parameters are then updated using standard gradient descent: \(\theta^{t+1} = \theta^t - \eta \nabla_{\theta^t}\) with learning rate \(\eta > 0\). This piecewise scheme keeps the unlearning intervention focused exclusively on the zone of friction, allowing non-conflicting weights to learn at full capacity.
Loss & Training¶
- Forgetting Loss \(\mathcal{L}_f\): Auto-regressive negative log-likelihood computed over the forgetting set \(\mathcal{D}_f\) with gradient ascent to suppress target output tokens: \(\mathcal{L}_f = \frac{1}{N_f} \sum_{i=1}^{N_f} \sum_{t=1}^{|A_i^f|} -\log p(a_{it}^f \mid I_i^f, Q_i^f, a_{i,<t}^f; \theta^\star)\).
- Retaining Loss \(\mathcal{L}_r\): Auto-regressive negative log-likelihood computed over the retaining replay set \(\mathcal{D}_r\) with gradient descent to preserve correct answer distributions: \(\mathcal{L}_r = \frac{1}{N_r} \sum_{i=1}^{N_r} \sum_{t=1}^{|A_i^r|} -\log p(a_{it}^r \mid I_i^r, Q_i^r, a_{i,<t}^r; \theta^\star)\).
- Optimization Details: LoRA fine-tuning applied to all linear layers (\(r=16, \alpha=16, \text{dropout}=0.05\)); fine-tuned for 1 epoch with batch size 4, learning rate \(1 \times 10^{-5}\) using AdamW and a linear schedule with 0 warmup steps; mixed-precision bfloat16 on 4 NVIDIA H800 GPUs (80 GB).
- Subspace Optimization: Rotation angle initialized at \(\alpha = 0.25\), optimized using SGD with learning rate \(10^{-2}\) for 10 steps per gradient iteration.
Key Experimental Results¶
Main Results¶
Performance was evaluated across two multimodal unlearning benchmarks: MLLMU-Bench (fictitious profiles and public celebrity evaluation) and CLEAR (character unlearning across visual and textual modalities), evaluated on four distinct model backbones:
| Model Architecture | Method | MLLMU-Bench Forget Class Acc (↓) |
MLLMU-Bench Retain Class Acc (↑) |
MLLMU-Bench Real-world Class Acc (↑) |
CLEAR Forget Class Acc (↓) |
CLEAR Retain Class Acc (↑) |
CLEAR Real-world Class Acc (↑) |
|---|---|---|---|---|---|---|---|
| LLaVA-1.5-7B | Vanilla | 38.33 | 39.34 | 32.74 | 63.82 | 58.48 | 50.95 |
| Grad_Diff | 28.33 | 28.54 | 19.06 | 44.68 | 55.69 | 46.80 | |
| KL_Min | 27.50 | 25.49 | 14.36 | 35.10 | 36.20 | 43.05 | |
| NPO | 26.66 | 25.20 | 12.66 | 57.97 | 56.45 | 40.13 | |
| MMUnlearner | 29.16 | 25.20 | 14.62 | 13.29 | 12.91 | 38.14 | |
| Ours (EASE) | 20.00 | 31.41 | 21.95 | 18.61 | 41.13 | 48.77 | |
| LLaVA-1.5-13B | Vanilla | 51.66 | 52.60 | 57.83 | 63.82 | 61.01 | 56.67 |
| Grad_Diff | 37.50 | 52.49 | 51.17 | 38.82 | 58.24 | 50.04 | |
| KL_Min | 40.00 | 41.90 | 50.26 | 51.59 | 46.58 | 43.05 | |
| NPO | 44.16 | 47.35 | 48.43 | 55.31 | 48.98 | 50.95 | |
| MMUnlearner | 50.00 | 51.42 | 49.21 | 13.82 | 14.68 | 38.41 | |
| Ours (EASE) | 29.16 | 52.51 | 54.96 | 46.46 | 45.06 | 52.86 | |
| Qwen2.5-VL-7B | Vanilla | 47.50 | 49.04 | 75.71 | 79.68 | 69.36 | 68.93 |
| Grad_Diff | 46.33 | 44.77 | 73.21 | 67.02 | 67.34 | 68.39 | |
| KL_Min | 46.66 | 42.43 | 74.41 | 71.21 | 67.07 | 68.11 | |
| NPO | 43.33 | 45.35 | 74.93 | 68.08 | 67.08 | 68.39 | |
| MMUnlearner | 52.50 | 46.34 | 75.64 | 69.14 | 52.65 | 68.39 | |
| Ours (EASE) | 33.33 | 46.61 | 75.58 | 45.74 | 67.85 | 66.21 |
Ablation Study¶
Ablation experiments conducted on LLaVA-1.5-7B on MLLMU-Bench dissect the individual contributions of Partial Parameter Identification and Manifold-Constrained Conflict Mitigation:
| Configuration | Forget Set Class Acc (↓) | Forget Set ROUGE (↓) | Forget Set Cloze Acc (↓) | Retain Set Class Acc (↑) | Retain Set ROUGE (↑) | Retain Set Cloze Acc (↑) | Note |
|---|---|---|---|---|---|---|---|
| Ours (Full EASE) | 20.00 | 0.021 | 2.083 | 31.41 | 0.081 | 11.70 | Full method: optimal Pareto forget-retain balance |
| w/o Partial Param | 21.66 | 0.058 | 2.083 | 20.67 | 0.039 | 8.133 | Updating all parameters blindly hurts retain accuracy (-10.74%) |
| w/ CAGrad | 24.16 | 0.052 | 10.41 | 24.31 | 0.044 | 3.902 | Sub-optimal manifold exploration leads to premature loss plateau |
| w/ PCGrad | 33.33 | 0.093 | 4.166 | 29.17 | 0.067 | 1.687 | Dropping conflicting components impairs unlearning efficacy |
| w/o Both | 22.50 | 0.058 | 10.41 | 21.18 | 0.036 | 2.109 | Training dynamics degrade severely without both components |
Key Findings¶
- Partial Parameter Identification protects retain utility: Removing dynamic coordinate identification (
w/o Partial Param) causes retaining accuracy to plummet from 31.41% to 20.67%. Non-conflicting parameters should never be unnecessarily perturbed, as doing so inflicts collateral damage on general multimodal reasoning. - Manifold direction search outperforms heuristic gradient projection: Replacing EASE's 2D manifold search with PCGrad causes forget accuracy to surge to 33.33% (vs. 20.00% for EASE). Simply dropping conflicting projections erases vital descent components, whereas rotational search finds an aligned consensus direction.
- Striking memory efficiency via sequential gradient realization: Benchmarking on Qwen2.5-VL-7B reveals that EASE requires 7.40 minutes per epoch (1.22× Grad_Diff's 6.05 min and comparable to MMUnlearner's 7.22 min), but slashes peak memory allocation to only 7.34 GB compared to 20.51–23.58 GB across all baselines. Materializing task gradients sequentially frees activation graphs immediately before the next pass.
Highlights & Insights¶
- Geometric simplification of complex multi-objective optimization: By proving that first-order descent directions lie within the 2D plane spanned by the task gradients, the complex high-dimensional conflict problem is reduced to optimizing a single scalar rotation angle \(\alpha\) via 10 quick SGD steps.
- Architectural-agnostic cross-modal unlearning: The mechanism operates purely at the gradient coordinate level, requiring no task-specific adapter heads or attention interventions, demonstrating consistent gains across both LLaVA and Qwen2.5-VL families.
- Transferable insight: The pattern of "elementwise sign conflict detection followed by manifold-constrained rotation alignment" is directly applicable to safety-alignment fine-tuning, catastrophic forgetting prevention in continual instruction tuning, and multi-task representation learning.
Limitations & Future Work¶
- Local first-order linear approximation assumption: The mathematical guarantee that the 2D subspace contains valid descent directions relies on first-order Taylor expansion within the local neighborhood, which restricts the effective update step size under steep curvature.
- Two-objective limitation: The single rotation parameterization is specifically formulated for the dual-task setup (\(\mathcal{D}_f\) vs. \(\mathcal{D}_r\)). Extending this framework to disentangle finer-grained objectives (e.g., visual retention, textual retention, and safety constraint losses) requires higher-dimensional multi-objective programming.
- Absence of asymptotic Pareto-stationary theoretical guarantee: While loss curves demonstrate smooth empirical convergence, because the 2D subspace is dynamically reconstructed at each step, theoretical convergence to a global Pareto-stationary point remains unproven.
Related Work & Insights¶
- vs MMUnlearner (Huo et al., 2025): MMUnlearner relies on static weight saliency computed prior to unlearning; EASE demonstrates that parameter conflicts shift dynamically as weights update, achieving substantially higher retention (e.g., CLEAR Retain Acc of 41.13% vs. 12.91%).
- vs PCGrad / CAGrad: PCGrad discards conflicting gradient components via orthogonal projection, whereas CAGrad optimizes a minimax surrogate around the average gradient. EASE resolves conflicts within the exact 2D span of the task gradients, achieving a cleaner balance between unlearning efficacy and knowledge preservation.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Elegantly models cross-modal unlearning conflicts via dynamic coordinate identification and 2D subspace rotation optimization.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across 4 MLLM architectures on 2 specialized multimodal unlearning benchmarks with comprehensive loss dynamics and cost profiles.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear mathematical formulation bridging PAC-Bayesian distance bounds directly into an efficient optimization algorithm.
- Value: ⭐⭐⭐⭐⭐ Delivers a lightweight, high-performance solution for privacy erasure and safety compliance in modern multimodal models.