Skip to content

AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/kaln27/AWARe
Area: Model Compression
Keywords: Catastrophic Forgetting, Multimodal Large Language Models, Activation-Weighted Retention, Continual Fine-Tuning, Parameter-Efficient Adaptation

TL;DR

Addressing catastrophic forgetting in Multimodal Large Language Models (MLLMs) during downstream adaptation, AWARe identifies critical neurons via activation saliency on a small calibration set and globally freezes the top 30% most salient weights in self-attention and projection layers, updating only ~17.5% of parameters while achieving superior stability and plasticity.

Background & Motivation

Multimodal Large Language Models (MLLMs) have established remarkable general reasoning and cross-modal perceptual capabilities by aligning pre-trained visual encoders with large language model backbones. To tailor these foundation models to specific downstream domains—such as specialized visual question answering, dense image captioning, or continuous instruction streams—fine-tuning remains standard practice. However, this adaptation process invariably incurs severe catastrophic forgetting: unconstrained backpropagation gradients overwrite parameters essential for general reasoning, causing catastrophic performance collapses across previously acquired upstream capabilities.

Existing forgetting mitigation frameworks struggle to scale gracefully to billions of multimodal parameters. Traditional experience replay incurs immense computational, memory, and privacy overheads from storing or replaying massive multimodal datasets; regularization-based methods (such as EWC) rely on computationally prohibitive Fisher information matrices. While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA and DoRA dramatically reduce the number of trainable weights, their unconstrained low-rank perturbations still degrade pre-trained feature geometries and cause substantial forgetting. Meanwhile, static weight pruning and gradient-based selective tuning rely either on weight magnitudes or transient backward gradients, failing to capture the dynamic, input-conditioned neuronal activation pathways that actually sustain functional stability during multimodal forward propagation.

Inspired by activation-aware parameter saliency in model quantization (AWQ) and network pruning (Wanda), which show that a small fraction of weights coupled with large activation magnitudes disproportionately dictate model behavior, this work repurposes activation saliency for selective forgetting mitigation. The paper posits that within the self-attention projections and multimodal connectors of an MLLM, neurons exhibiting consistently high activation across inputs underpin foundational general capabilities. Core idea: estimate neuron saliency via sequence-aggregated and cross-dimension normalized activation norms from a compact calibration set, globally freeze the top 30% most salient output weight rows, and update only the remaining ~17.5% low-saliency parameters to cleanly decouple downstream adaptation from upstream retention.

Method

Overall Architecture

AWARe operates as an architecture-agnostic, two-phase framework that imposes zero inference latency overhead upon deployment. In Phase 1 (Knowledge Profiling), the model executes a single forward pass over a compact calibration set (200–400 samples from upstream tasks or general benchmarks like MMMU) to collect activation tensors from target linear layers (self-attention projections \(W_q, W_k, W_v\) and the multimodal connector mm_projector). These activations undergo sequence \(L_2\)-norm aggregation and hidden-dimension normalization to yield a robust scalar saliency score for every output neuron channel. In Phase 2 (Constrained Fine-tuning), layer-channel pairs across all target layers are pooled and ranked globally; a binary gradient mask freezes the top \(\rho = 30\%\) most salient rows, allowing gradient updates only on the remaining non-salient weights.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Calibration Input<br/>Compact upstream or MMMU subset"] --> B["Forward Activation Profiling<br/>Extract Q/K/V and projector activations"]
    B --> C["Cross-Dimension Norm Normalization<br/>Sequence L2 aggregation and hidden-dim norm"]
    C --> D["Global Saliency Ranking and Masking<br/>Pool layer-channel pairs and freeze Top-ρ"]
    D --> E["Dynamic Mask Constrained Fine-Tuning<br/>AwareLinear decouples frozen and active weights"]
    E --> F["Zero-Overhead Deployment Checkpoint<br/>Unwrap operator to retain native architecture"]

Key Designs

1. Cross-Dimension Norm Normalization: Eliminating Sequence Length and Magnitude Scale Distortions To address the issue where varying sequence lengths and outlier-scale tokens distort the true functional importance of neurons across heterogeneous multimodal inputs, AWARe designs a gradient-free, three-step statistical aggregation pipeline. For a target linear layer \(W \in \mathbb{R}^{d_{out} \times d_{in}}\) producing activation tensor \(A \in \mathbb{R}^{B \times L \times d_{out}}\), the sequence-length dimension is first aggregated via \(L_2\)-norm for each sample \(i\) and output neuron \(k\): $\(a'_{i,k} = \sqrt{\sum_{j=1}^L a_{i,j,k}^2}\)$ Directly averaging \(a'_{i,k}\) would allow individual long-context or high-contrast samples to dominate the saliency metric. AWARe eliminates this scale bias by applying per-sample \(L_2\)-normalization across the hidden dimension \(d_{out}\), producing normalized activation coefficients \(a''_{i,k} = a'_{i,k} / \|a'_i\|_2\). Averaging across the batch yields the final neuron saliency score \(s_k = \frac{1}{B} \sum_{i=1}^B a''_{i,k}\), ensuring that selection reflects consistent structural activation rather than sporadic numerical outliers.

2. Global Saliency Ranking and Masking: Cross-Layer Quota Allocation Over Rigid Uniformity To overcome the limitations of conventional layer-balanced retention—which artificially forces every layer to freeze an identical fraction of weights regardless of its depth or specialization—AWARe implements a Global-Highest Selection strategy. Neurons in shallower Transformer layers often extract general low-level geometric and text features, whereas deeper layers model specialized task-specific semantics. By pooling all candidate layer-channel index pairs \((l, k)\) into a global set and ranking them jointly, AWARe identifies the globally highest-scoring \(\rho\) fraction of channels \(\mathcal{I}\). The binary gradient mask \(M_{grad} \in \{0, 1\}^{d_{out} \times d_{in}}\) is defined as: $\((M_{grad})_{k,:} = 0 \quad \text{if } k \in \mathcal{I}, \qquad (M_{grad})_{k,:} = 1 \quad \text{otherwise}\)$ Empirical inspection confirms this mechanism naturally freezes a higher fraction of early-layer \(q\_proj\) and \(k\_proj\) weights while distributing value projections (\(v\_proj\)) more uniformly across depth.

3. Dynamic Mask Constrained Fine-Tuning: Efficient Row-Wise Decoupling Without Inference Footprint To prevent training-time gradient overhead and eliminate the inference latency typical of additive adapters or multi-branch architectures, AWARe introduces AwareLinear, a lightweight training-time module wrapper. During downstream backpropagation, gradient updates are filtered via Hadamard product with the retention mask: $\(W^{(t+1)} = W^{(t)} - \eta \cdot \left(\nabla_W \mathcal{L}_{down} \odot M_{grad}\right)\)$ In implementation, AwareLinear physically slices target linear weight rows into frozen and active sub-matrices during optimization, saving gradient and optimizer state memory for frozen channels. Once fine-tuning concludes, the wrapper is discarded, restoring native PyTorch linear layers without auxiliary runtime operations. Restricting target layers to self-attention projections (\(W_q, W_k, W_v\)) and mm_projector while keeping all MLP blocks frozen limits trainable parameters to approximately \(17.5\%\) of the total block parameters (\(2.1h^2 / 12h^2\)).

Loss & Training

Downstream adaptation is trained solely with the supervised objective of the target task, requiring no composite regularizers or upstream cross-entropy replay: $\(\mathcal{L} = \mathcal{L}_{down}(M_\theta, \mathcal{D}_{down})\)$ Optimization uses AdamW with Cosine Annealing learning rate scheduling. The LLM backbone learning rate is set to \(2 \times 10^{-5}\), the multimodal projector learning rate is \(2 \times 10^{-4}\) (warmup ratio: 0.03), and training runs for 3 epochs with a global batch size of 16.

Key Experimental Results

Main Results

On single-task downstream adaptation using LLaVA-v1.5-7B (evaluated across upstream tasks OKVQA, OCRVQA, GQA, TextVQA, and downstream targets IconQA and COCO-Caption), AWARe was compared against Full Fine-Tuning, LoRA, DoRA, Orth-Reg, DARE, Model Tailor, LoRASculpt, and SPIDER. Reported metrics include upstream retention \(R\), downstream learning efficiency \(E\), and their harmonic mean \(H\):

Method IconQA: OKVQA IconQA: OCRVQA IconQA: GQA IconQA: TextVQA IconQA: Target IconQA: R IconQA: E IconQA: H COCO-Cap: Target COCO-Cap: R COCO-Cap: E COCO-Cap: H
Zero-shot 57.99 66.20 61.93 58.23 23.18 100.0 28.9 44.9 40.40 100.0 42.3 59.4
Full-FT 0.04 0.00 0.00 0.46 80.15 0.2 100.0 0.4 95.59 0.0 100.0 0.0
LoRA 45.06 48.40 49.22 36.47 63.65 73.3 79.4 76.2 110.27 81.6 115.4 95.6
DoRA 49.94 54.40 55.39 47.26 84.48 84.7 105.4 93.9 112.60 82.6 117.8 97.1
Orth-Reg 53.12 56.10 57.43 51.00 84.52 89.1 105.5 96.6 111.87 86.5 117.0 99.5
DARE 44.39 46.85 48.75 34.84 82.28 71.5 102.7 84.3 108.57 79.5 113.6 93.5
Model Tailor 50.07 53.85 53.04 46.92 81.76 83.4 102.0 91.8 117.64 89.9 123.1 103.9
LoRASculpt 53.52 59.50 57.63 53.76 85.26 91.8 106.4 98.6 120.35 88.8 125.9 104.2
SPIDER 50.41 61.49 56.54 51.54 85.29 90.0 106.4 97.5 114.74 89.2 120.0 102.3
AWARe (Ours) 56.95 65.10 60.83 57.66 86.92 98.4 108.4 103.2 120.94 94.2 126.5 108.0
AWARe (MMMU) 56.86 63.35 60.93 56.77 85.23 97.4 106.3 101.7 121.22 93.9 126.8 107.9

On the five-task continual instruction tuning benchmark MLLM-DCL (Remote Sensing \(\to\) Medical VQA \(\to\) Autonomous Driving \(\to\) Science \(\to\) Finance):

Continual Learning Method Average Accuracy (Avg ↑) Last Task Accuracy (Last ↑)
ModalPrompt 53.87 53.94
HiDe 57.04 55.31
LoRA-FT 58.52 53.00
MoELoRA 58.53 54.78
CL-MoE 59.30 54.88
O-LoRA 59.53 55.54
SEFE 60.96 58.51
DISCO 62.40 60.33
AWARe (Ours) 65.58 (+3.18) 62.10 (+1.77)

Ablation Study

Ablations on parameter selection criteria, distribution strategies, and retention ratios \(\rho\) on IconQA and COCO-Caption validate the necessity of activation statistics:

Dimension Configuration IconQA: Source Avg IconQA: Target COCO-Cap: Source Avg COCO-Cap: Target Note
Selection Criterion Random Selection (30%) 49.94 86.32 49.46 116.31 Upstream capabilities severely disrupted
Selection Criterion Weight Norm (30%) 58.51 85.51 53.52 115.19 Static magnitude misses dynamic inference flow
Selection Criterion Hybrid (0.3W + 0.7A) 58.69 84.66 53.90 115.24 Combining static norm dilutes saliency quality
Selection Criterion Activation (30%) 60.13 86.92 57.56 120.94 Optimal stability-plasticity balance
Allocation Strategy Layer-Balanced (30%) 58.95 85.29 - - \(R=96.5, E=106.4, H=101.2\)
Allocation Strategy Global-Highest (30%) 60.13 86.92 - - \(R=98.4, E=108.4, H=103.2\)
Retention Quota Global-Highest \(\rho=10\%\) 58.53 86.13 - - Under-protection: \(R=95.8, H=101.3\)
Retention Quota Global-Highest \(\rho=30\%\) 60.13 86.92 - - Sweet spot balancing plasticity & retention
Retention Quota Global-Highest \(\rho=50\%\) 60.55 85.97 - - \(R=99.1, E=107.3, H=103.0\)
Retention Quota Global-Highest \(\rho=90\%\) 60.98 54.62 - - Severe underfitting on target: \(E\) drops to 68.1

Key Findings

  • Distinct Functional Roles Between Attention and MLP Blocks: Updating MLP blocks induces catastrophic collapse on upstream tasks (scores drop below 30), supporting the finding that MLPs store factual foundation knowledge, whereas self-attention projections (\(W_q, W_k, W_v\)) provide adaptable contextual routing.
  • Dynamic Activations Outperform Static Magnitudes: Selecting weights purely by magnitude yields upstream averages of 58.51 and 53.52 on IconQA and COCO-Caption, trailing activation-based selection by 1.62 and 4.04 points. Actual data-driven activation strength is far more predictive of task retention.
  • Robustness to Proxy Calibration Sets: In data-restricted settings where upstream task samples are inaccessible, using 600 samples from the general-purpose MMMU benchmark achieves an \(H\) of 101.7 on IconQA and 107.9 on COCO-Caption, outperforming competitive baselines without original task data.
  • Calibration Sample Efficiency and Cross-Architecture Generalization: Increasing calibration size from 200 to 400 yields negligible standard deviations (0.23–0.30), demonstrating that 200 samples suffice for stable saliency profiling. AWARe also generalizes effectively to Qwen2.5-VL-7B and Qwen3-VL-8B.

Highlights & Insights

  • Repurposing Activation Saliency from Compression to Continual Learning: AWARe cleverly translates insights from AWQ and Wanda into anti-forgetting fine-tuning, demonstrating that salient forward-activation channels serve as an effective defensive barrier against gradient corruption.
  • Global Cross-Layer Budget Allocation: Moving beyond uniform per-layer budgets, global pooling automatically concentrates freezing in early query and key projections, honoring the hierarchical functional structure of Transformer layers.
  • Zero-Footprint Deployment: Unlike adapter-based or prompt-tuning schemes that introduce runtime parameters and latency, AWARe removes its training wrapper upon completion, remaining 100% natively compatible with standard inference runtimes.

Limitations & Future Work

  • Dependence on Calibration Modality Coverage: If calibration data is restricted to text-only benchmarks like MMLU, visual reasoning capabilities not captured in calibration (such as ChartQA) experience notable degradation (dropping from 74.5 to 66.5).
  • Heuristic Retention Quota: The retention ratio \(\rho\) is fixed empirically (e.g., 30%), lacking an automated closed-loop mechanism to dynamically assign per-layer quotas under drastic domain shifts.
  • Evaluation Limited to \(\le 8\text{B}\) Models: Experiments are conducted on LLaVA-v1.5-7B, Qwen2.5-VL-7B, and Qwen3-VL-8B; scaling behavior on \(\ge 70\text{B}\) parameters or natively omni-modal (audio/video) models remains to be explored.
  • vs LoRA / DoRA / DARE: Unconstrained low-rank adapters perturb the entire parameter subspace, inadvertently degrading pre-trained features. AWARe shows that selective row-level freezing outperforms homogeneous low-rank adaptation in preserving upstream reasoning.
  • vs Model Tailor / SPIDER: Gradient-based sparse tuning relies on single-step backward derivatives. In contrast, AWARe exploits forward activation dynamics, which are computationally lighter, non-iterative, and more structurally stable across sample sets.
  • vs HiDe-LLaVA / SEFE / DISCO: While continual learning baselines introduce complex modular routing or orthogonal feature regularizations, AWARe achieves superior sequential retention merely by recomputing activation masks across stages without architectural modifications.

Rating

  • Novelty: ⭐⭐⭐⭐☆ An elegant and effective transfer of activation-based saliency profiling to MLLM continual fine-tuning.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across single-task adaptation and 5-stage continual learning on three distinct backbones (LLaVA-v1.5, Qwen2.5-VL, Qwen3-VL).
  • Writing Quality: ⭐⭐⭐⭐⭐ Well-structured, crisp methodology, informative diagrams, and concise mathematical formulation.
  • Value: ⭐⭐⭐⭐⭐ Zero runtime overhead, high training efficiency, and strong empirical gains make it an attractive paradigm for downstream MLLM adaptation.