Model-Dowser: Data-Free Importance Probing to Mitigate Catastrophic Forgetting in Multimodal Large Language Models

Conference: ICML 2026
arXiv: 2602.04509
Code: None
Area: Multimodal VLM / Continual Learning / Sparse Fine-tuning
Keywords: MLLM, Catastrophic Forgetting, Sparse Fine-tuning, Parameter Importance, Data-Free Probing

TL;DR

Model-Dowser scores each parameter in an MLLM using the product of "weight magnitude × input activation × output Jacobian." High-scoring parameters are frozen, and only low-scoring ones are updated. This enables deep fine-tuning on LLaVA/NVILA to learn downstream tasks while retaining pretraining knowledge. Compared to SPIDER and ModelTailor, it consistently leads in H-score.

Background & Motivation

Background: MLLMs (e.g., LLaVA, NVILA) often require further fine-tuning for specialized tasks, but full-tuning severely damages pretrained general capabilities—this is "catastrophic forgetting" in MLLMs. Existing mitigation methods fall into two categories: post-merging (e.g., ModelTailor) fuses pre- and post-finetuning weights, and sparse fine-tuning (e.g., SPIDER) updates only a small subset of weights.

Limitations of Prior Work: (1) Post-merging works when only the last few layers are fine-tuned, but fails when fine-tuning extends to early decoder layers, as deep changes make latent space unrecoverable by merging; (2) Existing sparse methods (e.g., SPIDER) rely on gradient history and soft masks, requiring per-parameter accumulated gradients, which is memory-intensive and hard to scale to tens of billions of parameters; (3) Traditional magnitude-based importance assumes homogeneous activations, which is inaccurate for modern nonlinearities like GELU/SiLU/GLU.

Key Challenge: Achieving both "deep fine-tuning without forgetting" and "no increase in memory/computation." The former requires importance estimation to reflect functional impact under nonlinear activations, while the latter rules out storing gradient history.

Goal: To find a parameter importance metric that (i) does not rely on pretraining data, (ii) does not require extra gradient history, and (iii) remains accurate under heterogeneous activations, enabling hard-masked sparse fine-tuning.

Key Insight: The authors reframe "which parameters are most important" as "which parameter perturbations most affect model output"—using a first-order Taylor estimate of output shift \(\|\Delta f\|_2\), thus grounding importance in functional rather than numerical terms.

Core Idea: Use the three-factor product \(S_{ij}^{(l)}=\|J_i^{(l)}\|_2\cdot|W_{ij}^{(l)}|\cdot|h_j^{(l-1)}|\) as the importance score. Hutchinson estimator and model self-generated synthetic prompts enable data-free, memory-efficient probing, followed by hard freezing of high-scoring parameters.

Method

Overall Architecture

Model-Dowser is a three-stage pipeline: (1) Probing—use MLLM-generated synthetic prompts for forward passes to collect activations, and a few backward passes with the Hutchinson trick to collect Jacobian L2 norms; (2) Compute Score—score each weight as \(S=\|J_i\|_2\cdot|W_{ij}|\cdot|h_j|\), averaging over \(N\) Monte Carlo samples; (3) Sparse Fine-tune—within each layer, freeze the top-\((1-\rho)\) high-scoring weights, and use a binary mask to restrict gradients to the remaining \(\rho\) "non-critical" weights for standard SGD. The process requires no original pretraining data and maintains no gradient history.

Key Designs

1. Three-Factor Functional Importance Scoring:

   • Function: Quantifies "how much perturbing a weight shifts the final output" in MLLMs with heterogeneous activations.
   • Mechanism: Based on Theorem 3.1—first-order Taylor gives \(\|\Delta f\|_2\approx\|J_i^{(l)}\|_2\cdot|\Delta W_{ij}^{(l)}|\cdot|h_j^{(l-1)}|\). Substituting \(|\Delta W|\) with current weight magnitude \(|W|\) yields \(S_{ij}^{(l)}=\|J_i\|_2\cdot|W_{ij}|\cdot|h_j|\). The three terms capture "downstream output sensitivity" (Jacobian), "parameter scale" (weight), and "upstream activation strength" (activation).
   • Design Motivation: Pure magnitude (e.g., Wanda) ignores GELU/SiLU nonlinearity; pure gradient (e.g., SPIDER) is memory-intensive. This combination completes the local linear gradient path while avoiding gradient history.

2. Data-Free Jacobian/Activation Probing:

   • Function: Estimates \(\|J_i\|_2\) and \(|h_j|\) without original pretraining data and avoids explicit Jacobian construction.
   • Mechanism: Uses the Hutchinson Trace Estimator—projects output onto a random Rademacher vector \(\xi\in\{\pm 1\}^{d_{\text{final}}}\), so \(\mathbb{E}_\xi[(\partial(\xi^\top f)/\partial z_i)^2]=\|J_i\|_2^2\). Only a few backward passes are needed to obtain output sensitivities for all nodes. The MLLM self-generates \(N\) synthetic prompts \(\hat{x}_n=f(\epsilon;\theta_{\text{pre}})\) using random token seeds, and Monte Carlo averages are computed: \(\bar S=\frac{1}{N}\sum_n \|J_{i,n}\|_2\cdot|W_{ij}|\cdot|h_{j,n}|\). Total complexity is \(\mathcal{O}(N\cdot R)\) forward/backward passes, with \(N,R\ll d_{\text{final}}\).
   • Design Motivation: Pretraining data is usually unavailable; synthetic prompts activate "model-learned" functional structures rather than task-specific distributions. Hutchinson avoids \(d_{\text{final}}\)-scale backward passes.

3. Hard Binary Mask Sparse Fine-tuning:

   • Function: Translates "protect important parameters" into a per-element mask during training, introducing no extra learnable parameters or memory overhead.
   • Mechanism: Within each layer, sort \(\bar S\) in ascending order, select the bottom \(\rho\) fraction (e.g., \(\rho=0.1\)) as updatable (mask=1), and freeze the rest. Update rule: \(\theta^*=\theta-\lambda\cdot(M\odot\partial\mathcal{L}/\partial\theta)\). Freezing high \(\bar S\) directly suppresses dominant output perturbations under first-order Taylor.
   • Design Motivation: Compared to ModelTailor's post-merging or SPIDER's soft mask + dynamic updates, the hard mask uses memory equivalent to standard fine-tuning, is compatible with LoRA/full-parameter pipelines, and, since the mask is precomputed, avoids runtime importance score maintenance.

Loss & Training

Downstream tasks use standard instruction tuning loss, with gradients multiplied by the mask. The probing stage requires no loss—only forward/backward passes to collect activations and Jacobian L2 norms. For NVILA-Lite 2B, \(\rho=0.1\) and the last 20 decoder layers are fine-tuned; LLaVA 1.5 7B experiments similarly keep \(\rho\) small, emphasizing "minimal updates, maximal retention."

Key Experimental Results

Main Results

Method (NVILA-Lite 2B, COCO-Caption column, last 20 layers \(\rho=0.1\))	\(A_{\text{down}}\) ↑	Upstream Mean ↑	H-Score ↑
Zero-shot	36.8 (ref)	62.3	—
Full-FT	98.5	24.0	39.7
Grafting	115.7	38.7	49.2
DARE	96.8	24.9	39.1
ModelTailor	105.6	18.9	44.7
SPIDER	115.4	59.6	78.3
Model-Dowser	On par with strongest	68.8 (best/second-best COCO)	Significantly ahead of SPIDER

Data from Table 1 in the paper: Model-Dowser maintains downstream adaptation (\(A_{\text{down}}\)) close to the strongest baselines, while raising the mean of six upstream tasks above all other methods, thus ranking first in H-Score.

Ablation Study

Dimension	Observation
Fine-tuning depth (last 5 / 10 / 20 / 32 layers)	Post-merging (DARE, ModelTailor) quickly fails as depth increases; Model-Dowser and SPIDER are more stable, but SPIDER is more memory-intensive
Use of synthetic prompts	Masks obtained from synthetic prompts are nearly equivalent to those from real data, indicating synthetic prompts sufficiently activate functional structure (Appendix G)
Hutchinson estimator samples \(R\), MC count \(N\)	Small \(N,R\) (tens) suffice for stable ranking; probing overhead is much less than a full fine-tuning run
Different backbones (LLaVA 1.5 7B vs NVILA-Lite 2B)	H-Score consistently leads, robust to model scale/architecture

Key Findings

• Deep fine-tuning (updating early decoder layers) is the "death zone" for post-merging methods, yet is crucial for multimodal understanding in MLLMs; Model-Dowser remains stable here, its main advantage over ModelTailor and DARE.
• Importance is mainly driven by "output Jacobian × input activation," not just weight magnitude—explaining why pure magnitude (Wanda-style) ranks poorly under SiLU/GLU architectures.
• The data-free synthetic prompt approach naturally scales to tens of billions of parameters, as it requires neither pretraining data nor gradient history.

Highlights & Insights

• Shifts the perspective on "parameter importance" from weight values to "functional output sensitivity," providing a rigorous first-order Taylor bound—an elegant and practical transfer of Optimal Brain ideas from pruning literature to continual learning/forgetting prevention.
• The Hutchinson trick compresses "full Jacobian computation" into a few backward passes—a highly reusable technique for any scenario needing \(\|J\|_2\) but unable to afford full backward computation.
• Synthetic prompts decouple importance probing from data dependence, enabling models to "self-diagnose" upon delivery—especially useful for post-deployment fine-tuning scenarios.

Limitations & Future Work

• First-order Taylor is coarse under large perturbations; for large learning rates or highly divergent fine-tuning data, scores may underestimate nonlinear effects in some directions.
• The mask is a one-off "static" score, not dynamically updated during training; for long or multi-task continual fine-tuning, periodic recomputation may be needed.
• Experiments are mainly on classic vision-language benchmarks (ImageNet-R, COCO); not yet validated on true multimodal long-context, video, or agent tasks.
• The trade-off between "downstream performance vs upstream retention" still relies on the manually tuned hyperparameter \(\rho\), with no theoretical guidance for its selection.

Related Work & Insights

• vs SPIDER: Both are sparse fine-tuning methods, but SPIDER dynamically maintains soft masks and accumulated gradients during training, which is memory-intensive. Model-Dowser uses a one-off hard mask and Hutchinson Jacobian, with memory usage equivalent to standard fine-tuning and no reliance on training data.
• vs ModelTailor / DARE (post-merging): These rely on "post-hoc fusion" for retention, but deep changes make latent space unrecoverable. Model-Dowser freezes functional anchors before training, preventing drift at the source.
• vs Wanda / magnitude pruning: Both are "weight × activation" families, but Wanda lacks the Jacobian term and ranks poorly under heterogeneous activations. Model-Dowser's three-factor score is a more complete functional approximation.

Rating

• Novelty: ⭐⭐⭐⭐ Combines Optimal Brain ideas, Hutchinson trick, and synthetic prompts into a data-free MLLM forgetting mitigation scheme; novel combination though each component is from existing tools.
• Experimental Thoroughness: ⭐⭐⭐⭐ Covers two backbone types (LLaVA, NVILA), multiple depths, multiple downstream tasks, and multiple baselines, but lacks validation on multimodal long-context/video tasks.
• Writing Quality: ⭐⭐⭐⭐ Theorem and module breakdowns are clear, pipeline diagrams are intuitive; tables are dense but structure is somewhat scattered.
• Value: ⭐⭐⭐⭐⭐ Provides a directly applicable MLLM forgetting mitigation tool, memory-friendly, data-agnostic, scalable to tens of billions of parameters, with high industrial deployment value.

Related Papers

• [ICCV 2025] SMoLoRA: Exploring and Defying Dual Catastrophic Forgetting in Continual Visual Instruction Tuning
• [ICML 2026] Revis: Sparse Latent Steering to Mitigate Object Hallucination in Large Vision-Language Models
• [ICML 2026] Vision-aligned Latent Reasoning for Multi-modal Large Language Model
• [NeurIPS 2025] ACT as Human: Multimodal Large Language Model Data Annotation with Critical Thinking
• [ICML 2026] Instruction Lens Score: Your Instruction Contributes a Powerful Object Hallucination Detector for Multimodal Large Language Models