Curvature-Guided Mixing for MLLM Adaptation

Conference: ECCV2026
arXiv: 2606.24963
Code: https://github.com/zzsyjl/CGM-ECCV-2026
Area: Multimodal VLM
Keywords: Catastrophic forgetting, model merging, curvature guidance, Hessian approximation, MLLM fine-tuning

TL;DR

This paper proposes the Curvature-Guided Mixing (CGM) framework. By performing second-order Taylor expansion and Hessian diagonal approximation on both pre-training and fine-tuning loss surfaces, a closed-form soft mixing ratio based on relative curvature (CGM) is derived. Furthermore, a sparse hard mixing strategy based on curvature-aware scores (CGM†) is designed. Together, they achieve an optimal balance between downstream adaptation and general knowledge preservation in MLLM fine-tuning.

Background & Motivation

Multimodal Large Language Models (MLLMs), pre-trained on massive image-text datasets, exhibit strong zero-shot capabilities across a wide range of general tasks, such as visual question answering, image captioning, and visual reasoning. However, adapting MLLMs to specific downstream tasks via standard SFT almost inevitably triggers catastrophic forgetting—the model's performance on general benchmarks drops drastically, with newly learned skills acquired at the cost of existing foundational capabilities. This represents one of the most critical challenges in MLLM fine-tuning.

Recent mitigation approaches fall into three main directions: regularization methods add penalty terms to the fine-tuning loss to constrain changes in critical parameters, but balancing hyperparameters is notoriously difficult; parameter-efficient fine-tuning (PEFT, e.g., LoRA, Adapters) freezes most parameters and only trains a fraction of newly added ones to minimize interference, but often suffers from insufficient capacity when facing cross-domain transfer; model merging attempts to merge the pre-trained weights and fine-tuned weights into a unified model that balances both after fine-tuning is complete. However, existing merging methods possess fundamental flaws: Spider uses a heuristic gradient-and-magnitude-based scoring to select parameters, lacking theoretical support; Model Tailor utilizes the Hessian matrix but only optimizes downstream performance, completely ignoring the preservation of general knowledge. These paths represent two extremes—either lacking principled selection, or protecting new capabilities at the complete sacrifice of old knowledge.

The core insight of this work is that the geometric structures of the pre-training and fine-tuning loss surfaces are often anisotropic—the importance of the same parameter direction can vary drastically between the two tasks. Deciding whether to bias towards the pre-trained value or the fine-tuned value for each parameter according to its relative importance (i.e., curvature) allows each task to dominate the final model in its most proficient direction. Core Idea: Formulate model merging as a joint optimization problem that simultaneously minimizes loss increments near both local minima of pre-training and fine-tuning. Leverage second-order Taylor expansion of Hessian matrices to capture local curvature on loss surfaces, deriving a closed-form soft mixing ratio based on relative curvature (CGM). This formulation is further converted into sparse parameter selection to obtain a more robust hard mixing strategy (CGM†). Both methods are verified on MLLMs to demonstrate their capability to preserve general knowledge while maintaining downstream task accuracy.

Method

Overall Architecture

The core idea of the CGM framework is: given pre-trained weights \(w_{\text{pt}}\) and fine-tuned weights \(w_{\text{ft}}\), determine for each parameter how close it should be to either endpoint, thereby minimizing the sum of loss increments on both pre-training and fine-tuning tasks for the final model \(w^*\). The framework consists of three key steps: first, estimate the Hessian diagonal \(h_{\text{pt}}\) on the pre-trained model using a small set of calibration samples (reflecting the sensitivity of the pre-training task to each parameter); second, perform standard SFT on the downstream task to obtain \(w_{\text{ft}}\), while estimating \(h_{\text{ft}}\) on the fine-tuned model (reflecting the sensitivity of the fine-tuning task to each parameter); third, select either the soft mixing (CGM) or hard mixing (CGM†) strategy to generate the final weights.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Pre-trained model w_pt"] --> B["Estimate Hessian diagonal h_pt<br/>on calibration set"]
    C["Downstream SFT"] --> D["Fine-tuned model w_ft"]
    D --> E["Estimate Hessian diagonal h_ft<br/>on fine-tuning set"]
    B --> F{"Mixing Strategy Choice"}
    E --> F
    F -->|CGM Soft Mixing| G["λ_i = α·h_pt/(h_ft+α·h_pt)<br/>Continuous interpolation by curvature ratio"]
    F -->|CGM† Hard Mixing| H["Filter via score c_i<br/>Roll back lowest K% scoring parameters"]
    G --> I["Final model w*"]
    H --> I

Key Designs

1. Joint Optimization Objective and Second-Order Loss Surface Approximation

The major difficulty of the merging problem lies in the absence of pre-training data, given only the fine-tuning set. To preserve general knowledge, the model must leverage information inherent in the pre-trained weights. The pioneering contribution of CGM is formulating this problem as a solvable optimization problem—seeking a weight \(w\) that simultaneously minimizes its deviation from both loss surfaces. Assuming \(w_{\text{pt}}\) and \(w_{\text{ft}}\) are local minima (where gradients are zero) for pre-training and fine-tuning respectively, the loss increments of the two loss functions near their respective minima can be approximated via second-order Taylor expansion, while first-order gradients and higher-order terms are negligible. By adopting a diagonal approximation of the Hessian matrix (reducing complexity from \(O(d^2)\) to \(O(d)\) and decoupling the problem to treat each parameter independently), the joint objective simplifies to the sum of two quadratic penalties. Each parameter \(i\) only needs to optimize its own mixing ratio \(\lambda_i\), setting up a simple quadratic objective in \(\lambda_i\), which paves the way for the subsequent closed-form solution.

2. CGM Soft Mixing: Closed-Form Optimal Mixing Ratio Based on Relative Curvature

Parameterizing \(w_i\) as \(w_{\text{ft}, i} + \lambda_i (w_{\text{pt}, i} - w_{\text{ft}, i})\), substituting this into the joint objective, and taking the derivative with respect to \(\lambda_i\) yields the closed-form optimal solution:

\[ \lambda_i^* = \frac{\alpha\,h^{\text{pt}}_i}{h^{\text{ft}}_i + \alpha\,h^{\text{pt}}_i} \]

The physical meaning of this result is direct and intuitive: when the curvature \(h^{\text{pt}}_i\) of a parameter on the pre-training loss surface is large (meaning this parameter is crucial for the pre-training task), \(\lambda_i\) approaches 1, pushing the final weight closer to the pre-trained value. Conversely, when its curvature \(h^{\text{ft}}_i\) on the fine-tuning surface is large, \(\lambda_i\) approaches 0, biasing the weight towards the fine-tuned value. Here, \(\alpha\) is a balancing coefficient that controls the relative emphasis on the two tasks. This process performs soft mixing—each parameter is continuously interpolated between its fine-tuned and pre-trained values based entirely on their relative curvature, free from heuristic design. Notably, if the curvature is estimated using the Fisher Information Matrix (FIM), this formulation aligns mathematically with Fisher Merging. However, the theoretical derivation of CGM (joint loss minimization) fundamentally differs from Fisher Merging (Laplace posterior approximation), and CGM is not limited to FIM—using true Hessian diagonals estimated by Hutchinson's method yields even better results.

3. CGM† Hard Mixing: Sparse Rollback Based on Curvature-Aware Score

Although soft mixing is mathematically elegant, it alters every single parameter—such dense updates might unnecessarily perturb many parameters that do not require adjustment. CGM† takes a fundamentally different path: instead of mixing from the pre-trained value to the fine-tuned value, it starts from the fine-tuned value and selectively rolls back a portion of parameters to their pre-trained values. Making a binary choice (retaining the fine-tuned value or rolling back to the pre-trained value) for each parameter reduces the complexity of the joint objective into a simple sorting problem. Each parameter receives a score \(c_i = (h^{\text{ft}}_i - \alpha \cdot h^{\text{pt}}_i) \cdot \Delta^2_i\), where \(\Delta_i = w_{\text{pt}, i} - w_{\text{ft}, i}\). This score integrates three dimensions of information: fine-tuning curvature (lower encourages rollback), pre-training curvature (higher encourages rollback), and change magnitude (higher magnitude encourages cautious decisions). Parameters with the lowest scores—tasks being insensitive to fine-tuning but critical to pre-training—are selected for rollback to pre-trained values, while the rest are untouched. Experiments show that rolling back just 10% of parameters is sufficient to recover almost all general knowledge. Notably, the selected parameters display clear vertical band patterns on attention projection matrices. This demonstrates the score identifies structurally meaningful parameter groups rather than a chaotic random distribution.

Loss & Training

CGM itself requires no training process—it is applied as a post-processing step on weights after fine-tuning. The Hessian diagonal is estimated via the empirical Fisher Information Matrix (FIM, which is equivalent to the expectation of squared gradients at local minima). For the pre-trained model, squared gradients are calculated over a calibration set of 8 samples per task to represent \(h^{\text{pt}}\). For the fine-tuned model, squared gradients are accumulated over the fine-tuning dataset to represent \(h^{\text{ft}}\). In experiments, LLaVA-1.5-7B is fine-tuned on the last 12 layers of the language model and the vision projector (1 epoch, lr 1e-4, batch size 64), and Qwen-2.5VL-3B is fine-tuned on the last 6 layers and the vision projector (3 epochs, lr 1e-5). The extra overhead introduced by FIM estimation is about 7.9% (throughput decreases from 3.81 to 3.51 samples/s), which is negligible.

Key Experimental Results

Main Results

Taking the fine-tuning of LLaVA-1.5-7B on OKVQA as an example (core comparison, Hscore is the harmonic mean of general and target tasks):

Method	Pre-Avg (General)	OKVQA (Target)	Hscore	Avg
Pre-trained	66.1	52.8	58.7	64.6
Fine-tuned	50.4	58.0	53.9	51.2
Tailor	61.6	57.6	59.6	61.2
DARE	51.0	51.2	51.1	51.0
Grafting	62.0	57.1	59.5	61.5
Magnitude	54.0	54.5	54.9	55.1
Wanda	52.2	53.9	53.1	52.4
CGM (Ours)	64.2	59.8	61.9	63.7
CGM† (Ours)	65.7	60.2	62.8	65.0

CGM† simultaneously outperforms all baselines in general knowledge preservation (Pre-Avg of 65.7 vs. 50.4 for fine-tuned) and target score (60.2 vs. 58.0 for fine-tuned), recovering general capabilities to a level close to the pre-trained model (66.1) across 8 general benchmarks. Similar trends are observed on Flickr30k and Qwen-2.5VL, where the CGM series achieves the best comprehensive metrics.

Ablation Study

Ablating each component in the scoring function of CGM†, \(c_i = (h_{\text{ft}} - \alpha \cdot h_{\text{pt}}) \cdot \Delta^2\) (on LLaVA-OKVQA):

Ablation Variant	Scoring Metric	Hscore	Avg
Parameter Magnitude Only	\(\Delta^2\)	54.0	54.0
Fine-tuning Curvature + Magnitude	\(h_{\text{ft}} \cdot \Delta^2\)	53.1	52.4
Pre-training Curvature + Magnitude	\(-\alpha \cdot h_{\text{pt}} \cdot \Delta^2\)	61.7	65.1
Complete CGM†	\((h_{\text{ft}} - \alpha \cdot h_{\text{pt}}) \cdot \Delta^2\)	62.8	65.0

Using only \(\Delta^2\) or combined with fine-tuning curvature leads to severe forgetting (Hscore drops around 54). Adding pre-training curvature boosts the Hscore to 61.7, and the complete formulation achieves the best result of 62.8. Both pieces of curvature information are indispensable.

Key Findings

• Sparsity is Sufficient: Even when rolling back only 10% of the parameters (\(K=0.1\)), CGM† can recover nearly all general knowledge. Pre-Avg is virtually insensitive to \(K\) within the 10% to 90% range. This implies that the root cause of catastrophic forgetting might only involve a small subset of crucial parameters.
• Robustness of \(\alpha\): The balancing coefficient \(\alpha\) yields optimal performance around 0.1 to 0.15, yet Pre-Avg remains remarkably stable across a wide range of \(\alpha\) values, indicating that the joint objective itself firmly locks in foundational capabilities.
• Structured Selection: The parameters selected by CGM† in attention projection layers exhibit a clear vertical band pattern, rather than the chaotic, scattered dots seen in the Magnitude baseline. This indicates that the score identifies structured parameter groups at the input-dimension level, showcasing semantic coherence.
• Performance on LaTeX-OCR: For Qwen-2.5VL on the LaTeX-OCR task, CGM reaches an Hscore of 71.0, which is 4.5 points higher than the fine-tuned model (66.5), while keeping the target score nearly identical. This is a rare case where new capabilities are adapted with almost zero loss in general knowledge.

Highlights & Insights

• From Loss Surface Geometry to Closed-Form Optimal Solution: Formulating the seemingly empirical model merging problem as an analytically solvable joint optimization problem. The derived \(\lambda\) formula offers excellent explainability—where higher curvature biases towards the respective model—providing the optimal mathematical realization of the intuition that "each parameter dominates the task it is best at."
• Unified Dual-Mode Framework: Two complementary strategies naturally emerge from the same theoretical framework—dense continuous interpolation for soft mixing, and sparse binary selection for hard mixing (equivalent to applying an \(L_0\) constraint on updates). Experiments confirm that hard mixing is generally superior, validating the assumption of "minimal modification."
• Relation to Fisher Merging: While CGM matches Fisher Merging when using FIM for curvature estimation, the derivation framework of CGM is more general. It achieves better results with true Hessian diagonals (Hutchinson estimation) and is not restricted to FIM.
• Exquisite Ablation Study Design: The four ablation variants correspond directly to splitting the scoring formula term-by-term. Highly distinct contributions from each component make this a textbook example of ablation study design.

Limitations & Future Work

• Diagonal Hessian Approximation Introduces Independence Assumption: Assuming parameter independence ignores interaction effects. Block-diagonal or Kronecker approximations might further enhance performance, though at the expense of higher computational cost.
• Requirement of an Extra FIM Estimation Step: Although the overhead is only 7.9%, it modifies the standard fine-tuning procedure (requiring gradient square accumulation), meaning it cannot easily post-process arbitrary existing fine-tuned checkpoints directly.
• Hessian Estimation Quality Depends on Calibration Set: The pre-training Hessian uses only 8 samples per task. Mismatches in calibration set distribution might introduce estimation bias.
• Unexplored Multi-Task Sequential Fine-Tuning Scenarios: Currently, the method is designed for single-step fine-tuning. Performance in continual learning scenarios across multiple sequential tasks remains to be verified.

Related Work & Insights

• vs. Spider: Spider relies on heuristic scoring (gradient + magnitude) to select preserved parameters, showing high sensitivity to the score design and lacking theoretical grounding. CGM starts from a joint optimization objective to naturally derive a closed-form solution with both theoretical guarantees and superior empirical results.
• vs. Model Tailor: Model Tailor also uses the Hessian matrix but only optimizes downstream loss, which acts as protecting a subset of parameters most critical to the downstream task. CGM considers curvature on both sides to find an optimal balance instead of offering one-sided protection.
• vs. Fisher Merging: While CGM degenerates to Fisher Merging under FIM estimation, it originates from a different formulation, generalizes further, and supports non-FIM Hessian estimation methods for improved performance.
• vs. PEFT Methods like LoRA: PEFT methods mitigate forgetting by reducing trainable parameters but suffer from limited capacity during cross-domain transfer. CGM acts as a post-processing step after fine-tuning and is orthogonal to PEFT.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Formulates model merging as a joint loss minimization problem for the first time and derives closed-form curvature-guided solutions. The dual soft/hard mixing strategies are unified and theoretically rigorous.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Conducts comprehensive main experiments, ablations, hyperparameter analyses, and selection visualizations across two architectures (LLaVA-1.5 and Qwen-2.5VL) and multiple downstream tasks, with an exceptionally clear ablation design.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear motivation, progressive mathematical derivation (from joint objective to soft, then hard mixing), complementary formulas and intuition, and convincing ablations.
• Value: ⭐⭐⭐⭐⭐ Addresses the critical bottleneck of catastrophic forgetting in MLLM fine-tuning. The method is elegant (closed-form) and model-agnostic (no architectural assumptions), making it directly applicable to any pre-trained/fine-tuned checkpoint pair.

Related Papers

• [ACL 2026] Text-Guided Multi-Scale Frequency Representation Adaptation
• [AAAI 2026] UniFit: Towards Universal Virtual Try-on with MLLM-Guided Semantic Alignment
• [AAAI 2026] Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understanding
• [ICLR 2026] Flatness-Guided Test-Time Adaptation for Vision-Language Models
• [CVPR 2026] Vision-Language Model Guided Source-Free Domain Adaptation via Optimal Transport