Skip to content

Dual-Generalization-aware Minimization for Continual Fine-Tuning of Vision-Language Models

Conference: ECCV 2026
Paper: ECCV Official
Area: Multimodal VLM
Keywords: Continual Learning, Vision-Language Models, Sharpness-Aware Minimization, Zero-Shot Generalization, Loss Landscape Geometry

TL;DR

To prevent catastrophic forgetting and zero-shot capability collapse caused by single-task flatness failing under sequential distribution shifts, this paper proposes Dual-Generalization-aware Minimization (DGM), an optimization framework that couples pre-trained zero-shot alignment with task-specific visual uncertainty into adversarial perturbation generation to reach jointly flat minima.

Background & Motivation

Large-scale pre-trained vision-language models (VLMs), exemplified by CLIP, have emerged as foundation architectures across computer vision due to their open-vocabulary alignment and remarkable zero-shot transferability. However, adapting these foundation models to real-world sequential tasks under class-incremental learning (CIL) setups presents a fundamental dilemma: standard fine-tuning not only suffers from catastrophic forgetting of previously learned task knowledge, but also rapidly degrades the versatile zero-shot generalization inherited from pre-training. Existing continual learning approaches for VLMs primarily focus on parameter isolation (e.g., dynamic routing, task-specific adapters, or mixture-of-experts modules) and representation alignment constraints. Nevertheless, they largely overlook how the optimization process and loss landscape geometry dictate continual generalization behavior.

Recent optimization insights reveal that solutions located in flatter minima exhibit stronger resilience to distribution shifts, whereas sharp minima correspond to severe overfitting and rapid forgetting. However, naively applying standard flatness-aware methods like Sharpness-Aware Minimization (SAM) to VLM continual adaptation reveals a fundamental breakdown: continual adaptation entails a multi-distribution setting where the model constantly navigates between a broad pre-training distribution and narrow, sequential task-specific distributions. Standard SAM seeks parameter flatness exclusively on the current task's classification loss. Empirical loss landscape evaluations show that flat basins optimized on an isolated task quickly drift away as adaptation progresses, while the loss surface on unseen data (such as ImageNet-100 zero-shot evaluation) degrades into irregular, steep terrains.

Consequently, seeking parameter flatness on the current task distribution alone fails to build an effective safeguard against evolving distribution drift. The optimization mechanism must explicitly sense both the foundational pre-trained prior and downstream intra-task distribution variations. Core idea: expand adversarial perturbation generation from single-task loss into dual probes capturing pre-trained knowledge preservation and task-specific visual robustness (DGM), steering parameter convergence toward jointly flat minima across both pre-trained and task-adapted spaces.

Method

Overall Architecture

Dual-Generalization-aware Minimization (DGM) is a plug-and-play optimization framework requiring no additional parameters or architectural modifications. It seamlessly integrates into full-parameter tuning, parameter-efficient adaptation (Adapter, LoRA), and mixture-of-experts continual learning paradigms. Rather than relying solely on the current task loss to guide adversarial perturbations, DGM defines a composite perturbation objective \(\mathcal{L}_{perturb}\) composed of a zero-shot generalization probe and a task-specific robustness probe. In each training iteration, DGM identifies parameter perturbations along directions that would most severely compromise pre-trained representations or task robustness, and then computes base classification gradients at the perturbed location to update parameters toward jointly flat basins.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Current task batch samples (x, y)"] --> B["Dual-Generalization Perturbation Formulation<br/>Aggregate task loss and composite perturbation loss"]
    B --> C["Zero-Shot Generalization Probe<br/>Align evolving model with frozen pre-trained distribution"]
    B --> D["Task-Specific Robustness Probe<br/>Inject latent visual feature moment perturbations"]
    C --> E["Adversarial Perturbation Computation<br/>Compute neighborhood perturbation ฮต via joint gradient ascent"]
    D --> E
    E --> F["Perturbed Gradient Evaluation & Update<br/>Evaluate base gradient at ฮธ+ฮต and update parameters"]

Key Designs

1. Dual-Generalization Perturbation Formulation: Expanding single-task ascent to multi-distribution curvature sensing

Conventional Sharpness-Aware Minimization (SAM) determines parameter perturbation along the gradient ascent direction of the current task loss \(\mathcal{L}_{train}\), which leads the model into narrow basins that remain highly fragile to unseen classes and historic tasks. To guide optimization toward parameters resilient to both pre-trained knowledge decay and task-specific overfitting, DGM introduces a composite perturbation loss \(\mathcal{L}_{perturb}(\theta; \mathcal{D}_t) = \lambda_1 \mathcal{L}_{ZS}(\theta, \theta_0; \mathcal{D}_t) + \lambda_2 \mathcal{L}_{NV}(\theta; \hat{\mathcal{D}}_t)\). The minimax objective is formulated as:

\[\min_{\theta} \max_{\|\epsilon\|_2 \le \rho} \left[ \mathcal{L}_{train}(\theta + \epsilon; \mathcal{D}_t) + \mathcal{L}_{perturb}(\theta + \epsilon; \mathcal{D}_t) \right]\]

Via first-order Taylor approximation, the adversarial perturbation \(\epsilon\) incorporates the gradient sum of the task classification loss and the dual perturbation losses, driving the inner maximization to simultaneously probe current task curvature and multi-distribution vulnerability.

2. Zero-Shot Generalization Probe: Anchoring parameter sensitivity with frozen model priors

To preserve the open-vocabulary transferability established during large-scale pre-training, this probe uses the unperturbed, frozen pre-trained model \(f_0\) (with parameters \(\theta_0\)) as an anchor. For each input sample \(x \in \mathcal{D}_t\), the evolving model \(f_t\) outputs predicted class probabilities \(P_t(x)\), while the frozen model provides zero-shot reference probabilities \(P_{ZS}(x)\). The distributional divergence is quantified via Kullback-Leibler (KL) divergence:

\[\mathcal{L}_{ZS} = \text{KL}\big(P_t(x) \parallel P_{ZS}(x)\big)\]

Incorporating \(\nabla_\theta \mathcal{L}_{ZS}\) into the perturbation step ensures that parameter directions that severely distort the foundation model's zero-shot representations are actively explored. The outer minimization step then dampens sensitivity along these fragile directions, effectively anchoring the solution within flat valleys of pre-trained feature space.

3. Task-Specific Robustness Probe: Simulating intra-task distribution shifts via latent feature noise

To counter overfitting on limited current-task training samples and increase resilience against visual domain variations, this probe introduces stochastic feature perturbation. For each sample representation in latent visual space, the sample-wise mean \(\mu\) and variance \(\sigma^2\) across feature dimensions are computed to model visual uncertainty. A perturbed variant \(\tilde{x}_i\) is synthesized using noise ratio \(\delta\):

\[\tilde{x}_i = \delta \mathcal{N}(\mu, \sigma^2) + (1 - \delta) x_i\]

The resulting perturbed dataset \(\hat{\mathcal{D}}_t = \{(\tilde{x}_i, y_i)\}_{i=1}^{N_t}\) is evaluated under the base task loss \(\mathcal{L}_{NV} = \mathcal{L}_{train}(\theta; \hat{\mathcal{D}}_t)\). The gradient \(\nabla_\theta \mathcal{L}_{NV}\) steers the perturbation toward local worst-case visual variations, forcing the model to converge to regions that are flat with respect to visual feature fluctuations.

Loss & Training

The complete training workflow preserves the computational elegance of standard SAM: 1. Perturbation Estimation: Compute base classification loss \(\mathcal{L}_{train}\), zero-shot alignment loss \(\mathcal{L}_{ZS}\), and noisy visual loss \(\mathcal{L}_{NV}\), obtaining the joint gradient \(G = \nabla_\theta [\mathcal{L}_{train} + \lambda_1 \mathcal{L}_{ZS} + \lambda_2 \mathcal{L}_{NV}]\); 2. Adversarial Step: Normalize \(G\) under the \(\ell_2\) norm to generate the parameter perturbation \(\epsilon = \rho \frac{G}{\|G\|_2}\); 3. Weight Update: Re-evaluate the base training gradient at the perturbed location \(g = \nabla_\theta \mathcal{L}_{train}(\theta)|_{\theta+\epsilon}\), and apply the standard base optimizer step (e.g., Adam) \(\theta \leftarrow \theta - \eta g\). Each iteration requires 2 forward and 2 backward passes. Because feature extraction is shared between base and perturbed passes, practical computational overhead remains modest with zero additional parameter memory.

Key Experimental Results

Main Results

DGM is evaluated across six class-incremental learning benchmarks using CLIP ViT-B/16: CIFAR-100, ImageNet-R, CUB-200, Tiny-ImageNet, ImageNet-100, and ImageNet-1K. The baselines span full-parameter fine-tuning (ZSCL), adapter-based architectures (RAPF, DMNSP, MoE4CL), and LoRA-based methods (Finetune, MG-CLIP). Representative results showing Last Accuracy (Last) and Average Accuracy (Avg.) are detailed below:

Dataset Protocol Baseline Method Baseline Last / Avg. (%) Ours (w/ DGM) Last / Avg. (%) Gain (Last / Avg.)
CIFAR-100 B0_Inc10 Finetune (LoRA) 79.30 / 86.88 81.12 / 87.69 +1.82 / +0.81
CIFAR-100 B0_Inc10 ZSCL (Full-FT) 78.33 / 85.34 80.45 / 86.81 +2.12 / +1.47
CIFAR-100 B0_Inc10 MoE4CL (Adapter) 78.95 / 86.05 80.62 / 87.12 +1.67 / +1.07
ImageNet-R B0_Inc20 Finetune (LoRA) 81.63 / 87.15 82.47 / 87.54 +0.84 / +0.39
ImageNet-R B0_Inc20 MoE4CL (Adapter) 81.88 / 87.32 83.07 / 88.03 +1.19 / +0.71
CUB-200 B0_Inc20 Finetune (LoRA) 58.97 / 70.31 60.75 / 71.24 +1.78 / +0.93
CUB-200 B0_Inc20 MG-CLIP 65.72 / 73.48 67.05 / 74.75 +1.33 / +1.27
Tiny-ImageNet B0_Inc20 ZSCL (Full-FT) 72.78 / 81.48 75.46 / 83.06 +2.68 / +1.58
ImageNet-100 B0_Inc10 ZSCL (Full-FT) 64.32 / 78.51 66.60 / 80.26 +2.28 / +1.75
ImageNet-1K B0_Inc100 Finetune (LoRA) 71.83 / 80.97 73.40 / 82.08 +1.57 / +1.11

In zero-shot evaluation on unseen datasets (Food-101, ImageNet-100, ImageNet-1K) following CIFAR-100 continual adaptation, ZSCL equipped with DGM attains an average accuracy of 75.90%, delivering a +5.13% improvement over raw ZSCL (70.77%) and surpassing the unadapted CLIP zero-shot baseline (75.35%).

Ablation Study

Ablation on perturbation objective components using LoRA Finetune across three benchmarks confirms the individual and joint benefits of each probe:

Perturbation Components (\(\mathcal{L}_{train}\) / \(\mathcal{L}_{NV}\) / \(\mathcal{L}_{ZS}\)) CIFAR-100 Last / Avg. (%) CUB-200 Last / Avg. (%) Tiny-ImageNet Last / Avg. (%) Note
Only \(\mathcal{L}_{train}\) (Standard SAM) 80.42 / 87.38 60.40 / 71.18 76.20 / 84.20 Single-task flatness without cross-distribution probes
\(\mathcal{L}_{train} + \mathcal{L}_{NV}\) 80.92 / 87.66 61.43 / 71.28 76.91 / 84.30 Adds visual noise to bolster task-specific robustness
\(\mathcal{L}_{train} + \mathcal{L}_{ZS}\) 81.20 / 87.90 61.35 / 71.27 76.85 / 84.22 Adds zero-shot alignment to retain pre-trained generality
Full DGM (\(\mathcal{L}_{train} + \mathcal{L}_{NV} + \mathcal{L}_{ZS}\)) 81.48 / 87.99 61.80 / 71.32 77.05 / 84.32 Dual probes combined achieve best overall stability

Key Findings

  • Complementarity of Dual Probes: Both \(\mathcal{L}_{ZS}\) and \(\mathcal{L}_{NV}\) individually surpass standard SAM, while their joint integration delivers optimal performance. \(\mathcal{L}_{ZS}\) preserves foundational semantic representations against drift, whereas \(\mathcal{L}_{NV}\) flattens the local loss landscape against visual sample variations.
  • Loss Landscape Flattening: Hessian spectrum analysis demonstrates that DGM substantially lowers the spectral norm \(\lambda_{max}\) and trace compared to standard fine-tuning. This smoother curvature directly yields higher Forward Transfer (FWT) and less negative Backward Transfer (BWT).
  • Accelerated Convergence vs. Compute: Although each iteration requires extra perturbation passes, DGM models match or exceed the performance of 3-4 baseline training epochs within a single epoch, providing an efficient compute-accuracy trade-off.

Highlights & Insights

  • Geometric Perspective on VLM Continual Adaptation: Rather than relying purely on architectural expansion or empirical replay, the paper demonstrates that single-task sharpness optimization is insufficient under sequential distribution drift, establishing the necessity of jointly flat multi-distribution minima.
  • Dual Perturbation Guidance: Cleverly converting teacher-student divergence and latent feature uncertainty into adversarial perturbation gradients ensures that parameter exploration actively targets the most fragile directions of pre-trained knowledge and visual variance.
  • Plug-and-Play Generality: Operates strictly at the optimizer level without modifying network architectures or adding inference parameters, making it universally applicable across diverse adaptation paradigms.

Limitations & Future Work

  • Computational Overhead: Similar to standard SAM, DGM requires two forward and backward passes per step, which increases per-step wall-clock time compared to standard first-order optimizers.
  • Feature Perturbation Simplicity: The feature-space Gaussian perturbation \(\mathcal{L}_{NV}\) relies on overall feature moment statistics, which may overlook fine-grained structural or spatial correlations in dense prediction tasks.
  • Extension to Multimodal LLMs: Future investigations could explore extending dual-generalization perturbation minimization to generative multimodal models and instruction-tuning continual adaptation.
  • vs SAM / GAM / C-Flat: Prior sharpness-aware optimizers target static single tasks or from-scratch continual learning; DGM addresses the unique vulnerability of foundation models where single-task flatness fails under multi-distribution sequential shifts.
  • vs ZSCL / RAPF: While ZSCL and RAPF impose outer loss regularization or parameter fusion, DGM reshapes inner optimization dynamics and can be stacked directly on top of these frameworks for cumulative gains.

Rating

  • Novelty: โญโญโญโญโ˜† [Geometric analysis of multi-distribution drift in VLM continual fine-tuning; elegant dual-probe perturbation design]
  • Experimental Thoroughness: โญโญโญโญโญ [Extensive evaluations across 6 CIL benchmarks, 6 adaptation frameworks, Hessian spectrum analysis, and unseen zero-shot datasets]
  • Writing Quality: โญโญโญโญโญ [Clear structural flow from geometric empirical observations to targeted mathematical design and empirical validation]
  • Value: โญโญโญโญโญ [A parameter-free, model-agnostic continual learning optimizer offering universal improvements for VLM adaptation]