Mitigate Modality-Asymmetric Forgetting via Stabilizing Visual Representations in CLIP-Based Class-Incremental Learning

Conference: ECCV 2026
Paper: ECCV 2026
Area: Model Compression
Keywords: Class-Incremental Learning / Vision-Language Models / CLIP / Modality-Asymmetric Forgetting / Structural Distillation

TL;DR

To tackle modality-asymmetric forgetting in CLIP-based class-incremental learning—where the visual branch degrades far more severely than the text branch—this paper introduces Visual Anchor-based Structural Transfer (VAST), which builds stable visual anchors from intermediate cross-layer self-similarity and aligns cross-task manifold distributions via matrix Rényi entropy distillation, achieving SOTA performance without replay data.

Background & Motivation

Class-incremental learning (CIL) aims to enable deep learning systems to sequentially acquire new categories without task identifiers while retaining discrimination on previously learned classes. With the emergence of vision-language foundation models, continually adapting pre-trained CLIP via parameter-efficient fine-tuning (PEFT, e.g., LoRA or Adapters) has gained extensive attention for balancing plasticity on downstream tasks with the preservation of zero-shot generalization. However, existing multimodal continual learning paradigms typically assume symmetric forgetting pressure across modalities, overlooking the unequal representation dynamics between the visual and textual encoders during continuous adaptation.

An empirical analysis of parameter sensitivity across tasks reveals a critical asymmetry: the textual encoder consistently maintains low Fisher Information (FI) across deep layers because its concept-level linguistic embeddings provide a rigid semantic reference anchor. Conversely, the visual encoder lacks comparable intrinsic structural protection; its high-level layers exhibit a sharp surge in Fisher Information, indicating severe adaptation pressure and sensitivity to parameter updates. This over-adaptation to current task classes distorts the geometric structure of high-level visual features, causing CLS token drift, cluster dispersion, and acute catastrophic forgetting of past visual knowledge. Crucially, conventional knowledge distillation objectives (such as point-wise MSE or KL divergence) merely force the current model to imitate the unanchored and shifting outputs of the old model, thereby propagating or even amplifying representational instability across successive stages.

To address the visual branch's missing structural reference, this paper approaches the problem by leveraging the task-invariant spatial topology preserved in intermediate layers. The core idea is: propose Visual Anchor-based Structural Transfer (VAST), which extracts second-order self-similarity attention from mid-level visual features fused with text-guided high-level semantics to construct a stable visual structural anchor, and employs matrix-based Rényi entropy distillation to transfer manifold geometry across tasks, fundamentally mitigating modality-asymmetric forgetting.

Method

Overall Architecture

VAST orchestrates continual adaptation around two synergistic thrusts: constructing a reliable visual anchor and transferring geometric structures across tasks. The framework comprises a Visual Representation Reinforcement Module (VRRM) and a distribution-aware structural distillation objective. Input images pass through the CLIP visual encoder; the module extracts task-invariant spatial relations from intermediate layers via self-similarity attention and refines high-level patch values using current textual embeddings. The reinforced dense tokens are injected into the global visual CLS token via attention-based residual fusion, yielding a geometrically grounded visual anchor \(V^t\). During sequential updates, VAST bypasses point-wise matching and instead enforces manifold consistency by computing the matrix-based Rényi joint entropy between feature Gram matrices of consecutive tasks, reinforced by a random permutation regularizer.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    IN["Input Image / Text Pair"] --> ENC["Dual-Tower PEFT Adaptation<br/>Visual ViT & Text Transformer"]
    ENC --> SSA["Structure-Preserving Attention<br/>Intermediate Key-Key 2nd-order topology"]
    SSA --> REFV["Text-Guided Value Refinement<br/>High-layer values weighted by text cosine"]
    REFV --> GSR["Global Semantic Reinforcement & Visual Anchor<br/>Dense token attention residual injection to CLS"]
    GSR --> DASD["Distribution-Aware Structural Distillation<br/>Matrix Rényi joint entropy & permutation reg"]
    DASD --> OUT["Classification & Rehearsal-Free Adaptation"]

Key Designs

1. Structure-Preserving Attention: query-independent spatial topology from intermediate layers Standard multi-head self-attention relies on query-key interactions (\(QK^\top\)). In class-incremental adaptation, query projections are directly driven by downstream task gradients, causing query vectors to drift sharply and distorting spatial attention patterns. To eliminate task-induced query shifts, VAST extracts self-similarity attention from an intermediate layer (layer \(M=7\)), where spatial topology remains relatively stable. Specifically, it computes symmetric first-order affinities directly from key projections: \(A_{self} = \text{softmax}(K_M K_M^\top / \sqrt{d})\). To reinforce local semantic cluster connectivity across neighboring patches, second-order affinities are propagated via matrix multiplication \(A_{rel} = A_{self} \times A_{self}\). The final structure-preserving attention matrix is formulated as: $\(A_M = \text{softmax}\left(\frac{A_{self} A_{rel}}{\sqrt{d}}\right)\)$ Because this attention relies purely on intrinsic visual patch correlations, it remains impervious to classification-head task interference and supplies a stable spatial foundation for the visual anchor.

2. Text-Guided Value Refinement: adaptive weighting of high-layer semantics aligned with text While intermediate spatial attention preserves geometry, building an effective visual anchor also requires high-level semantic abstraction. Directly picking the final layer risks ingesting over-adapted noise. VAST extracts patch features \(\{V_l\}\) from high layers (\(l \in \{9, 10, 11\}\)) and maps them using the frozen final projection matrix \(W_v^{(L)}\). To emphasize features that faithfully reflect true category concepts, each layer's semantic alignment is measured by the average cosine similarity with current task textual embedding \(E_t\): \(s_l = \mathbb{E}_{v \sim V_l}[\cos(v, E_t)]\). Softmax-normalized weights \(\alpha_l = \frac{\exp(s_l)}{\sum_{l'} \exp(s_{l'})}\) aggregate the layers into refined visual values: $\(V_{refine} = \sum_{l \in \{9, 10, 11\}} \alpha_l V_l\)$ This mechanism anchors visual feature selection to the stable textual semantic subspace, ensuring semantic fidelity without overfitting single-layer shifts.

3. Global Semantic Reinforcement: dense spatial feature residual injection into CLS token To integrate spatial topology and refined semantics into the global decision token, dense features are aggregated and projected into the shared embedding space: \(Z_v = \text{Proj}(A_M \cdot V_{refine})\). VAST then employs a lightweight fusion module \(f_\phi\) (a two-layer MLP) to compute attention coefficients between the global CLS token \(V_{cls}\) and each dense token \(z_i \in Z_v\), updating the CLS representation via an attention-based residual connection: $\(V'_{cls} = V_{cls} + \beta \sum_{i} \text{softmax}\big(f_\phi([V_{cls}; z_i])\big) z_i\)$ where \(\beta\) regulates the injection strength. The reinforced feature, denoted as \(V^t\), fuses global semantic discrimination with task-invariant intermediate topology, serving as the robust visual structural anchor for cross-task distillation.

4. Distribution-Aware Structural Distillation: manifold consistency via matrix-based Rényi entropy Traditional point-to-point distillation objectives (e.g., MSE or cosine loss) fail when historical representations drift, as they force rigid alignment to moving targets. VAST introduces matrix-based Rényi joint entropy to align the geometric distributions of feature representations between tasks \(t\) and \(t-1\). Given normalized Gram matrices \(G_V^t\) and \(G_V^{t-1}\) computed over batch features \(V^t\) and \(V^{t-1}\), the visual joint entropy loss is defined as: $\(\mathcal{L}_{vis}^{joint} = S_\alpha(G_V^t, G_V^{t-1}) = \frac{1}{1-\alpha} \log_2 \left( \text{tr}\left( \left[ \frac{G_V^t \circ G_V^{t-1}}{\text{tr}(G_V^t \circ G_V^{t-1})} \right]^\alpha \right) \right)\)$ where \(\circ\) is the Hadamard product and \(\alpha = 1.01\). Minimizing \(S_\alpha\) maximizes structural mutual information between consecutive feature spaces. To avoid fitting spurious sample correlations, a random permutation matrix \(\Pi\) is introduced to form a regularization term: \(\mathcal{L}_{vis}^{reg} = S_\alpha(G_V^t, \Pi G_V^{t-1} \Pi^\top)\). The overall visual distillation loss is \(\mathcal{L}_{vis} = \mathcal{L}_{vis}^{joint} \mathcal{L}_{vis}^{reg}\). The text branch is regularized symmetrically by \(\mathcal{L}_{text}\), resulting in the joint structural objective \(\mathcal{L}_{distill} = \mathcal{L}_{vis} \mathcal{L}_{text}\).

Loss & Training

The overall training objective of VAST for each incremental task balances task adaptation with structural preservation: $\(\mathcal{L} = \mathcal{L}_{ce} + \lambda \mathcal{L}_{distill}\)$ where \(\mathcal{L}_{ce}\) is cross-entropy computed over current-task classes, and \(\lambda = 0.5\) balances the two objectives. The architecture uses an OpenAI pre-trained CLIP ViT-B/16 backbone with LoRA rank 8 inserted into both modalities while keeping the backbone frozen. For hyperparameters, \(\beta = 0.1\), self-similarity is extracted at layer \(M=7\), and \(\alpha = 1.01\). Training runs for 5 epochs per task with batch size 128, optimized via Adam and a cosine learning rate schedule starting at 0.001, operating entirely without historical data replay.

Key Experimental Results

Main Results

VAST is evaluated across five benchmark datasets under multiple incremental splits, measuring both average accuracy over all tasks (Avg) and final accuracy after the last task (Last).

Dataset	Setting	Metric	Continual-CLIP	L2P++	DualPrompt	CODA-Prompt	PROOF (Replay)	MG-CLIP	DMNSP	VAST (Ours)	Gain vs Prev SOTA
CIFAR100	B10_Inc10	Last / Avg	66.66 / 75.15	73.08 / 81.90	72.51 / 81.45	79.01 / 85.94	79.11 / 86.77	79.56 / 86.99	79.57 / 87.23	80.40 / 87.40	+0.83 / +0.17
CIFAR100	B50_Inc10	Last / Avg	66.66 / 69.66	73.27 / 74.41	76.63 / 77.83	77.83 / 79.63	79.73 / 83.32	79.72 / 83.61	82.39 / 86.43	82.48 / 86.62	+0.09 / +0.19
ImageNet-R	B20_Inc20	Last / Avg	71.73 / 78.67	64.32 / 65.55	67.98 / 68.76	71.83 / 73.14	78.50 / 84.11	82.62 / 87.65	81.58 / 87.05	82.83 / 87.61	+0.21 / -0.04
ImageNet-R	B100_Inc20	Last / Avg	71.73 / 75.20	58.73 / 60.23	62.87 / 64.39	66.55 / 68.09	78.03 / 81.06	82.78 / 84.88	83.20 / 85.39	83.85 / 85.54	+0.65 / +0.15
TinyImageNet	B20_Inc20	Last / Avg	65.92 / 74.64	70.98 / 75.38	72.53 / 76.72	75.14 / 79.05	58.76 / 72.03	75.87 / 83.56	75.81 / 82.54	76.05 / 83.98	+0.18 / +0.42
TinyImageNet	B100_Inc20	Last / Avg	65.92 / 69.99	68.52 / 72.02	70.98 / 74.18	72.38 / 76.33	57.21 / 66.52	77.09 / 81.41	78.05 / 81.62	78.72 / 82.19	+0.67 / +0.57
Food101	B10_Inc10	Last / Avg	86.19 / 90.73	78.13 / 85.77	78.49 / 85.95	78.77 / 86.11	84.73 / 90.04	88.19 / 93.30	87.75 / 92.74	88.71 / 93.42	+0.52 / +0.12
Food101	B50_Inc10	Last / Avg	86.19 / 88.14	73.13 / 80.42	72.75 / 80.00	74.13 / 80.98	84.74 / 87.52	87.43 / 91.26	89.31 / 91.78	89.81 / 91.86	+0.50 / +0.08
ImageNet-1K	B100_Inc100	Last / Avg	67.69 / 75.55	69.60 / 79.30	69.79 / 79.39	66.96 / 76.99	63.01 / 73.66	73.33 / 81.71	72.87 / 81.57	73.85 / 81.92	+0.52 / +0.21
ImageNet-1K	B200_Inc200	Last / Avg	67.69 / 74.53	69.24 / 72.84	69.44 / 72.90	69.78 / 73.29	69.12 / 71.40	74.25 / 81.77	74.55 / 82.22	74.87 / 82.43	+0.32 / +0.21

Ablation Study

Table 3 investigates the complementary impact of VRRM and \(\mathcal{L}_{distill}\) against the standard CLIP + LoRA baseline:

Configuration	VRRM Module	\(\mathcal{L}_{distill}\) Objective	CIFAR100 (B10_Inc10) Avg (%)	CIFAR100 (B10_Inc10) Last (%)	ImageNet-R (B20_Inc20) Avg (%)	ImageNet-R (B20_Inc20) Last (%)	Note
Baseline	✗	✗	84.81 ± 0.21	77.33 ± 0.18	84.01 ± 0.19	79.15 ± 0.22	Pure LoRA fine-tuning with severe drift
Only VRRM	✓	✗	86.90 ± 0.08	79.35 ± 0.10	87.24 ± 0.07	81.95 ± 0.10	Visual anchor alone boosts Avg by +2.09% / +3.23%
Only Distill	✗	✓	86.96 ± 0.07	78.93 ± 0.11	87.32 ± 0.09	82.32 ± 0.09	Entropy distillation alone boosts Avg by +2.15% / +3.31%
Full VAST	✓	✓	87.40 ± 0.06	80.40 ± 0.08	87.61 ± 0.07	82.83 ± 0.07	Combined modules maximize stability and performance

Table 4 reports the Fisher Discriminant Ratio (FDR, between-class variance over within-class variance) from Task 0 to Task 1, quantitatively tracking representation drift:

Dataset	Method	\(\text{FDR} \uparrow (0 \to 1)\)	\(\text{inter-class} \uparrow (0 \to 1)\)	\(\text{intra-class} \downarrow (0 \to 1)\)	Cluster Evolution Verdict
CIFAR100	CLIP + LoRA Baseline	\(2.19 \to 2.08 \ (\downarrow)\)	\(0.19 \to 0.18 \ (\downarrow)\)	\(0.09 \to 0.08 \ (\downarrow)\)	Inter-class margin shrinks; cluster confusion
CIFAR100	VAST (Ours)	\(2.48 \to 2.97 \ (\uparrow)\)	\(0.24 \to 0.29 \ (\uparrow)\)	\(0.10 \to 0.10 \ (\to)\)	Inter-class variance expands; clusters remain sharp
TinyImageNet	CLIP + LoRA Baseline	\(1.75 \to 1.74 \ (\downarrow)\)	\(0.19 \to 0.18 \ (\downarrow)\)	\(0.11 \to 0.10 \ (\downarrow)\)	Boundary erosion on prior categories
TinyImageNet	VAST (Ours)	\(2.00 \to 2.35 \ (\uparrow)\)	\(0.23 \to 0.28 \ (\uparrow)\)	\(0.12 \to 0.12 \ (\to)\)	FDR increases counter-intuitively, confirming stability

Key Findings

• Synergy resolves distillation blind spots: Standalone distillation (Only Distill) improves performance but struggles because the distillation reference itself drifts over tasks. VRRM anchors the intermediate geometry, enabling the full VAST framework to improve CIFAR100 final accuracy by +3.07% over LoRA (80.40% vs. 77.33%) while cutting variance from 0.18 to 0.08.
• Retention of out-of-distribution zero-shot capabilities: When evaluated on unseen datasets (Food101, ImageNet-100, ImageNet-1K) following CIFAR100 adaptation (Table 2), replay methods like PROOF suffer catastrophic zero-shot drops (52.49% average accuracy due to downstream overfitting). In contrast, VAST achieves 74.28%—closely tracking the frozen CLIP baseline (75.35%) and surpassing non-replay competitors ZSCL (70.77%) and DMNSP (71.10%).
• Middle-layer trade-off: Layer sensitivity analysis (Figure 7b) demonstrates that layer 7 achieves optimal Avg accuracy (87.40%), striking the best balance between task-invariant spatial layout in shallow layers and semantic richness in deep layers.

Highlights & Insights

• Diagnosing modality-asymmetric forgetting: Through Fisher Information analysis across transformer blocks, the authors reveal that dual-tower VLMs suffer from a fundamental imbalance: language encoders serve as natural semantic anchors while visual encoders experience severe parameter sensitivity and representation collapse.
• Mid-level second-order self-similarity as a structural anchor: Bypassing task-sensitive query projections, the method leverages key-key self-correlations and their second-order matrix product in intermediate layers to construct an invariant geometric scaffold.
• Information-theoretic distribution alignment: Replacing naive point-to-point loss functions with matrix Rényi joint entropy allows distillation to preserve global manifold geometry while random permutation regularization eliminates spurious dependencies.

Limitations & Future Work

• Quadratic computational complexity with batch size: Computing matrix Rényi joint entropy requires trace operations over \(N \times N\) Gram matrices, incurring \(\mathcal{O}(N^2)\) memory and time overhead that scales unfavorably for large batch training.
• Fixed intermediate layer selection: The anchor extraction layer is statically set to layer 7 for ViT-B/16; dynamic or automated layer selection strategies across diverse backbones (e.g., ViT-L or hierarchical vision models) remain unexplored.
• Future directions: Investigating low-rank randomized approximations for kernel Gram matrices and extending visual anchor transfer to video continual learning and embodied agent policies.

Related Work & Insights

• vs PROOF / RAPF: Replay-based approaches store exemplar images or feature prototypes, which violate strict privacy constraints and cause severe overfitting that degrades zero-shot generalization (PROOF zero-shot drops to 52.49%); VAST is strictly rehearsal-free and maintains 74.28% zero-shot accuracy.
• vs ZSCL / C-CLIP: Traditional distillation methods enforce point-wise feature mimicry via MSE or contrastive alignment, which amplifies drift when old features lack stability; VAST first anchors visual geometry via VRRM and then matches distributions via matrix entropy.
• vs DMNSP: DMNSP restricts parameter updates using dynamic null-space projections; VAST approaches stability from feature-space geometry and modality asymmetry, yielding superior cluster separation (FDR 2.97 vs. 2.08 on CIFAR100).

Rating

• Novelty: ⭐⭐⭐⭐⭐ Formulates modality-asymmetric forgetting from a Fisher Information perspective, proposing intermediate-layer self-similarity visual anchors and matrix Rényi entropy distillation.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorously tested across five datasets with multiple splits, providing zero-shot evaluations, FDR geometric drift metrics, layer sensitivity sweeps, and attention map visual inspection.
• Writing Quality: ⭐⭐⭐⭐⭐ Highly organized narrative, clear problem formulation, mathematically grounded derivations, and well-designed figures.
• Value: ⭐⭐⭐⭐⭐ Provides a foundational framework for multimodal continual learning, demonstrating that stabilizing visual representations is critical for rehearsal-free knowledge retention.