title: >- [Paper Note] SCoT: Similarity-guided Conflict-aware Task Consolidation for Continual VQA description: >- [ECCV 2026][Multimodal VLM][Continual Learning] SCoT introduces anchor-relative task vector consolidation with layer-wise cosine similarity modulation, conditional conflict projection, and depth-aware stabilization to achieve near-zero forgetting and positive backward transfer in continual VQA. tags: - ECCV 2026 - Multimodal VLM - Continual Learning - Visual Question Answering - Task Vectors date: 2026-09-19 content_hash: 790b6373e1e0ec14

SCoT: Similarity-guided Conflict-aware Task Consolidation for Continual VQA¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://anand-patel05.github.io/SCoT
Area: Multimodal VLM
Keywords: Continual Learning, Visual Question Answering, Task Vectors, Parameter-space Consolidation, Positive Backward Transfer

TL;DR¶

SCoT introduces an anchor-relative task vector consolidation framework that measures layer-wise cosine alignment to adaptively modulate plasticity, removes destructive directional conflicts via minimal-norm conditional projection, and stabilizes composition-sensitive encoder-decoder interfaces, achieving near-zero forgetting (AF 0.07 and -1.90) and rare positive backward transfer while outperforming joint multi-task training on VQAv2 and NExT-QA.

Background & Motivation¶

Modern vision-language models (VLMs) have achieved remarkable success in visual question answering (VQA) through pretraining on massive web-scale cross-modal data. However, real-world deployment requires these systems to continually adapt to non-stationary environments, absorbing a continuous stream of new question types, reasoning patterns, specialized domains, and visual concepts. Naive sequential fine-tuning across successive tasks triggers catastrophic forgetting, abruptly destabilizing cross-modal alignment and erasing foundational reasoning capabilities inherited from the pretrained backbone. Prevailing continual VQA approaches adapt unimodal strategies such as replay buffers, activation penalties, or architectural heuristics, but they predominantly treat each task update as an opaque optimization step without explicit control over how task parameters accumulate and interact in weight space.

The core tension stems from the opposing dynamics of synergy and interference across layers when learning non-stationary task streams. While related tasks share visual grounding and linguistic priors, successive gradient updates frequently point in conflicting directions, leading conventional rehearsal or regularization methods to strike brittle compromises between stability and plasticity. Furthermore, while emerging parameter-space techniques like task vectors and model merging offer explicit control over weight updates, most existing frameworks rely on post-hoc offline merging of independently trained models—a luxury unavailable in online streaming setups. Recent continual merging heuristics like MagMax focus solely on parameter magnitudes, remaining oblivious to directional conflicts and behaving overly conservatively, which throttles both forward and backward knowledge transfer.

This paper addresses this fundamental bottleneck by reframing continual multimodal learning as directed geometric consolidation directly in parameter space. Instead of allowing updates to compete for shared capacity, the model anchors every task update relative to a fixed pretrained backbone and analyzes their geometric alignment layer by layer. Core idea: represent sequential tasks as anchor-relative task vectors, adaptively scale consolidation strength via layer-wise cosine alignment, surgically excise destructive directional conflicts through minimal-norm conditional projection, and anchor composition-sensitive encoder-decoder interfaces with depth-aware stabilization to turn continual learning into a constructive rather than competitive process.

Method¶

Overall Architecture¶

The SCoT framework anchors the continual adaptation of an encoder-decoder multimodal transformer around a fixed pretrained parameter anchor $\theta_0$. As each task $T_i$ arrives from the sequential task stream, the model is initialized with the current shared consolidated parameters $\bar{\theta}_{i-1}$ and fine-tuned on the task data (supplemented with a modest replay buffer) to obtain fine-tuned weights $\theta_i^{\mathrm{ft}}$. The task-induced displacement $t_i$ is extracted relative to $\theta_0$. SCoT's continual task consolidation engine then integrates $t_i$ into the cumulative consolidated offset $m_i$ across layers: it evaluates the layer-wise cosine similarity between the incoming update and the accumulated offset, applies conditional geometric projection to eliminate negative interference, maps the similarity score into a bounded consolidation weight, and caps the update strength at composition-critical interface layers. The updated shared model $\bar{\theta}_i = \theta_0 + m_i$ is immediately ready for deployment and serves as the starting state for task $T_{i+1}$.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Task Stream T_i<br/>and Shared Weights θ_bar_i-1"] --> B["Task Fine-Tuning & Vector Extraction<br/>Calculate Displacement t_i Relative to θ_0"]
    B --> C["Anchor-Relative State Representation<br/>Decouple Task Displacement from Offset m_i-1"]
    C --> D["Conflict-Aware Layer-wise Projection<br/>Detect Angle Conflict & Project Destructive Component"]
    D --> E["Similarity-Guided Layer-wise Consolidation<br/>Map Cosine Alignment to Interpolation Weight α"]
    E --> F["Depth-Aware Stabilization<br/>Cap Consolidation at Encoder-Decoder Interface"]
    F --> G["Output: Consolidated Model θ_bar_i<br/>Deploy and Initialize Task T_i+1"]

Key Designs¶

1. Anchor-Relative State Representation: Decoupling task learning from cumulative consolidation

Traditional continual learning updates model weights along an unanchored sequential trajectory, causing severe parameter drift and obscuring the geometric orientation of task-specific representations. SCoT maintains a fixed pretrained checkpoint $\theta_0$ as a global origin. For task $T_i$, after fine-tuning from $\bar{\theta}_{i-1}$ to yield $\theta_i^{\mathrm{ft}}$, the task knowledge is encoded as a displacement from $\theta_0$: $$t_i = \theta_i^{\mathrm{ft}} - \theta_0$$ The shared state is maintained as a single cumulative consolidated offset $m_i$ (initialized as $m_0 = \mathbf{0}$), reconstructing the operational model at any stage via: $$\bar{\theta}_i = \theta_0 + m_i$$ This formulation cleanly decouples what a task modifies ($t_i$) from how knowledge is accumulated over time (the update law for $m_i$), allowing the model to perform precise vector-space algebra without compounding optimization noise.

2. Similarity-Guided Layer-Wise Consolidation: Adaptive plasticity from first-order parameter geometry

Multimodal transformers distribute semantic and functional representations non-uniformly across depth, rendering uniform merging suboptimal and second-order importance metrics (such as Fisher information matrices) computationally prohibitive for online streaming. SCoT derives an efficient, layer-specific plasticity signal directly from first-order parameter geometry. For each layer $\ell$, it measures the cosine alignment $s_{\ell,i}$ between the incoming task vector $t_i^{(\ell)}$ and the consolidated offset $m_{i-1}^{(\ell)}$: $$s_{\ell,i} = \frac{\langle t_i^{(\ell)}, m_{i-1}^{(\ell)} \rangle}{\|t_i^{(\ell)}\| \|m_{i-1}^{(\ell)}\| + \epsilon}$$ where $\epsilon$ is a small constant for numerical stability. This alignment is mapped monotonically into a bounded consolidation weight $[\alpha_{\min}, \alpha_{\max}]$: $$\alpha_{\ell,i} = \text{clip}\left(\alpha_{\min} + (\alpha_{\max} - \alpha_{\min}) \frac{s_{\ell,i} + 1}{2}\right)$$ Layers exhibiting strong synergy ($s_{\ell,i} \to 1$) undergo aggressive consolidation, whereas weakly aligned or orthogonal layers are integrated conservatively to preserve foundational representations. The consolidated offset is updated via weighted interpolation: $$m_i^{(\ell)} = (1 - \alpha_{\ell,i}) m_{i-1}^{(\ell)} + \alpha_{\ell,i} \Delta_i^{(\ell)}$$ where $\Delta_i^{(\ell)}$ represents the conflict-free effective task displacement, ensuring that consolidation scales only task-specific novelty rather than double-counting past updates.

3. Conflict-Aware Layer-Wise Projection: Minimal-norm surgical removal of destructive interference

Catastrophic forgetting fundamentally arises from destructive interference, which occurs when parameter updates for a new task oppose previously established directions. When $s_{\ell,i} < 0$, the incoming task vector actively negates consolidated knowledge in layer $\ell$. Rather than uniformly decaying the vector magnitude, SCoT applies a conditional geometric projection that excises only the conflicting component: $$\tilde{t}_i^{(\ell)} = \begin{cases} t_i^{(\ell)} - \frac{\langle t_i^{(\ell)}, m_{i-1}^{(\ell)} \rangle}{\|m_{i-1}^{(\ell)}\|^2 + \epsilon} m_{i-1}^{(\ell)}, & \text{if } s_{\ell,i} < 0 \\ t_i^{(\ell)}, & \text{otherwise} \end{cases}$$ This projection constitutes the minimum-norm perturbation of $t_i^{(\ell)}$ that guarantees non-negative inner product $\langle \tilde{t}_i^{(\ell)}, m_{i-1}^{(\ell)} \rangle \ge 0$. By setting $\Delta_i^{(\ell)} = \tilde{t}_i^{(\ell)}$, SCoT completely removes destructive anti-parallel vectors while fully preserving aligned and orthogonal components essential for new knowledge acquisition and forward transfer.

4. Depth-Aware Stabilization: Preserving compositional structure at the encoder-decoder interface

Continual VQA demands not only retention of isolated facts, but also compositional generalization—recombining known reasoning skills with novel visual objects. Empirical analysis across layer groups reveals that compositional sensitivity in encoder-decoder architectures is concentrated at the interface between high-level visual-textual encoding and generative decoding. To shield these vulnerable structures from parameter erosion, SCoT applies a targeted depth-dependent upper bound on consolidation: $$\alpha_{\ell,i} \leftarrow \min(\alpha_{\ell,i}, \alpha_\ell^{\mathrm{cap}}), \quad \ell \in \mathcal{L}_{\mathrm{comp}}$$ Specifically, late encoder layers in $\mathcal{L}_{\mathrm{comp}}$ are tightly constrained with $\alpha_\ell^{\mathrm{cap}} = \alpha_{\min}$, while early decoder layers receive a moderate cap of $1.5 \times \alpha_{\min}$. Remaining layers retain full adaptive plasticity up to $\alpha_{\max}$, establishing an optimal balance between continual adaptability and compositional resilience.

A Worked Example¶

Consider a continual VQA sequence where the model has accumulated diverse object recognition and counting skills in $m_{i-1}$, and encounters a new task requiring spatial relationship reasoning over novel object classes. 1. Fine-Tuning & Displacement: The model fine-tunes from $\bar{\theta}_{i-1}$ on the spatial task, yielding $t_i = \theta_i^{\mathrm{ft}} - \theta_0$. 2. Aligned Feature Processing: In the early visual projection layers, features are complementary to existing visual embeddings, yielding $s_{\ell,i} = +0.72 > 0$. No projection is triggered; the consolidation weight is computed as $\alpha_{\ell,i} \approx 0.387$, enabling substantial plasticity to incorporate new visual primitives. 3. Conflict Detection & Projection: In encoder layer 9, attention query-key matrices for spatial queries conflict with prior category-discrimination directions, yielding $s_{\ell,i} = -0.34 < 0$. The conflict-aware engine projects $t_i^{(\ell)}$ onto the hyperplane orthogonal to $m_{i-1}^{(\ell)}$, eliminating the opposing component. Furthermore, because layer 9 falls within $\mathcal{L}_{\mathrm{comp}}$, its consolidation rate is capped at $\alpha_{\min} = 0.08$. 4. Knowledge Consolidation: The updated offset $m_i$ smoothly integrates spatial reasoning without corrupting earlier counting and recognition attention pathways, allowing subsequent evaluation on previous counting tasks to demonstrate performance gains rather than degradation.

Loss & Training¶

During the task fine-tuning stage, the generative VQA model optimizes the standard autoregressive cross-entropy loss: $$\mathcal{L}_{\mathrm{FT}} = - \sum_{k=1}^{|Y|} \log P_{\theta}\left(Y_k \mid Y_{<k}, V, Q\right)$$ Training is conducted using the Adam optimizer with an initial learning rate of $10^{-4}$ and batch size 80 for 3 epochs per task. The episodic replay buffer $M$ stores 5,000 samples for VQAv2 and 500 samples for NExT-QA. Consolidation hyperparameters are set to $\alpha_{\min} = 0.08$ and $\alpha_{\max} = 0.45$.

Key Experimental Results¶

Main Results¶

SCoT is evaluated against state-of-the-art continual learning baselines on VQAv2 and NExT-QA under both standard testing (Std. Test) and novel composition testing (Comp. Test). Metrics include Final Average Performance (AP, $\uparrow$) and Average Forgetting (AF, $\downarrow$, negative values denote performance gains on past tasks).

Method	#Mem	VQAv2 Std AP(↑)	VQAv2 Std AF(↓)	VQAv2 Comp AP(↑)	VQAv2 Comp AF(↓)	NExT-QA Std AP(↑)	NExT-QA Std AF(↓)	NExT-QA Comp AP(↑)	NExT-QA Comp AF(↓)
Joint (Upper Bound)	–	41.32	–	40.88	–	35.92	–	36.24	–
Vanilla (Seq FT)	None	14.92	30.80	11.79	27.16	12.68	25.94	12.59	28.04
EWC	None	15.77	30.62	12.83	28.16	13.01	24.06	11.91	27.44
MAS	None	20.56	11.16	23.90	6.24	18.04	10.07	21.12	10.09
ER	5000 / 500	36.99	5.99	33.78	5.76	30.55	4.91	32.20	5.57
DER	5000 / 500	35.35	8.62	31.52	8.59	26.17	5.12	21.56	12.68
VS	5000 / 500	34.03	8.79	32.96	5.78	28.13	4.45	29.47	6.14
VQACL	5000 / 500	37.46	6.96	35.40	4.90	30.86	4.12	33.85	3.80
QUAD	5000 / 500	39.25	4.91	40.00	3.81	31.70	2.91	33.21	4.16
SCoT (Ours)	5000 / 500	42.78	0.07	42.11	2.14	32.28	-1.90	33.94	2.04

Ablation Study¶

A progressive ablation analyzes the contribution of each architectural component: - C1: Anchor-relative task vector state with uniform averaging; - C2: Similarity-guided layer-wise consolidation; - C3: Conflict-aware layer-wise projection; - C4: Depth-aware stabilization for compositional generalization.

Configuration (C1 / C2 / C3 / C4)	VQAv2 Std AP(↑)	VQAv2 Std AF(↓)	VQAv2 Comp AP(↑)	VQAv2 Comp AF(↓)	NExT-QA Std AP(↑)	NExT-QA Std AF(↓)	NExT-QA Comp AP(↑)	NExT-QA Comp AF(↓)
C1 alone	39.23	-0.35	38.51	-0.28	27.35	-1.79	28.44	-5.67
+ C2	42.62	1.42	41.08	2.99	32.15	1.79	34.00	3.33
+ C3	43.02	0.92	41.31	2.95	32.39	0.49	33.75	3.07
+ C4 (Full SCoT)	42.78	0.07	42.11	2.14	32.28	-1.90	33.94	2.04

Key Findings¶

Surpassing Multi-Task Joint Training: On VQAv2 standard testing, SCoT achieves an AP of 42.78%, exceeding the Joint training upper bound (41.32%). This reveals that sequential consolidation with geometric conflict removal effectively circumvents cross-task gradient competition that plagues simultaneous multi-task optimization.
Positive Backward Transfer (BWT): SCoT achieves an AF of -1.90% on NExT-QA, accompanied by positive BWT scores of +5.64 on VQAv2 and +6.97 on NExT-QA, demonstrating that learning downstream tasks constructively refines shared representations for previous tasks.
Distinct Roles of Components: C2 yields the single largest plasticity boost (+3.39% AP on VQAv2, +4.80% on NExT-QA); C3 significantly mitigates the accompanying forgetting; C4 trades a fraction of standard plasticity to dramatically elevate compositional generalization (Comp AP reaches 42.11% on VQAv2 and 33.94% on NExT-QA).

Highlights & Insights¶

Minimal-Norm Geometric Projection: Rather than applying broad parameter masking or uniform vector shrinkage, SCoT uses Gram-Schmidt style projection to surgically strip away only destructive negative components, maintaining optimal representation capacity.
Rethinking the Joint Training Ceiling: The finding that continual geometric consolidation can outperform joint training provides strong empirical evidence that structured sequential parameter merging can discover superior shared optima compared to entangled joint SGD.
Interface-Specific Stabilization: Discovering that compositional vulnerability resides primarily at the late-encoder and early-decoder junction allows selective capacity regulation without freezing or bottlenecking the entire model.

Limitations & Future Work¶

Linearity of the Tangent Space: SCoT relies on linear vector arithmetic relative to $\theta_0$. Across extremely prolonged task sequences with radical distribution drift, the tangent space approximation around the initial anchor may experience curvature degradation.
Unconstrained Intra-Task Optimization: Geometric conflict resolution occurs during post-hoc consolidation. In-task gradient descent may still locally overfit before consolidation takes place.
Empirical Sensitivity Thresholds: While $[\alpha_{\min}, \alpha_{\max}]$ is robust, dynamic adaptation based on online Hessian curvature estimation could offer an attractive parameter-free alternative.

vs VQACL (Zhang et al., CVPR 2023): VQACL relies on prototype rehearsal and hierarchical task grouping within activation space; SCoT operates directly in weight space, suppressing forgetting from 6.96% down to 0.07%.
vs QUAD (Marouf et al., ICCV 2025): QUAD employs question-only replay and attention distillation; SCoT avoids distillation overhead by executing explicit geometric projections in parameter space, achieving positive backward transfer (+5.64 BWT vs -4.76 BWT).
vs MagMax (Marczak et al., ECCV 2024) / TIES-Merging: MagMax and TIES merge weights based on element-wise max-magnitude or sign consensus, ignoring directional alignment. In replay-free settings (Tab. 6A), SCoT attains 31.44% AP on VQAv2, significantly outpacing MagMax (25.16%) and TIES (28.84%).

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Elegant reformulation of continual VQA as geometric parameter-space consolidation, combining layer-wise cosine alignment with minimal-norm projection.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous validation across VQAv2, NExT-QA, LLaVA-7B, class-incremental, and domain-incremental benchmarks with extensive layer-depth ablations.
Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear mathematical formulation, informative visualizations, and crisp logical structure.
Value: ⭐⭐⭐⭐⭐ Delivers rare positive backward transfer and surpasses joint training, setting a compelling new direction for multimodal continual learning.