title: "Interference-Aware Continual Vision–Language Learning via Instance-Level Expert Routing" description: >- ECCV2026 paper note: IAR quantifies cross-task update conflict via Fisher-whitened projection energy onto historical adapter subspaces, routing samples to least-interfering experts with subspace packing and residual rank growth. tags: - ECCV2026 - Continual Learning - Vision-Language Models - LoRA - Mixture of Experts - Parameter-Efficient Fine-Tuning date: 2026-09-19 content_hash: 6e18b2dfc46bf468

Interference-Aware Continual Vision–Language Learning via Instance-Level Expert Routing¶

Conference: ECCV 2026
Paper: ECCV Paper
Area: Model Compression
Keywords: continual learning, vision-language models, LoRA, mixture of experts, interference-aware routing

TL;DR¶

IAR addresses catastrophic forgetting in continual vision-language learning by turning cross-task parameter conflict into an instance-level signal via Fisher-whitened projection energy, routing inputs to least-interfering experts with subspace packing and expanding rank along residual principal directions.

Background & Motivation¶

Continual adaptation of vision-language systems under non-stationary real-world streams remains difficult because data distributions, visual concepts, and instruction targets drift over time, while privacy and memory constraints rule out full rehearsal buffers. Full model fine-tuning or naive adapter training overwrites previously consolidated representations, inducing catastrophic forgetting. While parameter-efficient fine-tuning (PEFT) and mixture-of-experts (MoE) architectures provide scalable, modular alternatives, prior multimodal continual learners predominantly rely on global regularization, heuristic input-distribution gating, or pre-scheduled capacity milestones.

The fundamental limitation of existing routing and modular adaptation strategies is the absence of a metric-consistent, instance-level estimation of update interference. When incoming samples force weight updates that overlap with directions encoding past competencies, shared adapters or uncoordinated static experts compete within identical low-dimensional parameter subspaces, eroding historical abilities. Furthermore, heuristic input-similarity gates cannot assess whether a candidate parameter adjustment will geometrically collide with earlier updates, and static rank allocations fail to address instances where current expert capacity is genuinely exhausted.

The paper approaches continual adaptation by converting parameter-space conflict from an unmonitored failure mode into an explicit, sample-wise steering signal through a measure–select–grow loop. Core idea: by evaluating a candidate update's projection energy onto historical expert subspaces under a diagonal Fisher metric, IAR routes each instance to the least interfering experts, enforces principal-angle subspace packing to maintain inter-expert separation, and selectively expands LoRA rank along residual directions when current capacity is insufficient.

Method¶

Overall Architecture¶

IAR maintains a frozen vision-language backbone augmented with a bank of LoRA adapter experts across designated layers. For each incoming training sample, a probe gradient is first extracted from a backbone-only forward-backward pass and whitened under an exponential moving average (EMA) diagonal Fisher information matrix. The framework computes the metric-consistent projection energy of this whitened gradient onto each expert's historical update subspace to quantify potential destructive interference. A capacity-balanced router assigns the sample to the Top-\(K_r\) least interfering experts. During parameter updates, the experts optimize the task loss alongside a subspace packing regularizer that pushes inter-expert principal angles toward orthogonality. Whenever incoming gradient energy lies substantially outside the union of current expert subspaces, LoRA rank is incrementally expanded along the leading singular vector of the unwhitened residual. At inference time, a lightweight router distilled from frozen backbone features produces sparse mixture weights without requiring test-time gradients.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Sample (x, y) & Frozen Backbone"] --> B["Metric-Consistent Whitening & Projection Energy<br/>Backbone-only probe gradient + diagonal Fisher whitening"]
    B --> C["Min-Interference Routing & Capacity Control<br/>Select Top-Kr lowest energy experts + EMA utilization penalty"]
    C --> D["Sparse Expert Aggregation & Forward Pass<br/>Gated linear combination of LoRA adapter updates"]
    D --> E["Subspace Packing Regularization<br/>Penalize cross-coherence to maximize principal angles"]
    D --> F["Residual-Guided Dynamic Rank Growth<br/>Monitor residual energy ratio; expand rank via SVD at peak layer"]
    E --> G["Updated Expert Bank & Distilled Router"]
    F --> G

Key Designs¶

1. Metric-Consistent Whitening & Projection Energy: quantifying parameter conflict via geometry Estimating interference directly on parameter-adapted models risks mutual dependence between gating choices and model parameters, while standard Euclidean gradient norms fail to reflect functional predictive changes. IAR isolates a clean probe direction \(g_\ell(x, y) = \nabla_{\text{vec}(W_\ell)} \mathcal{L}(f_\theta(x), y)\) through a backbone-only forward pass with frozen weights \(\theta\). To align update overlap with local predictive shifts approximated by the Fisher quadratic form \(\delta w^\top G \delta w\), an EMA diagonal Fisher matrix \(G_\ell = \text{diag}(\mathbf{f}_\ell)\) provides whitened gradients \(\tilde{g}_\ell = L_\ell g_\ell\) with \(L_\ell = \text{diag}(\sqrt{\mathbf{f}_\ell + \epsilon})\). For each expert \(e\), its vectorized rank-1 LoRA components span an update subspace \(U_{\ell, e}\), tracked via an EMA basis \(\bar{U}_{\ell, e}\) and whitened into \(\tilde{U}_{\ell, e}^E = L_\ell \text{sg}[\bar{U}_{\ell, e}]\). The interference energy measures the squared norm of the regularized projection of \(\tilde{g}_\ell\) onto span(\(\tilde{U}_{\ell, e}^E\)):

\[\mathcal{I}_e(x, y) = \sum_{\ell \in \mathcal{S}} \tilde{g}_\ell(x, y)^\top \tilde{U}_{\ell, e}^E \left( (\tilde{U}_{\ell, e}^E)^\top \tilde{U}_{\ell, e}^E + \varepsilon_p I \right)^{-1} (\tilde{U}_{\ell, e}^E)^\top \tilde{g}_\ell(x, y)\]

A high projection energy indicates that adapting expert \(e\) would alter directions heavily relied upon by that expert's prior training, whereas near-zero energy denotes geometric orthogonality and minimal functional interference.

2. Min-Interference Routing & Capacity Control: balancing collision avoidance with expert load Routing purely based on minimal interference can trigger expert collapse, where a small subset of experts absorbs most tokens while others starve. To ensure balanced specialization, IAR combines the instance-level interference score with an exponential utilization penalty. Tracking the EMA utilization \(u_e\) and mean usage \(\bar{u}\) across all \(E\) experts yields the penalty \(\rho_e = (u_e / \bar{u})^{\gamma_c}\). The aggregate routing score is:

\[s_e(x, y) = \mathcal{I}_e(x, y) + \lambda_C \rho_e\]

The model selects the Top-\(K_r\) experts (\(K_r=2\) by default) possessing the lowest combined scores and evaluates soft gating coefficients \(\alpha_e(x, y)\) via temperature-scaled Softmax over the selected indices. Discrete selection utilizes a straight-through estimator, decoupling backpropagation into chosen experts from non-differentiable index masking.

3. Subspace Packing Regularization: separating expert subspaces via principal angles Over long sequential task streams, independently updated expert subspaces can drift and become co-linear, eroding routing selectivity and compounding interference. Inspired by Grassmannian subspace packing, IAR penalizes mutual coherence across all expert pairs. Gradients flow directly into current LoRA parameters by defining \(\tilde{U}_{\ell, e}^P = \text{sg}[L_\ell] U_{\ell, e}\) and extracting orthonormal bases via QR decomposition \(\widehat{U}_{\ell, e} = \text{qf}(\tilde{U}_{\ell, e}^P)\). The packing objective minimizes pairwise cross-correlation:

\[\mathcal{R}_{\text{pack}} = \sum_{\ell \in \mathcal{S}} \sum_{1 \le i < j \le E} \left\| \widehat{U}_{\ell, i}^\top \widehat{U}_{\ell, j} \right\|_F^2\]

Minimizing this penalty widens the principal angles \(\theta_{i, j}^{(1)}\) toward \(\pi/2\) (orthogonality), actively maintaining geometric separation between specialized modules across sequential adaptation phases.

4. Residual-Guided Dynamic Rank Growth: targeted capacity expansion for under-covered representations When a distribution shift introduces entirely unfamiliar concepts, forcing candidate updates into existing low-rank subspaces results in underfitting or elevated interference. IAR constructs the union projector \(\tilde{P}_{\ell, \cup}\) from all experts' whitened bases \(\tilde{U}_{\ell, \cup}^E\) and monitors the whitened residual vector \(\tilde{r}_\ell = (I - \tilde{P}_{\ell, \cup}) \tilde{g}_\ell\). The global residual energy ratio and maximal residual layer are computed as:

\[\eta(x, y) = \frac{\sum_{\ell \in \mathcal{S}} \|\tilde{r}_\ell\|_2^2}{\sum_{\ell \in \mathcal{S}} \|\tilde{g}_\ell\|_2^2}, \qquad \ell^\star = \arg\max_{\ell \in \mathcal{S}} \|\tilde{r}_\ell\|_2^2\]

Samples with \(\eta(x, y) > \tau\) (\(\tau=0.2\)) are queued for rank expansion. At batch boundaries, rank is expanded by \(+1\) strictly at layer \(\ell^\star\) for the optimal expert \(e^\star = \arg\min_e s_e(x, y)\). The whitened residual is unwhitened back into parameter space \(r_{\ell^\star} = L_{\ell^\star}^{-1} \tilde{r}_{\ell^\star}\), reshaped into matrix form \(R_{\ell^\star} = \text{mat}(r_{\ell^\star})\), and factored via leading singular vectors \((a, b) = \text{topSVD}_1(R_{\ell^\star})\) to append rank-1 columns and rows to \(A_{\ell^\star, e^\star}\) and \(B_{\ell^\star, e^\star}\) (bounded by \(r_{\max}\)). This injects capacity exclusively where the parameter bottleneck resides.

Loss & Training¶

The overall training objective combines task loss, subspace packing, and load regularization:

\[\mathcal{J} = \mathbb{E}_{(x, y)} \left[ \mathcal{L}(f_{\theta, \alpha}(x), y) + \mu \mathcal{R}_{\text{pack}} + \nu \mathcal{R}_{\text{load}} \right]\]

To eliminate gradient probing during inference, a feed-forward router \(r_\psi(\phi(x))\) is distilled via Kullback-Leibler divergence to match the oracle gate distributions \(\alpha(x, y)\) using frozen backbone features \(\phi(x)\). At test time, \(r_\psi\) directly provides Top-\(K_r\) sparse blending weights for the LoRA adapters, adding negligible latency.

Key Experimental Results¶

Main Results¶

IAR is evaluated on multi-domain incremental classification (11-domain MTIL and 10-domain task-agnostic X-TAIL, 16-shot), structured concept matching (ConStruct-VL), generative VQA (VQACL on VQA v2 and NExT-QA with LLaVA-1.5-7B), and 5-domain continual retrieval (CVLR) using CLIP ViT-B/16. All baselines operate under frozen backbones with matched trainable parameter budgets (0.8%–1.2%).

Table 1: Cross-domain classification and structured concept matching performance

Method	MTIL Avg (%)	MTIL Last (%)	MTIL Transfer (%)	X-TAIL Avg (%)	X-TAIL Last (%)	X-TAIL Transfer (%)	ConStruct-VL Acc (%)
LwF (TPAMI'17)	64.3	62.2	70.8	61.5	58.4	74.2	68.7
ZSCL (ICCV'23)	69.8	66.5	74.2	66.0	63.1	77.0	73.7
Mod-X (ICML'23)	68.9	65.0	72.8	65.3	62.0	75.4	74.1
C-CLIP (ICLR'25)	71.1	68.0	75.0	68.1	66.2	80.0	75.8
ZAF (NeurIPS'24)	70.3	67.2	76.1	67.6	65.7	80.4	74.6
DDAS (CVPR'24)	71.0	68.5	74.0	67.2	65.0	78.6	75.0
RAIL (NeurIPS'24)	70.9	68.8	77.4	67.4	65.9	79.2	74.9
DIKI (ECCV'24)	71.5	69.2	76.0	68.3	66.4	79.6	76.1
GIFT (CVPR'25)	71.6	69.4	77.2	68.6	66.7	80.7	76.5
LADA (ICML'25)	71.9	69.6	76.9	69.0	67.1	80.9	76.6
IAR (Ours)	73.0	70.7	78.1	69.8	68.0	81.2	77.3
Gain over Best Baseline	+1.1	+1.1	+0.7	+0.8	+0.9	+0.3	+0.7

Table 2: Generative VQA (VQACL) and continual cross-modal retrieval (CVLR)

Method	VQA v2 AP (%) ↑	VQA v2 Forget (%) ↓	NExT-QA AP (%) ↑	NExT-QA Forget (%) ↓	CVLR R@1 (%) ↑	CVLR R@5 (%) ↑	CVLR R@10 (%) ↑	CVLR Last R@1 (%) ↑
Vanilla	21.3	14.4	18.5	8.7	19.2	26.6	35.1	17.8
ER (NeurIPS'19)	47.8	6.2	31.0	3.4	29.6	57.4	66.8	28.5
DER (NeurIPS'20)	48.6	5.9	31.8	3.1	30.3	58.0	67.2	29.1
InfLoRA (CVPR'24)	50.4	4.5	33.4	2.7	30.9	58.8	67.9	30.1
CL-MoE (CVPR'25)	51.3	2.9	33.5	2.6	31.4	59.4	68.6	30.3
SMoLoRA (arXiv'24)	50.9	2.5	33.4	2.5	—	—	—	—
C-CLIP (ICLR'25)	50.7	3.4	33.0	2.7	32.5	61.2	70.8	31.1
IAR (Ours)	52.4	2.3	34.2	2.4	33.7	62.8	71.9	32.6
Gain over Best Baseline	+1.1	-0.2	+0.7	-0.1	+1.2	+1.6	+1.1	+1.5

Ablation Study¶

The single-factor ablation isolates each design choice across the evaluation suites.

Table 3: Single-factor ablation experiments for IAR

Configuration	MTIL Avg ↑	X-TAIL Last ↑	ConStruct Acc ↑	VQA v2 AP ↑	VQA v2 Forget ↓	NExT-QA AP ↑	CVLR R@1 ↑	CVLR Last R@1 ↑
Full IAR Model	73.0	68.0	77.3	52.4	2.3	34.2	33.7	32.6
❶ w/o Fisher whitening (Euclidean metric)	72.3 (-0.7)	67.4 (-0.6)	76.7 (-0.6)	51.8 (-0.6)	2.6 (+0.3)	33.7 (-0.5)	33.1 (-0.6)	32.0 (-0.6)
❷ w/o router (uniform adapter mixing)	71.5 (-1.5)	66.6 (-1.4)	75.8 (-1.5)	51.2 (-1.2)	3.0 (+0.7)	33.0 (-1.2)	32.3 (-1.4)	31.1 (-1.5)
❸ w/o subspace packing (\(\mu=0\))	72.6 (-0.4)	67.3 (-0.7)	76.5 (-0.8)	52.0 (-0.4)	2.5 (+0.2)	33.9 (-0.3)	33.3 (-0.4)	32.0 (-0.6)
❹ Fixed rank \(r=8\) (no rank growth)	72.6 (-0.4)	67.7 (-0.3)	76.9 (-0.4)	51.8 (-0.6)	2.6 (+0.3)	33.7 (-0.5)	33.4 (-0.3)	32.3 (-0.3)
❺ No rank growth (fixed \(r=4\))	72.3 (-0.7)	67.4 (-0.6)	76.6 (-0.7)	51.4 (-1.0)	2.7 (+0.4)	33.4 (-0.8)	32.9 (-0.8)	31.9 (-0.7)
❻ w/o capacity penalty (\(\lambda_C=\nu=0\))	72.5 (-0.5)	67.1 (-0.9)	76.8 (-0.5)	52.0 (-0.4)	2.6 (+0.3)	33.8 (-0.4)	33.0 (-0.7)	31.8 (-0.8)
❼ Top-\(K_r=1\) (hard gating)	72.7 (-0.3)	67.5 (-0.5)	76.9 (-0.4)	52.1 (-0.3)	2.4 (+0.1)	33.9 (-0.3)	33.4 (-0.3)	32.3 (-0.3)
❽ w/o EMA basis (instantaneous \(U\))	72.4 (-0.6)	67.2 (-0.8)	76.7 (-0.6)	51.9 (-0.5)	2.6 (+0.3)	33.6 (-0.6)	33.1 (-0.6)	32.0 (-0.6)

Key Findings¶

Impact of Router Selection: Removing the router (variant ❷) causes the largest performance drop across benchmarks (-1.5% MTIL Avg, +0.7% VQA v2 forgetting), proving that uncoordinated adapter updates severely degrade retention.
Metric Whitening Fidelity: Replacing Fisher whitening with Euclidean projection (variant ❶) degrades Last R@1 by 0.6% on CVLR and increases forgetting on VQA, demonstrating that Fisher information is necessary to calibrate parameter-space updates against predictive function drift.
Correlation with Downstream Forgetting: Diagnostic analysis reveals that instance-level interference energy \(\mathbb{E}[\mathcal{I}]\) strongly correlates with empirical degradation on prior tasks (Pearson \(r=0.622\), Spearman \(\rho=0.769, N=4259\)). In the high-energy regime, variance expands, supporting sample-wise conservative routing.
Hierarchical Rank Growth: Rank expansion events are concentrated in mid-to-high transformer layers rather than distributed uniformly across the model. These layers correspond to semantic alignment and complex multimodal reasoning, yielding the highest marginal gains in \(\Delta\text{AP}\) and \(\Delta\text{Last}\).

Highlights & Insights¶

Decoupling Probe Signals from Adaptive Modules: Utilizing a backbone-only forward pass to compute probe gradients eliminates circular dependencies between gating outputs and adapter parameter optimization.
Grassmannian Geometry in Modular Continual Learning: Formulating subspace packing via Frobenius norms of cross-correlations pushes pairwise principal angles toward \(\pi/2\), providing a structural mechanism to prevent expert collapse.
Selective Residual-Guided Plasticity: Instead of expanding capacity uniformly across all parameters, monitoring the projection residual ratio enables surgical rank additions strictly at saturated network layers.

Limitations & Future Work¶

Training Computational Overhead: Computing probe gradients via an initial backbone forward-backward pass increases per-step training FLOPs compared to conventional fine-tuning, posing throughput bottlenecks for large-scale streaming regimes.
Distilled Router OOD Sensitivity: The inference-time feed-forward router is trained via distillation on in-distribution features, which may suffer confidence degradation when encountering out-of-distribution shifts during unconstrained test streams.
Future Directions: Combining interference-aware routing with episodic rehearsal strategies, extending the paradigm to audio-visual foundation models, and adapting the method to resource-constrained edge computing environments.

vs InfLoRA (CVPR'24): InfLoRA strictly constrains task updates to the null space of prior activations, which can exhaust parameter plasticity over long sequences. IAR uses soft metric-consistent routing and dynamic rank expansion to maintain plasticity.
vs CL-MoE (CVPR'25) / DDAS (CVPR'24): Existing multimodal MoE continual learners rely on input-clustering or momentum-based heuristic gating without directly measuring parameter-level interference. IAR establishes a principled connection between projection energy and forgetting risk.
vs C-CLIP (ICLR'25): C-CLIP preserves cross-modal topological geometry to protect zero-shot performance, whereas IAR provides instance-level modular routing with dynamic capacity expansion across diverse vision-language objectives.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Formulates instance-level interference via Fisher-whitened projection energy, effectively bridging representation geometry, modular routing, and dynamic LoRA rank growth.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation spanning classification, concept matching, generative VQA, and cross-modal retrieval, supported by rigorous geometric and statistical analyses.
Writing Quality: ⭐⭐⭐⭐⭐ Mathematical formulation is elegant, figures are informative, and empirical claims are substantiated.
Value: ⭐⭐⭐⭐⭐ Offers a robust, parameter-efficient paradigm for continual adaptation in non-stationary multimodal environments under realistic memory constraints.