Attention-Logit Steering to Compositional Generalization for Continual VQA

Conference: ECCV2026
Paper: Official paper page / PDF
Area: Multimodal VLM
Keywords: continual visual question answering, compositional generalization, question-evidence coupling, MoE-LoRA, attention steering

TL;DR

Q-STEER uses a shared controller to coordinate new expert capacity with corrections to where answer tokens attend, achieving 53.68 AP, 4.51 AF, and 51.00/51.48 Novel/Seen composition scores on continual VQA v2 without inference-time task labels or sample replay.

Background & Motivation

Continual visual question answering learns recognition, localization, counting, color, and other question types sequentially, while remaining responsible for earlier tasks. EWC and MAS constrain important parameters, replay methods revisit earlier data, and expert methods isolate updates. These approaches primarily protect capabilities against parameter changes, but do not necessarily preserve the binding between an object named in a question and the appropriate image region.

For example, a model may still recognize red and know how to answer color questions, yet shift its attention from the queried object to the background after subsequent training. The concept itself need not have disappeared: the question-evidence coupling has drifted. This can also disrupt new combinations of familiar skills and concepts. Conversely, forcing a new task to reproduce old attention is undesirable when a distribution shift genuinely requires different evidence.

Core idea: use attention drift as a diagnostic input rather than a distillation target, and let one question-conditioned controller coordinate old/new expert mixing with late-layer attention corrections, assigning plasticity to expansion and retention to evidence routing.

Method

Overall Architecture

The input is an image and a question; the output is an autoregressive answer. The multimodal backbone stays frozen, while LoRA experts, a shared controller, and low-rank attention-correction bases are learned. Before each subsequent task, a short probe decides whether candidate experts should become active. For each actual input, question-token representations and a prompt-only diagnostic forward pass supply context and drift; the controller then produces expert weights, a shared gate, and attention-correction coefficients.

Inference requires no task ID, but training still uses task boundaries for probing and consolidation. Probing in the diagram is a task-level action, whereas diagnosis and control are input-dependent. The old-task attention reference is a consolidated dataset-level statistic, not a paired teacher answer retrieved for each new image.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Image and question"] --> Drift["Question-conditioned<br/>drift diagnosis"]
    Drift --> Controller["Shared controller"]
    Probe["Probe-based expert expansion<br/>At subsequent task boundaries"] --> Route["Factorized old/new routing"]
    Controller --> Route
    Controller --> ICR["Low-rank attention steering"]
    Route --> ICR
    ICR --> Output["Generate answer<br/>Consolidate after the task"]

Key Designs

1. Question-conditioned drift diagnosis: detect evidence-routing changes without forcing old attention

Controller context pools only question-token hidden states, preventing teacher-forced ground-truth answers from leaking into routing. A separate prompt-only forward pass extracts attention from the answer-start query position to visual tokens at each controlled late layer, and compares it with a reference consolidated from completed tasks. Both distributions are smoothed and renormalized. The first task uses a reference initialized from the frozen backbone; thereafter the reference remains fixed within each task and is updated only after that task finishes.

The essential operation in Eq. (3) averages KL divergence over the controlled late layers:

\[ d(x)=\frac{1}{|\mathcal L_{\rm late}|}\sum_{l\in\mathcal L_{\rm late}}D_{\rm KL}\!\left(p_{\rm cur}^{l}(x)\,\|\,p_{\rm ref}^{l}\right). \]

This compares attention distributions, not answer probabilities, and supplies no attention-map supervision. Given question context and this scalar, the shared controller outputs layer-specific old/new routing logits, a shared gate \(\lambda(x)\) in \([0,1]\), and ICR low-rank and per-head coefficients. A large drift is not automatically classified as an error: answer cross-entropy teaches the controller how to use it, leaving room for legitimate changes in visual attention. The main text does not fully specify reference aggregation weights, smoothing constants, or controller dimensions; these should not be invented as established implementation choices.

2. Probe-based expert expansion: grow capacity according to candidate utilization

Each expandable layer preallocates 32 LoRA slots and initially activates 8. At a subsequent task, a 128-sample probe temporarily exposes candidate inactive slots. Candidate utilization is its routing weight averaged over tokens and then over task samples. A candidate is promoted if its utilization exceeds the layer's mean utilization minus \(\beta_{\rm thr}\) times its standard deviation; the default is \(\beta_{\rm thr}=0.5\). This is a utilization-based growth heuristic, not a rigorous capacity-insufficiency test derived from new-task error.

Expansion changes only the active mask, keeping parameter shapes fixed and avoiding runtime reconstruction of the expert bank. Inactive experts receive negative infinity before softmax and cannot be routed to, but their preallocated parameters still consume storage. Figure 4 reports at most 18 active slots in any controlled layer, leaving at least 14 unused. This shows that the evaluated stream does not exhaust the 32-slot bank, not that arbitrary long streams will remain sparse.

3. Factorized old/new routing: normalize each bank before shared-gate interpolation

Carried-over experts and newly activated experts undergo separate masked softmax operations, after which the shared gate mixes the two distributions. The corresponding weighted LoRA update is added to the frozen linear transform. The point is not merely another softmax: increasing new-expert logits no longer directly suppresses old-expert probabilities through the same normalization denominator.

\[ \widetilde g^{l}(x)=(1-\lambda(x))p_{\rm old}^{l}(x)+\lambda(x)p_{\rm new}^{l}(x). \]

The distributions are embedded into the full expert space on their respective slots. When no candidate is promoted, the new branch is bypassed. With \(\lambda\) held fixed, an old expert's weight has zero cross-derivative with respect to new-bank logits; the controller itself remains trainable, so this removes a local competition path rather than freezing all old routing. Forward computation uses a soft mixture, while backward updates are restricted to the top \(K=2\) routed experts. This must not be confused with evaluating only two experts in the forward pass. On subsequent tasks, consolidated experts are frozen and new experts, the controller, and ICR bases are updated, reducing direct overwriting of old knowledge.

4. Low-rank attention steering: correct evidence allocation before softmax at answer queries

ICR does not match old attention. It adds an input-dependent bias to each controlled late-layer attention head's logits. Queries and keys are projected into low-rank spaces, controller coefficients \(\alpha\) modulate their components, and the shared gate plus per-head coefficients \(\gamma\) scale the correction. Following Eqs. (10)-(11), with attention logits denoted by \(A_h^l\):

\[ \Delta A_h^l(x)=(Q_h^lU_{q,h}^l)\operatorname{diag}(\alpha^l(x))(K_h^lU_{k,h}^l)^\top, \qquad A_h^l\leftarrow A_h^l+s_{\rm icr}\lambda(x)\gamma_h^l(x)\Delta A_h^l(x). \]

The correction applies only at answer-query positions, not across all prompt-token attention. Changing logits before softmax lets the model reallocate relative evidence weights instead of merely correcting answers at the final output layer. Defaults are the last 6 layers, rank 8, and \(s_{\rm icr}=0.3\). Sharing \(\lambda\) between new-expert contribution and attention perturbation gives both adaptation paths a common amplitude control, but may tie together two needs that should sometimes vary independently.

A Worked Example

Consider "What color is the cup?" as a mechanism illustration, not an additional measured example from the paper. The model pools question tokens for context, then uses the image and question in a diagnostic pass to measure how answer-start visual attention differs from the historical reference. The ground-truth answer "red" is excluded from these diagnostic inputs.

If candidates were promoted before the current task, the controller allocates weights separately within old and new banks, and the same gate modulates new-expert contributions and the ICR bias. Low-rank steering acts at answer-generation positions, making attention to the cup rather than irrelevant background a learnable choice. Answer cross-entropy supplies training supervision; inference generates the answer directly. Neither a "color task" label nor replayed cup images are required. No particular drift value, gate value, or single-example correction success rate is established by this illustration.

Loss & Training

The objective is standard autoregressive cross-entropy over answer tokens, with no KL attention-matching term. The initial task trains initial experts, the controller, and ICR bases. Subsequent tasks probe first, freeze consolidated experts, train new experts and shared components, and then consolidate experts and the attention reference. Both LoRA and ICR ranks are 8, and attention softmax uses fp32. The cached main text does not fully specify learning rates, optimizer, training epochs, or exact backbone configuration, so it does not establish a complete reproduction recipe.

The shared gate also admits a limited theoretical interpretation. Let \(m_{\rm old}(x)\) be the old-path prediction margin, and let \(\Psi(x)\) collect layer-wise MoE and ICR perturbations weighted by local Lipschitz constants. The paper states:

\[ \|z(x)-z_{\rm old}(x)\|_\infty\leq\lambda(x)\Psi(x), \qquad m(x)\geq m_{\rm old}(x)-2\lambda(x)\Psi(x). \]

Only when \(\lambda(x)\Psi(x)<m_{\rm old}(x)/2\) does this bound guarantee an unchanged old prediction. It depends on local assumptions and unknown perturbation magnitudes, is not automatically enforced during training, and does not guarantee that the old prediction is correct. The full derivation is deferred to supplementary material absent from this cache.

Key Experimental Results

Main Results

VQA v2 follows the VQACL protocol with 10 sequential question types. Methods share a LLaVA-style initialization, task order, answer set, evaluator, and task-agnostic inference setting. AP averages final task performance; AF averages the gap between each earlier task's best historical and final performance, so lower is better. With \(a_{i,t}\) denoting performance on task \(t\) after training through task \(i\), Eq. (14) is:

\[ \mathrm{AP}=\frac{1}{T}\sum_{t=1}^{T}a_{T,t},\qquad \mathrm{AF}=\frac{1}{T-1}\sum_{t=1}^{T-1}\left(\max_{t\leq i\leq T}a_{i,t}-a_{T,t}\right). \]

The table combines AP/AF from the paper's Table 1 with average composition scores from Table 2. Novel denotes held-out skill-concept compositions and Seen denotes observed compositions; scores are on a percentage scale.

Method	AP ↑	AF ↓	Novel ↑	Seen ↑
ER	41.95	10.20	40.03	40.92
DER	41.16	11.28	39.14	39.86
VQACL	43.49	9.10	41.78	42.61
CL-MoE	42.63	10.56	41.02	42.09
QUAD	44.16	8.97	42.47	43.24
Q-STEER	53.68	4.51	51.00	51.48

Against QUAD, AP improves by 9.52 percentage points, AF decreases by 4.46 points, and Novel/Seen improve by 8.53/8.24 points. Three-seed results are \(53.62\pm0.28\) AP, \(4.55\pm0.15\) AF, \(50.96\pm0.22\) Novel, and \(51.43\pm0.24\) Seen. The main text does not explicitly define the statistical convention behind these error terms, so they should not be labeled confidence intervals. Gains do not cover every question type: causal performance is 23.61, below QUAD's 25.53.

On the heterogeneous CVIT stream, Q-STEER achieves 85.15/4.33 AP/AF versus SMoLoRA's 83.02/6.50. This protocol aggregates native dataset metrics, and Flickr30k uses a retrieval-style native score that can exceed 100. CVIT's 85.15 must not be compared with VQA v2's 53.68 as though they were the same accuracy measure.

Ablation Study

The following numbers come from Table 5. Removing factorized control removes both the old/new expert split and the shared gate, not just the normalization scheme.

Configuration	AP ↑	AF ↓	Novel ↑	Seen ↑
Full Q-STEER	53.68	4.51	51.00	51.48
Constant drift input	52.87	5.15	49.65	50.38
Cosine drift	53.14	4.91	50.12	50.82
Without factorized control	48.73	7.62	44.91	45.62
Without ICR	51.94	5.90	47.88	49.26
Without expansion	50.96	6.47	47.11	47.85
Without active mask	51.28	6.28	47.54	48.12
Without top-K	50.41	6.89	46.35	46.98

Key Findings

The factorized-control ablation is most damaging: AP falls by 4.95 points and AF rises by 3.11 points, but this benefit cannot be attributed independently to bank-specific softmax. Removing ICR raises AF by 1.39 points and lowers Novel by 3.12 points. Removing expansion lowers AP by 2.72 points. These findings support the intended retention/plasticity division, without implying that the components have mutually exclusive effects.

Table 6 uses Evidence-Mass to measure generated-answer attention assigned to the top 10% of visual patches most similar to question object or attribute nouns. Attention entropy measures dispersion over visual tokens. These are grounding proxies, not accuracy against annotated regions or causal evidence. The similarity model and full calculation details are deferred to the supplement and cannot be recovered from this cache.

ICR configuration	Evidence-Mass ↑	Attention entropy ↓	AP ↑	AF ↓
Without ICR	0.421	2.31	51.94	5.90
Scale 0.1	0.456	2.20	52.76	5.22
Scale 0.3	0.487	2.07	53.68	4.51
Scale 0.5	0.473	2.14	52.98	4.96

Stronger steering is not always better. The intervention window behaves similarly: controlling the last 6 layers achieves 53.68 AP, while extending to 8 layers reduces it to 52.94. Table 4 reports 45.6M add-on parameters for Q-STEER, with +6.5% FLOPs, +8.2% memory, +15.6% training overhead, and +6.1% inference overhead; CL-MoE reports 40.0M, +5.4%, +7.6%, +13.2%, and +5.0%, respectively. Costs include the diagnostic pass and, for training, probing. Low-rank parameter counts alone therefore do not establish runtime cost; the main text does not fully specify measurement hardware or normalization-baseline details.

Highlights & Insights

• Drift serves as information rather than a penalty, allowing legitimate attention changes on new tasks. The transferable idea is to use historical statistics to inform decisions instead of treating every departure from history as an error.
• Separate normalization and shared gating control expert competition, while a low-rank bias controls answer-evidence routing. The two paths target parameter adaptation and inference information flow, making the retention mechanism more specific than backbone freezing alone.
• Utilization-driven mask expansion preserves fixed parameter shapes as capacity grows. This makes growth compatible with a fixed graph, but does not eliminate the storage cost of preallocated slots.

Limitations & Future Work

• The authors acknowledge expert storage and the compute cost of diagnostic passes and per-task probes; long streams may also exhaust available slots. Expert compression, merging, or recycling are concrete next steps.
• The authors acknowledge that KL drift and Evidence-Mass are not causal explanations, and the theory provides only a sufficient condition. Occluding relevant regions or replacing irrelevant backgrounds could test whether better evidence use, rather than changing answer priors, explains the gains.
• The shared gate may be unsuitable when an input needs new experts but no attention correction. Separate yet coordinated routing and steering gates deserve comparison.
• From an experimental-design perspective, the compound ablation cannot isolate normalization from gating. The main text lacks a complete reproduction configuration, and the supplement is not cached. Independent ablations, measurement conditions, reference-construction details, and error-term definitions would strengthen the evidence.
• Task-agnostic inference does not imply boundary-free training. Reliability across backbones, unsegmented streams, or long-term open-world deployment remains unproven, and causal questions remain a clear weakness.

Related Work & Insights

• Versus QUAD: QUAD uses questions-only replay and attention distillation. Q-STEER stores no replay samples and has no attention-matching objective; drift instead informs routing and steering. These are distinct uses of historical information, not two forms of teacher-attention supervision.
• Versus CL-MoE: Both use modular experts. Q-STEER additionally controls competition through old/new grouping and a shared gate, and directly modifies late-layer attention logits. Higher performance comes with extra parameters and diagnostic computation, so quality and cost should be considered together.
• Versus VQACL / SMoLoRA: VQACL supplies the question-type continual-VQA evaluation context, while SMoLoRA supplies the heterogeneous instruction-stream protocol. Testing both streams examines adaptation and retention, but is not evidence of generalization across model families.

Rating

• Novelty: 4/5. Drift diagnosis, factorized expert control, and logit steering form a specific mechanism, although the constituent techniques have precedents.
• Experimental Thoroughness: 4/5. Main-task, composition, heterogeneous-stream, three-seed, ablation, and cost analyses are included; causal attribution and independent gate ablations remain incomplete.
• Writing Quality: 4/5. The paper clearly separates diagnostics from losses and proxies from explanations, but some reproduction details depend on uncached supplementary material.
• Value: 4/5. The method offers a useful perspective on preserving evidence routing in continual multimodal systems, subject to the cost of additional forward passes and expert storage.