How to Teach Large Multimodal Models New Skills¶
Conference: ECCV 2026
Paper: ECCV Paper
Area: Multimodal VLM
Keywords: continual learning, selective fine-tuning, catastrophic forgetting, output-distribution drift, counting bias
TL;DR¶
The paper attributes part of multimodal-model forgetting to task-induced response bias and shows that tuning only attention projections or MLP Gate&Up yields current-task gains of +24.9 / +30.5 percentage points on LLaVA-OneVision across five sequential tasks, while limiting held-out changes to -0.6 / -2.1 percentage points.
Background & Motivation¶
A multimodal model that can describe images and answer general questions may still struggle with fine-grained bird recognition, precise counting, or reading clocks. Fine-tuning on specialized data can teach these skills but degrade previously reliable visual question answering. Continual learning commonly protects prior knowledge through data replay, distillation, or network expansion, yet original pretraining data may be unavailable and additional modules increase training and deployment costs. The question here is whether selecting where parameters can change is enough to improve learning and retention without replaying earlier tasks.
The authors observe something more specific than the usual fine-tuning-induced forgetting: general performance lost at one stage can partially recover during subsequent training on another skill, even without revisiting old samples. A lower test score therefore need not imply that knowledge has been permanently erased. After counting adaptation, for example, a model may still recognize objects in an open-ended captioning image but recast its answer as numerical statements. The failure may involve response preferences rather than purely visual recognition. Conflating these effects risks applying expensive memory-preservation mechanisms to a problem that parameter selection could partly address.
Transformer components offer a testable starting point. Attention projections primarily adjust contextual associations and information mixing, whereas MLPs write activated features into the residual stream and thereby influence vocabulary probabilities. Rather than assuming these roles are completely independent, the authors compare component updates and examine their output changes with a counting-bias probe. Core Idea: preferentially update attention projections, or change MLP feature activation while freezing its Down write-back projection, to reduce specialized training's disruption of general output distributions.
Method¶
Overall Architecture¶
Inputs remain images and text instructions, and the vision encoder, multimodal projector, and autoregressive language model retain their original architecture. Each stage uses only the current skill's data, with target answers providing supervision. The intervention changes which parameters may be updated, rather than adding a new inference module.
The two recommended configurations are alternatives. SA Proj. updates attention projections throughout the language model; MLP Gate&Up updates the gating and up projections while freezing the down projection. After training, the model generates answers through its original path without a task router, additional adapters, or a historical-data buffer. The counting-bias probe in the diagram is analysis-only: it contributes no optimization signal and is not required during deployment.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
Input["Pretrained model<br/>Current-task images and instructions"] --> Select["Select trainable parameters"]
Select -->|Option one| Attention["Attention Projection Updates"]
Select -->|Option two| Features["Feature Activation Updates"]
Labels["Current-task reference answers"] -.->|Cross-entropy supervision| Train["Current-stage fine-tuning"]
Attention --> Train
Features --> Train
Train --> Model["Adapted model<br/>Standard autoregressive generation"]
Model -.->|Analysis only| Probe["Counting-Bias Probe"]
Captions["Non-counting caption images"] -.-> Probe
Key Designs¶
1. Attention Projection Updates: learn to use existing information without rewriting the entire decoder
SA Proj. trains \(W_Q,W_K,W_V,W_O\) in every layer while freezing the remaining components. Query and Key determine which context positions to read, Value determines which content to mix, and the output projection writes the mixed representation into the residual stream. For a model with rich visual and linguistic features, specialized skills may not require relearning every representation; more effective retrieval and combination of existing information can suffice. This restriction leaves substantial adaptation capacity while avoiding the strong output drift associated with updating the entire MLP.
SA Proj. should not be reduced to "routing only, with no write-back changes": it does include \(W_O\). The authors separately test a QKV-only variant that freezes \(W_O\) to distinguish these effects. That variant is more stable but reduces current-task gains from +24.9 to +14.9 percentage points, suggesting that freezing all write-back paths can overconstrain learning. The recommended configuration therefore follows from jointly measuring learning and forgetting, not from assuming that a module's name determines its behavior.
2. Feature Activation Updates: preserve the MLP output mapping while learning which features to activate
A gated MLP uses the Gate branch and SiLU to produce a gating signal, while the Up branch produces expanded features. Their elementwise product passes through the Down projection into the model-width residual stream. Full-MLP tuning changes both what is detected and what is written back when it is detected, making it easier to pull the model's overall output preferences toward the current task's answer format. Gate&Up trains only the first two projections and fixes the existing Down mapping, acquiring skills primarily by changing feature activation.
Freezing Down does not make MLP outputs constant, because its incoming activations still change. The constraint fixes the write-back mapping rather than preventing new information from entering the residual stream. The critical comparison is that full MLP and Gate&Up learn similar amounts on the current task, but full MLP damages general benchmarks and previously learned tasks much more. This contrast is more informative than trainable-parameter counts alone and motivates Gate&Up when stronger adaptation is desirable but broad degradation is unacceptable.
3. Counting-Bias Probe: test whether the model increasingly prefers numbers on non-counting inputs
The probe uses a fixed set of LCS-558K captioning examples and a fixed numeric-token set containing digits and common spelled numerals. For each image, the current model first generates a caption using greedy decoding. At each generated position, the probe takes the largest next-token probability among numeric tokens, averages it over the generated sequence, and then averages across images to obtain numeric-token bias (NTB). It measures the probability of the most likely individual numeric token, not the summed probability of all numeric tokens or the number of digits in the final text.
This separates learning to count from developing a general preference for numeric answers. PixmoCount accuracy measures the former; captioning images that do not request counts reveal the latter. During LLM and full-MLP tuning, numeric bias rises while held-out performance falls, whereas SA Proj. remains near the baseline bias. Because the probe follows the model's own generated prefixes, it reflects both distribution changes and changes in generation trajectories. It is useful diagnostic evidence, but cannot establish by itself that all internal knowledge remains intact.
A Worked Example¶
Figure 4 compares models after 1K training steps on PixmoCount. Full MLP and SA Proj. both correctly answer that a counting image contains 13 people, showing that both acquire the target skill. Success on that skill alone does not establish that open-ended answering remains reliable.
For an image containing a presentation slide and augmented-reality examples, the full-MLP model responds to a general description request by saying that there are 2 photos in the photo. The SA Proj. model instead continues to explain the presentation and scene. The distinction is not that the latter counts more objects, but that it better preserves the ability to choose an answer format appropriate to the question. The bias probe makes this behavior measurable over training.
Loss & Training¶
The basic recipe uses teacher-forced next-token cross-entropy on current-task data only, without mixing in earlier-task samples. Selective fine-tuning requires no auxiliary loss, additional parameters, or stage-specific blending coefficient, and each new skill starts from the preceding checkpoint. Embeddings, the language-model output head, and layer-normalization parameters remain frozen in the selective configurations discussed in the paper.
Learning without Forgetting (LwF) is an additional baseline. It freezes the previous-stage model as a teacher, constrains student outputs with KL divergence on current-task examples, and combines distillation with the task loss using a weighting coefficient. Distillation samples token positions uniformly, using at most 1000 positions to cap computation and memory. This demonstrates that output changes can also be constrained without storing old data, but LwF is a separate comparison mechanism, not a default component of SA Proj. or Gate&Up.
The default curriculum is CUB200 bird classification, PixmoCount counting, PathVQA medical question answering, TextVQA text reading, and TimeClock time reading. It contains 107,910 training examples, including 34,602 from TextVQA. Table 1 averages three five-task orders, whereas the additional backbones in Table 2 use the default order. These aggregation conditions should not be conflated into a claim that every model was tested on three orders.
The local cache contains only the main paper and references. Detailed data splits, learning rates, batch sizes, and further implementation settings are deferred to supplementary material, so unverified hyperparameters are not supplied here. Several extracted equations are visibly damaged; the descriptions above explain the operations and objectives without presenting speculative repairs as the authors' exact formulas.
Key Experimental Results¶
Main Results¶
All values below are LLaVA-OneVision results from the paper's Table 1. Unless explicitly stated otherwise, they are percentage-point changes relative to the base model, not final accuracies. The baseline mean target score is 43.9 and the mean held-out score is 76.4.
Target Learning averages each current task's improvement over the base model immediately after training on that task. Target Forgetting compares earlier tasks immediately after their training with their performance at the end of the sequence; negative values denote decline. Target Overall is the final average change across all target tasks relative to the base model. Held-out Change is the final mean change across eight held-out benchmarks.
The held-out suite comprises AI2D, ChartQA, DocVQA, InfoVQA, RealWorldQA, SeedBench, ScienceQA, and MMStar, none used to train these five skills. DocVQA and InfoVQA use ANLS, an answer score based on normalized edit similarity; the remaining benchmarks follow the paper's accuracy-based aggregation. Scores are averaged on a common scale, which is not equivalent to pooling all test questions into a single accuracy calculation.
| Updated Components | Target Learning | Target Forgetting | Target Overall | Held-out Change |
|---|---|---|---|---|
| Full: entire model | +29.9 | -25.9 | +9.2 | -27.4 |
| LLM: full language model | +31.8 | -23.5 | +13.0 | -23.3 |
| SA Proj. | +24.9 | -2.3 | +23.1 | -0.6 |
| Full MLP | +31.1 | -19.5 | +15.5 | -15.7 |
| MLP Gate&Up | +30.5 | -4.2 | +27.1 | -2.1 |
The important outcome is the net benefit over the whole sequence. Full-LLM tuning learns more at individual stages, yet its Target Overall gain is only +13.0, below Gate&Up's +27.1. Evaluating only the task that has just been trained can therefore overstate the value of unrestricted fine-tuning for continual learning.
Ablation Study¶
The following parameter-freezing comparisons also come from Table 1, under the same three-order averaging protocol. They test whether constraining write-back is always beneficial, rather than representing an additional independent experiment.
| Parameter Comparison | Target Learning | Held-out Change | Analysis |
|---|---|---|---|
| SA Proj., train QKVO | +24.9 | -0.6 | Retains adaptation in the attention output projection |
| SA Proj., train QKV only | +14.9 | +0.2 | More stable, but learns 10.0 percentage points less |
| Full MLP, train Gate, Up, Down | +31.1 | -15.7 | Strong current-task gains with substantial general degradation |
| MLP Gate&Up, freeze Down | +30.5 | -2.1 | Learns only 0.6 less while reducing the held-out drop by 13.6 percentage points |
Key Findings¶
- Cross-model results do not guarantee zero forgetting. In Table 2, SA Proj. still changes held-out performance by -7.7 on LLaVA-NeXT (LLaMA-3 8B), versus +0.6 on Qwen2.5-VL 7B. Absolute stability depends on the backbone.
- Output bias provides mechanistic evidence. Section 5.2 reports a Spearman correlation of 0.84 between numeric bias and forgetting, and describes partial recovery through subsequent specialized fine-tuning. Correlation and recoverability support the interpretation but do not establish causality across all skills.
- Simple parameter selection need not be slower than low-rank adaptation. Section 5.4 reports throughput of 1.46, 1.27, and 0.44 samples/sec for SA Proj., LoRA, and MoE on 4 H100 GPUs. These are configuration-specific measurements, not universal comparisons across batch sizes, implementations, or hardware.
Highlights & Insights¶
- The paper distinguishes biased answer styles from lost knowledge. Comparing counting and captioning behaviors helps identify what needs intervention more clearly than an aggregate score drop alone.
- Freezing Down constrains feature write-back, not merely parameter count. Learning gains close to full-MLP tuning make this interpretation more persuasive.
- Current-task learning, previous-skill retention, and general-capability retention are measured separately. They can rank methods differently, so adaptation settings should follow the intended use.
Limitations & Future Work¶
- The authors explicitly note that the evaluation excludes massive or commercial multimodal models and does not establish effectiveness at billions-of-samples training scale.
- This note's assessment: NTB targets numeric outputs and cannot directly diagnose bias toward medical terminology, category names, or other answer formats. Broader controlled tasks and intervention experiments are needed.
- This note's assessment: a five-stage curriculum and a small set of backbones establish useful trends, not long-stream stability. The remaining degradation on LLaVA-NeXT also shows why the recommended settings still require evaluation.
- Reproducibility boundary: complete splits and training hyperparameters are not in the main text, and code is only announced as forthcoming. Fuller verification requires the supplementary material and actual implementation.
Related Work & Insights¶
- vs LwF: LwF constrains the current model using a previous-stage teacher's outputs; the recommended recipes directly restrict the trainable parameter set. Both seek to limit output drift, but selective fine-tuning does not require teacher forward passes by default.
- vs LoRA / MoE: The comparisons apply low-rank adaptation or expert expansion to MLP layers. Selective fine-tuning retains the original architecture without additional modules. The findings apply to these configurations, not every possible LoRA or MoE design.
- vs WiSE-FT: Weight interpolation can also preserve general performance, but the paper selects blending coefficients by task. Selective fine-tuning provides a more direct default update scope.
- Transferable lesson: For narrow-domain VLM adaptation, compare SA Proj. with Gate&Up and inspect out-of-task open-ended answers for response bias before deciding whether replay or distillation is necessary.
Rating¶
- Novelty: 4/5. The main contribution is a mechanistic view of recoverable forgetting and evidence-based parameter selection, not a new architecture.
- Experimental Thoroughness: 4/5. Five skills, eight held-out benchmarks, three backbones, and several baseline families provide substantial coverage, while large-scale and long-sequence tests remain absent.
- Writing Quality: 4/5. Metrics and central comparisons are clear, although complete reproduction depends on supplementary material.
- Value: 4/5. The recipes offer directly comparable starting points for multimodal continual adaptation with little architectural modification.