VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuning¶

Conference: ECCV2026
arXiv: 2603.01195
Code: https://dmk041218.github.io/VisNec/
Area: Multimodal VLM
Keywords: Visual Necessity, Data Selection, Multimodal Instruction Tuning, Visual Redundancy, Cross-Modal Reasoning

TL;DR¶

VisNec quantifies the dependence of each training sample on visual inputs by comparing the cross-entropy loss difference of a multimodal model between "with-image vs. without-image" forward passes. This enables the selection of a high-value subset that truly requires cross-modal reasoning, recovering or even exceeding full-dataset training performance with only 15% of the LLaVA-665K data.

Background & Motivation¶

Multimodal instruction tuning is a critical step for large models to acquire visual dialog capabilities. However, current mainstream datasets (such as LLaVA-665K and Vision-Flan-186K) contain a large number of "pseudo-multimodal" samples. Some questions can be answered solely by language priors (e.g., "What color is grass?" \(\rightarrow\) green), where visual information contributes nothing. Worse still, the images in some samples contradict their labels, causing visual inputs to interfere with predictions. These two types of samples not only waste computing resources during training but also reinforce the model's reliance on textual shortcuts, weakening genuine cross-modal alignment. Existing data selection methods—whether based on gradient influence (ICONS), clustering coverage (COINCIDE), or text-side loss (IFD)—evaluate sample importance as a whole without explicitly distinguishing the independent contribution of the visual modality. Consequently, the selected subsets may still be dominated by samples that are "linguistically easy but visually useless," or even retain harmful mislabeled samples.

The Key Challenge is that the value of multimodal data cannot be measured by overall "difficulty." Instead, it should be determined by "whether prediction difficulty rises significantly after removing the image"—meaning the value of a sample is proportional to the marginal reduction in prediction uncertainty provided by the visual information. If the model can still answer correctly without the image, the sample has almost no value for cross-modal training; if it answers incorrectly without the image but correctly with it, this is a valuable sample that truly requires visual reasoning.

Core Idea: By calculating the difference in cross-entropy loss of a multimodal model between a "blind (text-only)" and a standard "visual" forward pass, the visual necessity score (VisNec) of each sample is quantified. This is combined with semantic clustering for stratified sampling to select a high-quality instruction-tuning subset that is both visually necessary and task-diverse.

Method¶

Overall Architecture¶

The data selection pipeline of VisNec consists of two main stages. In the first stage, two forward passes are performed for each sample—one masking the visual input (blind forward pass) and one normally receiving the complete multimodal input. The difference between their cross-entropy losses is calculated to obtain a scalar VisNec score, which intuitively reflects the marginal contribution of visual information to the answer. A positive score indicates that visual information is critical, a near-zero score suggests redundancy, and a negative score implies that the visual input is misleading. In the second stage, the instruction questions of all samples are semantically clustered using K-Means. Within each cluster, samples are ranked from highest to lowest based on their VisNec scores. After filtering out samples with a score \(\le 0\), the top-\(r\%\) (default 15%) is selected from the remaining samples to form the final fine-tuning subset. This preserves the visual necessity of each sample while ensuring the diversity of task types through clustering.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Training Sample<br/>(v, t, y)"] --> B["Blind Forward Pass<br/>Masking visual tokens"]
    A --> C["Standard Forward Pass<br/>Full multimodal input"]
    B --> D["L_blind<br/>Text-only loss"]
    C --> E["L_mm<br/>Multimodal loss"]
    D --> F["Calculate VisNec<br/>S = L_blind - L_mm"]
    E --> F
    F --> G{"S > 0 ?"}
    G -->|"S ≤ 0 (Redundant/Misleading)"| H["Discard"]
    G -->|"S > 0 (Visually Critical)"| I["Instruction Semantic Clustering<br/>K-Means (K=20)"]
    I --> J["Rank by VisNec within cluster<br/>Select top-15%"]
    J --> K["Final subset D_select"]
    K --> L["LoRA fine-tuning MLLM"]

Key Designs¶

1. Blind Forward Pass: A Controlled Experiment to Isolate Visual Contributions in MLLMs

The critical technical challenge in comparing "with-image vs. without-image" loss is how to cleanly construct a text-only inference condition without modifying the model parameters. VisNec solves this elegantly by replacing all image token positions with padding tokens, while setting the corresponding attention masks to zero. This ensures that visual information cannot propagate through attention computations to the representations of the response tokens. Both forward passes use identical model weights and batch configurations, with the only varying variable being the presence of visual input. To ensure the comparability of the two losses, image token positions are excluded from the loss calculation using ignore_index, resulting in both L_blind and L_mm being average cross-entropy losses computed solely over response tokens:

\[ \mathcal{L}_{\text{Blind}}(\theta;t,y)=-\sum_{j=1}^{L}\log P(y_{j}\mid y_{<j},t;\theta) \]

Although the blind input (only text instructions) is out of the standard training distribution for MLLMs, this does not affect the ranking. VisNec uses relative ranking within clusters rather than absolute thresholds, so any systematic bias cancels out among similar samples and does not affect the final selection.

2. VisNec Score: A Quantitative Scale for Visual Necessity

The VisNec score is defined elegantly as the difference between the blind loss and the multimodal loss:

\[ S_{\text{VisNec}}(v,t,y)=\mathcal{L}_{\text{Blind}}(\theta;t,y)-\mathcal{L}_{\text{MM}}(\theta;v,t,y) \]

This difference corresponds to the marginal reduction of prediction uncertainty by visual variables in the V-usable Information framework. Based on this score, samples are clearly categorized into three types: S > 0 indicates a significant visual gain, where the model's loss drops substantially after seeing the image, identifying samples that require true cross-modal reasoning; S ≈ 0 indicates visual redundancy, where the model can generate a similar prediction without looking at the image, representing samples that heavily rely on language shortcuts; S < 0 indicates visual misdirection or incorrect labels, where visual input degrades prediction performance. The key difference between VisNec and other methods (such as IFD) is that IFD only looks at the text-side loss, falling short of distinguishing between "samples that are difficult due to visual misdirection" and "samples that are difficult due to visual necessity." By explicitly isolating the independent contribution of vision through subtraction, VisNec precisely filters out visually misleading samples. Ablation studies demonstrate that selecting data using either text-only loss or multimodal loss alone achieves relative performance of 95.6% and 94.0%, respectively, whereas using their joint VisNec difference achieves 100.2%, proving the necessity of the differential signal.

3. Semantic-Aware Stratified Sampling: Selection Strategy for Preserving Task Diversity

If samples are selected based globally on the top-\(r\%\) of VisNec scores, a severe issue arises: different task types naturally have varying degrees of dependence on visual inputs. For instance, spatial reasoning samples typically exhibit much higher VisNec scores than OCR or commonsense QA samples. A direct global ranking would bias the selected subset toward certain tasks, causing a loss of diversity. VisNec solves this via a two-stage "cluster-then-select" strategy: first, it extracts the user question part (excluding system prompts) from each instruction and applies K-Means clustering (\(K=20\)) with sentence embeddings to group semantically similar questions automatically; second, within each cluster, it filters out redundant or misleading samples with VisNec scores \(\le 0\), and then selects the top-\(r\%\) based on the ranking of remaining scores. This ensures that the final subset retains the most visually dependent samples within each task type while maintaining proportional representation across all task categories. Ablation results show that removing the clustering layer (Top-VisNec scheme) drops the relative performance from 100.2% to 97.0%, confirming that stratified sampling is indispensable for maintaining task coverage.

Loss & Training¶

VisNec itself is a data selection framework and does not introduce new training objectives. The selected subset is fine-tuned using a standard LoRA configuration: a learning rate of \(2\times 10^{-4}\), training for 1 epoch, built upon the LLaVA-v1.5 Stage-1 pretraining checkpoint (feature alignment). Experiments show that 15% is the optimal selection ratio (5% recovers 99.1% of performance, 15% reaches 100.2%, and 20% slightly decreases to 99.2%), confirming the "less is more" phenomenon in multimodal instruction tuning.

Key Experimental Results¶

Main Results¶

Fine-tuning LLaVA-v1.5-7B with 15% data (98K samples) on LLaVA-665K:

Metric	Random	IFD	PreSel	XMAS	COINCIDE	CoIDO	VisNec(15%)	Full-Data(100%)
Rel.	94.2%	92.1%	97.7%	97.3%	95.8%	97.7%	100.2%	100%
VQAv2	75.3	74.0	76.5	75.1	76.1	75.8	78.0	79.1
LLaVA-Wild	58.8	62.3	65.6	62.4	64.9	66.7	69.8	67.9
MM-Vet	30.2	28.1	29.6	31.0	28.5	31.4	32.1	30.0

On Vision-Flan-186K (comprising 191 diverse tasks with fewer samples per task), VisNec achieves a relative performance of 115.8% with only 15% of the data (28K samples), significantly outperforming full-data training (100%) and random selection (96.7%). This indicates that a large quantity of samples in this dataset are not only redundant but even detrimental to training, highlighting VisNec's exceptionally effective filtering.

Cross-Architecture Transferability¶

Model	Full-Data	VisNec(15%)
Qwen2.5-VL-3B	100%	103.8%
Qwen2.5-VL-7B	100%	104.0%
Qwen2.5-VL-32B	100%	102.4%

VisNec scores are highly effective across different architectures (LLaVA vs. Qwen2.5-VL) and multiple parameter scales (3B/7B/32B), indicating that it captures the inherent visual necessity of the data itself rather than a bias toward a specific model.

Key Findings¶

Differential signal is key to success: Selecting data using either text-only loss or multimodal loss alone yields relative performances of 95.6% and 94.0%, respectively, which are far lower than the joint performance of 100.2%. Visually misleading samples exhibit low loss in standard multimodal forward passes because the model "learns" to be misled; this makes the blind forward pass a crucial comparative baseline to unmask them.
Clustering layer contributes 3.2% gain: Eliminating the clustering step (i.e., the Top-VisNec scheme) drops performance from 100.2% to 97.0%. However, this still outperforms all non-Top-VisNec baselines, suggesting that the VisNec score itself is a very strong signal and clustering acts as an "icing on the cake" to guarantee diversity.
High robustness to hyperparameters: Changing the number of clusters \(K\) from 10 to 30 results in a performance fluctuation of only 1.3%. The selection ratio recovers over 99% of full-data performance across the range of 5% to 20%.
High computational efficiency: Filtering 665K samples with VisNec requires only 12 GPU-hours (H100), which is significantly lower than Self-Filter (73.5) and COINCIDE (55.5), and does not rely on any external APIs.

Highlights & Insights¶

The differential comparison method elegantly simplifies the "quality assessment" problem: It compresses the data selection task from complex gradient tracing or multi-model ensembles into "performing two forward passes and calculating one difference score." The concept is clean and the implementation is lightweight, making it an excellent example of applying information-theoretic thinking in practice.
Natural extension of the V-usable Information framework: Extending from the textual domain to the multimodal domain, the method aligns the marginal loss reduction directly with the information gain of the visual variable, possessing a solid theoretical foundation.
Empirical proof of "less is more": Achieving 115.8% relative performance with 15% of the data on Vision-Flan-186K demonstrates that a large amount of noise and redundant samples in current datasets actually hinder training. This phenomenon is particularly prominent in scenarios with a large number of tasks and few samples per task, offering direct insights for future data construction strategies.
No external APIs, no multi-model ensembles, no gradient backpropagation: Compared to methods like PreSel and CoIDO that require expensive GPT-4 evaluations, VisNec is entirely self-contained and deployable on any basic GPU environment.

Limitations & Future Work¶

The quality of the VisNec score is bound by the capability of the MLLM itself: if the model is too weak, the difference between the blind and multimodal loss might not reliably reflect true visual necessity. Although the authors verified transferability in the range of 3B to 32B models, the boundary for smaller or larger models remains to be explored.
The setting of \(K=20\) clusters is used consistently across different datasets (LLaVA-665K has about 10 tasks, whereas Vision-Flan-186K contains 191 tasks). Although robust in experiments, datasets with extreme diversity (e.g., thousands of tasks) might require an adaptive choice of \(K\).
Is there a risk of category imbalance within the selected 15% subset? The clustering strategy ensures that "every category is represented" but cannot guarantee sample diversity within each category (e.g., a cluster might end up consisting purely of easy questions). See the original paper for further details.

vs. IFD (NAACL 2024): IFD uses the textual response prediction loss as a quality signal, which cannot differentiate between "samples that are difficult due to visual misdirection" and "samples that are difficult due to visual necessity." The differential formula of VisNec directly addresses this confounding factor.
vs. PreSel / CoIDO: These methods rely on external APIs like GPT-4 to evaluate image quality, which is expensive and presents privacy risks. VisNec is entirely self-contained.
vs. COINCIDE (EMNLP 2024): COINCIDE uses gradient clustering for selection but features high complexity (55.5 GPU-hours), whereas VisNec (12 GPU-hours) is about \(4.6\times\) faster and yields better performance.
vs. Classic active learning / uncertainty sampling methods (EL2N, TypiClust): These methods are suitable for supervised learning with labels but lack cross-modal consistency awareness when applied to multimodal instruction tuning.

Rating¶

Novelty: ⭐⭐⭐⭐½ Quantifying visual necessity using differential loss is conceptually novel, although the theoretical innovation under the V-usable Info framework is incremental. The engineering implementation is highly clever.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Covers 2 datasets \(\times\) 10 benchmarks \(\times\) 3 model scales, with a complete ablation analysis (loss components ablation, clustering ablation, parameter sensitivity) and no critical misses.
Writing Quality: ⭐⭐⭐⭐⭐ The logic is extremely clear. The design motivation and implementation details of the blind forward pass are thoroughly explained, and the case studies visually demonstrate the meaning of scores across the three categories.
Value: ⭐⭐⭐⭐⭐ Standardized performance is recovered/exceeded with only 15% data, with low computational cost (12 GPU-hours) and no external API dependencies, offering direct practical value for training large models.