AutoV: Loss-Oriented Ranking for Visual Prompt Retrieval in LVLMs

Conference: ECCV2026
Paper: Official page · PDF
Code: https://github.com/Gumpest/AutoV
Area: Multimodal VLM
Keywords: Visual prompt retrieval, query awareness, pairwise ranking, loss supervision, cross-model transfer

TL;DR

Instead of using one visual prompt for every question, AutoV labels candidate preferences with a frozen LVLM's reference-answer losses and trains a lightweight instance-level ranker, raising LLaVA-OneVision 7B's VizWiz score from 58.2 to 68.4 while allowing the selection strategy to transfer to other models.

Background & Motivation

Visual prompting adds guidance directly to an input image: a red circle highlights an object, a blur mask suppresses irrelevant regions, or a text-guided attention mask emphasizes useful evidence. It changes what a model sees without retraining the entire vision-language backbone. Yet the same operation can help one question and hurt another. Highlighting an object may aid recognition while obscuring small text needed for OCR; blurring the background may improve localization but remove evidence needed to describe the scene.

RedCircle, FGVP, and API mainly address how to construct prompts or which configuration works well across a benchmark. AutoV identifies a finer-grained decision: which prompt should this particular image-question pair use? Obtaining supervision is the next obstacle. Humans cannot always recognize the most effective prompt, while answer correctness supplies a coarse signal that can reward guesses based on language priors. The paper therefore uses reference answers from existing image-question data to compare candidates' conditional language modeling losses, avoiding new human prompt-preference annotations.

Core idea: formulate visual prompt selection as query-conditioned relative preference ranking, use downstream LVLM answer losses to produce offline supervision, and train a small ranker to approximate that expensive candidate comparison.

Method

Overall Architecture

The input is an image and a textual question; the output remains an LVLM-generated answer. AutoV first constructs several prompted versions of the image. During training, reference answers provide candidate rankings; during inference, only the image and question are used to predict scores, and the full decoder processes one selected candidate.

Here, retrieval does not query an external image database or insert retrieved demonstrations into the context. Candidates are different prompted versions of the same input. The objective is to select the visual input most helpful for answering the current question, not the image with the highest visual similarity.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Image and question"] --> Pool["Diverse Candidate<br/>Representations"]
    Pool --> Supervision["Loss-Based Preference<br/>Supervision"]
    Answer["Reference answer<br/>Offline training only"] --> Supervision
    Pool --> Ranker["Query-Aware Ranking"]
    Supervision -->|Train mapping modules| Ranker
    Ranker --> Select["Robust Single-Candidate<br/>Inference"]
    Pool -->|Visual-feature pre-filtering| Select
    Select --> Output["Selected prompt and question<br/>Full decoding for the answer"]

Key Designs

1. Diverse Candidate Representations: make alternative visual evidence selectable

The main experiments use 6 prompted images: 4 attention prompts produced by API from different CLIP layers, plus one RedCircle prompt and one FGVP prompt. These are combinations of existing visual prompting methods, not newly trained prompt generators. Their diversity allows instance-specific choices among object emphasis, contextual information, and fine-grained regional cues. It also sets a hard boundary: selection can only exploit evidence already present in the pool and cannot recover information destroyed by every candidate.

Each candidate passes through the LVLM's existing visual encoder and projector, producing visual tokens with the same dimensionality as the language embeddings. Both components retain their pretrained weights and remain frozen. The ranker therefore does not need to relearn visual semantics from pixels or train a separate classification head for each prompt type. It subsequently compares the resulting visual representations in the context of the current question.

2. Loss-Based Preference Supervision: compare candidates within a question instead of regressing difficulty across questions

Offline data construction pairs every prompted image with the same question and runs a pretrained LVLM to compute its conditional language modeling loss against the reference answer. Lower loss defines the preferred candidate. The claim of avoiding manual annotation refers specifically to new prompt-level preference labels; it does not mean reference answers are unnecessary, nor does it mean correct answers are available for selecting prompts at test time.

Two filters remove unreliable training groups. Low variance across candidate losses suggests insensitivity to the visual prompt, while excessively high mean loss suggests an outlier or a question poorly supported by all candidates. The authors associate the former with reliance on language priors, but low variance alone does not establish that explanation: all candidates might preserve sufficient evidence. The cached main text does not provide filtering thresholds, so the exact selection of training data cannot be reconstructed from it.

For each retained group, all unordered candidate pairs are formed. The lower-loss member is chosen and the higher-loss member is rejected. Training thus asks which candidate is better for the same question rather than calibrating losses across questions of very different difficulty. This is useful when prompt differences are small and task difficulty varies substantially. The trade-off is that the number of comparisons grows combinatorially with the candidate count.

3. Query-Aware Ranking: reuse shallow interaction and train two small mapping modules

The ranker concatenates candidate visual tokens with question tokens and passes them through LLM layer 0, the first decoder layer. Separate visual and textual FFNs, each containing two linear layers, then map the resulting representations into a shared low-dimensional space. Cross-attention followed by mean aggregation produces a scalar reward for each candidate. Selection therefore depends on the question's context rather than only on a prompt type's average usefulness during training.

The paper states that only the visual and text mapping modules require additional training; the visual encoder, projector, and reused LLM layer are not additionally fine-tuned. For language embedding dimension \(D\) and mapping output dimension \(h\), with \(h\ll D\), the reported total parameter count of the two mapping modules is \(2h(D+h+2)\). The lightweight design comes from reusing semantic representations and compressing the mapping dimension, not from retraining an entire vision-language model.

Equations (2) and (3) are incompletely extracted in the cache. This note therefore retains the interaction, mapping, cross-attention, and aggregation operations supported by the surrounding text without inventing Q/K/V assignments, attention masks, or exact tensor reductions. In particular, mean aggregation of cross-attention must not simply be replaced with cosine similarity; those are different operations.

4. Robust Single-Candidate Inference: reject a feature outlier and decode only one candidate

At inference there is no reference answer, and the system does not run the complete LVLM separately on every candidate to obtain supervision losses. It first removes the prompt whose visual features are most dissimilar to the other candidates under cosine distance, then selects the remaining candidate with the highest predicted reward. The cached main text does not specify how distances across multiple candidates are aggregated, so this note does not reinterpret the rule as distance to a centroid or mean pairwise distance.

The selected visual tokens and text embeddings are passed to the LLM to generate the answer. This avoids repeated full answer decoding, not multiple-image encoding: candidate construction, encoding, shallow interaction, and scoring still cost computation. The authors report that pre-filtering discards the optimal prompt in fewer than 0.5% of cases, an empirical statistic rather than a guarantee under arbitrary distribution shifts. For closed-source transfer, an AutoV trained on an open model selects the prompted image; it does not access the closed model's hidden layers or train its parameters.

A Worked Example

The supervision illustration in Figure 3 provides losses for 3 candidates: 0.131 for an attention mask, 0.178 for a blur mask, and 0.264 for a red circle. These yield 3 preferences: attention mask over blur mask, attention mask over red circle, and blur mask over red circle. The ranker learns these orderings rather than directly predicting those loss values.

These numbers come from the paper's supervision illustration, not an additional benchmark experiment. At test time, only the question and candidate images are supplied, and predicted rewards determine selection. Reference answers belong exclusively to offline labeling, and the learned ranker is not guaranteed to reproduce the illustrated ordering every time.

Loss & Training

The following is a normalized restatement based on Section 3.3 and the definitions around Equation (4), not a verbatim reconstruction of the corrupted cached equation. Let \(n\) be the number of candidates, \(\mathcal P\) the chosen/rejected pair set determined by reference-answer losses, \(s\) the ranker score, and \(\sigma\) the sigmoid function:

\[ \mathcal L_r=-\frac{1}{\binom{n}{2}}\sum_{(c,r)\in\mathcal P}\log\sigma\bigl(s(\mathrm{VP}_c)-s(\mathrm{VP}_r)\bigr). \]

Minimizing this objective gives higher rewards to candidates with lower answer losses. It borrows preference training from reward modeling but does not update the LVLM through reinforcement learning; supervision trains only the prompt ranking modules. With the main pool of 6 candidates, each retained instance yields 15 unordered candidate pairs. The cached main text does not explain how equal losses are handled.

The main text reports 40 training epochs over 6 hours on 8 A100 GPUs, but the GPU memory specification is malformed. Training data composition and implementation details are deferred to Appendices A and C, whereas the available cache contains only the main paper and references. Sample counts, learning rate, optimizer, and the numerical value of \(h\) are therefore not supplied here.

Key Experimental Results

Main Results

The following values are selected from the original Table 1. Gains are absolute differences in benchmark scores, not relative percentage improvements. Each row uses its own model configuration; API is a strong visual prompting baseline in that table, not a claim about the historical best result among all published methods.

Model	Benchmark	Base	API	AutoV	Gain over Base
LLaVA-1.5 7B	MMMU	36.3	37.4	38.7	+2.4
LLaVA-1.5 7B	VizWiz	50.0	51.3	52.1	+2.1
LLaVA-OneVision 7B	MMMU	49.4	50.8	54.0	+4.6
LLaVA-OneVision 7B	VizWiz	58.2	66.9	68.4	+10.2
InternVL2 8B	VizWiz	58.6	61.2	64.8	+6.2
Qwen2.5-VL 7B	MMMU	50.1	51.0	53.9	+3.8

The headline +10.2 is relative to the unprompted Base; the same row improves over API by +1.5. Attributing the entire gain to ranking would therefore be misleading. InternVL2 and Qwen2.5-VL use the retrieval strategy trained on LLaVA-OneVision, supporting transfer of prompt selection across architectures.

Ablation Study

The original Table 2 compares retrieval strategies using the same candidate pool and LLaVA-1.5 7B backbone. Avg. is the average score across the three displayed benchmarks.

Retrieval strategy	MMMU	VizWiz	MMVet	Avg.
Base	36.3	50.0	30.6	39.0
Random selection	37.1	50.7	31.3	39.7
Absolute loss regression	36.9	50.9	31.1	39.6
MoE / GateNet	37.6	50.7	32.0	40.1
List-wise ranking	38.0	51.3	32.4	40.6
AutoV pairwise ranking	38.7	52.1	32.9	41.3

The original Table 4 separately examines interaction depth and pre-filtering. The first two rows below change only the interaction layer; the remaining rows explicitly identify candidate count. Differences between pool sizes must not be attributed solely to filtering.

Configuration	MMMU	VizWiz	MMVet
Layer 24 interaction	38.1	51.6	32.3
Layer 12 interaction	38.4	51.7	32.3
Layer 0, 6 candidates, with pre-filtering (default)	38.7	52.1	32.9
6 candidates, without pre-filtering	38.3	52.0	32.6
8 candidates, without pre-filtering	38.4	52.2	32.7
8 candidates, with pre-filtering	38.9	52.6	33.1

Key Findings

• Pairwise ranking exceeds MoE by 1.2 average points and list-wise ranking by 0.7. Learning relative preferences contributes more than merely constructing additional images. This is evidence for the current data and pool, not proof that pairwise objectives universally outperform list-wise objectives.
• With the default 6 candidates, pre-filtering adds +0.4, +0.1, and +0.3; with 8 candidates, the gains are +0.5, +0.4, and +0.4. Removing an outlier becomes more useful as the pool expands, but these local ablations do not establish a global ranking of every module's contribution.
• In the original Table 3, the average grows from 41.3 with 6 candidates to 41.6 with 8, indicating diminishing returns. Table 6 reports 0.74T extra and 5.08T total FLOPs for 4 candidates. The main text reports latency increasing from 254.8 ms to 261.7 ms, an additional 6.9 ms; these measurements are not a fixed overhead for every resolution or serving environment.

Highlights & Insights

• Asking which prompt is better for this question makes complementarity among existing methods learnable. A prompt with weak average performance may still help particular instances, so retrieval differs from retaining only the benchmark-wide winner.
• Producing dense preferences from reference-answer losses without regressing their absolute values is the most reusable idea. It suggests possible extensions to crop, resolution, or frame-subset selection, although those applications were not validated in this paper.
• Keeping the backbone frozen, training only mappings, and transferring the selected prompt to another LVLM separates training costs from the deployment interface. Closed-source adaptation requires replaceable image input, not access to the model's internal representations.

Limitations & Future Work

• Supervision still requires reference-answer data and a teacher LVLM. Low teacher loss is neither objective correctness nor proof of visual grounding. Useful follow-ups include comparing teachers and auditing samples removed by the low-variance filter.
• Pre-filtering assumes visual-feature outliers are more likely to be harmful, but a rare, correct prompt could also be an outlier. The reported optimal-prompt removal rate below 0.5% needs verification under distribution shifts and is not a safety guarantee.
• Candidate generation, multiple-image encoding, and offline per-candidate evaluation are not free. Runtime measurements belong to a specific setting; the cache does not fully disclose end-to-end candidate generation and closed-source service costs.
• Missing appendices prevent verification of training data proportions, exact hyperparameters, and complete multi-seed statistics. The reported MMMU multi-seed mean gain of +2.06 differs in reporting scope from the main table's +2.4 and should not be substituted for it.
• The cached Table 1 has arithmetic inconsistencies: InternVL2 MathVista is listed as 58.3 to 70.5 but labeled +2.2; LLaVA-OneVision Circle/MMVet is listed as 48.8 to 47.2 but labeled +1.6. This note neither draws conclusions from those entries nor silently repairs the source data.

Related Work & Insights

• Compared with API, RedCircle, and FGVP: these methods create different visual cues, whereas AutoV includes them in a candidate pool. Its contribution is query-dependent selection, not a replacement for their prompt generation mechanisms.
• Compared with MoE / GateNet: GateNet learns selection through differentiable gating, while AutoV explicitly exploits the ordering of candidate losses. Comparisons controlling data and training budget provide stronger evidence for the ranking objective itself.
• Compared with external retrieval augmentation: AutoV adds neither external knowledge nor new image evidence; it changes how the same input is presented. It can improve attention allocation but does not solve missing visual information or questions requiring outside facts.

Rating

• Novelty: 4/5. Applying answer-loss preferences to instance-level visual prompt retrieval is a clear contribution, although pairwise reward modeling is an established tool.
• Experimental Thoroughness: 4/5. Multiple backbones, retrieval objectives, interaction layers, pre-filtering, and pool sizes are evaluated; missing appendices restrict the evidence verifiable here.
• Writing Quality: 3/5. The main argument is accessible, but corrupted cached equations and conflicting table deltas increase the verification burden.
• Value: 4/5. Input-side improvements without backbone fine-tuning are useful, provided gains are weighed against candidate construction costs and strong prompting baselines.