ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answering¶

Conference: ECCV2026
Paper: Official page · Paper PDF
Code: https://github.com/DingWu1021/Promsa
Area: Agent / Knowledge-Based Visual Question Answering
Keywords: Progressive retrieval, multimodal tools, retrieval deduplication, sequence-level reinforcement learning, tool budgets

TL;DR¶

ProMSA turns knowledge-based visual question answering into a budgeted loop of retrieval, evidence checking, and either further search or answering, using rejection-sampling SFT followed by TN-GSPO to bring Qwen3-VL-8B to overall scores of 52.6 on E-VQA and 53.4 on InfoSeek.

Background & Motivation¶

Knowledge-based visual question answering requires more than recognizing what appears in an image: a system must connect a specific visual entity to external knowledge and then answer a question about its attributes or relationships. Identifying the lake in a photograph and determining its country are related but distinct steps. Long-tail entities make this particularly difficult. A model may recognize the category "lake" without knowing which lake it is, while differences in viewpoint and background between user photographs and Wikipedia images can put a visually similar but incorrect place at the top of retrieval results.

Fixed RAG pipelines such as EchoSight typically retrieve a set of pages, filter or rerank them, and generate an answer. Wiki-PRF improves processing before retrieval, while ReAG judges the relevance of candidate content. However, if the initial evidence concerns the wrong entity, selecting more carefully from the same candidates may not recover the correct answer. Other questions already identify the entity but lack one attribute, making another image search poorly targeted. The problem is therefore not simply choosing a larger top-k, but deciding whether the current state calls for entity identification, knowledge completion, or stopping.

The paper also argues that the difficulty of training a search trajectory depends on its external decisions, not just on how many words it generates. Two trajectories with three tool calls can have very different explanation lengths, so text length alone need not provide the most appropriate update scale. Core idea: let a single multimodal policy alternate between image search, text search, and stopping within explicit budgets, use visited-result deduplication to support correction, and incorporate tool-interaction depth into sequence-level reinforcement learning normalization.

Method¶

Overall Architecture¶

The input is an image-question pair and the output is a final answer. Despite the plural "Agents" in the title, the decision maker in the method is one multimodal model policy, not a collection of roles debating with one another. Conditioned on the image, question, earlier reasoning, and retrieved evidence, it generates the next reasoning segment, action, and arguments. EVA-CLIP handles image retrieval, BGE handles text retrieval, and a Qwen3-VL-8B service summarizes retrieved content before it enters the next context.

At inference, budgeted action selection leads to deduplicated retrieval and question-conditioned summarization, then returns to the same policy for another decision. Training first uses executable, correctly answered trajectories for SFT and then optimizes the interaction with TN-GSPO. The dashed edge below represents training the policy, not an additional training module invoked during inference.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Image and question"] --> B["Budgeted action selection"]
    B -->|Image or text search| C["Deduplicated retrieval"]
    C --> D["Question-conditioned<br/>summarization"]
    D -->|Update context| B
    B -->|Stop| E["Final answer"]
    F["Executable-trajectory training<br/>and TN-GSPO"] -.-> B

Key Designs¶

1. Budgeted action selection: distinguish unidentified entities from missing knowledge

The action space contains only img_search, text_search, and stop. Each round produces reasoning followed by an action and arguments, such as a rewritten text query, top-k, and an exclusion list. Selecting stop initiates answer generation. If the visual entity remains uncertain, image search can locate its Wikipedia page; once the entity is mostly identified, a text query can target a missing attribute or a relation involving another page. Tool selection thus follows the observed evidence rather than a retrieval modality chosen in advance for the dataset.

The default configuration permits at most three image searches and three text searches, returning three pages or three text sections per call, respectively. These are upper bounds, not a requirement to spend all six calls; the model may also answer without retrieval. The paper allows up to seven interaction steps, consistent with a budget of six tool calls followed by termination. Adaptivity here primarily concerns tool modality, query content, search depth, and stopping. It should not be interpreted as evidence that the default experiments freely vary top-k at every step.

2. Deduplicated retrieval: make another search leave the previous candidates behind

Allowing multiple calls is insufficient if the same image repeatedly retrieves the same incorrect page. In that case, subsequent reasoning can simply reinforce the initial misidentification. ProMSA maintains a set of retrieved results, passes it to the tool as an exclusion list, and adds newly retrieved results afterward. Image retrieval excludes visited pages, while the framework figure depicts text retrieval excluding visited sections, directing subsequent calls toward new evidence. The prose describes this mechanism using a unified page set; the exact page-versus-section implementation should therefore be checked against the code.

This effectively explores farther into the available ranking; it does not imply that the retriever itself has become stronger or that correction is guaranteed. The policy still decides whether to continue. It must recognize a mismatch between the retrieved page and visual evidence before spending another call. When the entity is already known but an attribute is missing, it can instead rewrite the text query without repeatedly expanding visual candidates. Deduplication supplies the ability to escape old results, while reasoning and training determine whether the resulting evidence is trustworthy.

3. Question-conditioned summarization: retain evidence needed for the next decision

Wikipedia pages returned by image search can be long, as can chunks returned by text search. Appending every full result at every round would rapidly expand the context. The system instead summarizes retrieved content with respect to the question, retaining entity descriptions, key attributes, and relationship cues. These snippets become tool observations appended to the context. Both retrieval modalities consequently feed a common textual evidence interface, rather than requiring the policy to consume full pages and sections through different mechanisms.

Summarization is not merely token compression: the next search decision depends on whether the snippets preserve evidence about entity compatibility and answer completeness. Losing a visual contradiction or an important qualifier may change the stopping decision, making summarization a potential information bottleneck. The paper explicitly deploys Qwen3-VL-8B as the summarization service. A 2B policy variant therefore does not mean that the entire system uses only a 2B model; the auxiliary service must be included in reproduction and cost accounting.

4. Executable-trajectory training and TN-GSPO: make update scale aware of tool depth

Cold-start training samples multiple interaction trajectories for each image-question pair and retains only those with valid formats, executable calls, respected budgets, and correct final answers, keeping at most one trajectory per question. SFT therefore teaches runnable call structures and interaction patterns, not just final-answer demonstrations. Both SFT and RL train on model-generated tokens only. Tool-returned content conditions subsequent generation but is masked out as a prediction target.

RL assigns a sparse return to the whole trajectory, combining answer correctness, format compliance, and tool cost. With notation cleaned up from Equation (12), the return is given below. Here, \(H_{\max}\) is the maximum permitted number of tool calls and \(\lambda=0.5\). The cached paper does not specify the complete scoring rules for the two correctness components, so they should not be assigned an invented binary scoring scheme.

\[ R(\tau)=r_{\mathrm{ans}}(\tau)+r_{\mathrm{fmt}}(\tau) -\lambda\frac{\#\mathrm{tool\_calls}(\tau)}{H_{\max}}. \]

Standard GSPO already normalizes a sequence likelihood ratio by generated length. TN-GSPO additionally incorporates the tool horizon. Let \(L\) count trainable generated tokens, let \(H=1+\#\mathrm{tool\_calls}\), and let \(\Delta_t\) be the log-probability difference between the current and reference policies for the same generated token. The central normalization is:

\[ D(\tau)=L(\tau)\bigl(1+cH(\tau)^\alpha\bigr),\qquad r_\theta(\tau)=\exp\!\left(\frac{\sum_{t\in T_{\mathrm{gen}}}\Delta_t}{D(\tau)}\right), \quad c=0.04,\ \alpha=1. \]

The method then uses a centered advantage \(A(\tau)=R(\tau)-b(x)\) and optimizes the minimum of the unclipped and clipped sequence ratios multiplied by that advantage, with clipping bounds \([0.8,1.28]\). A deeper tool trajectory applies a larger denominator to the same accumulated log-probability change. This changes the update scale; it does not award a separate bonus for using more tools. The return's cost term, by contrast, penalizes calls. These mechanisms target optimization stability and search efficiency, respectively, and should not be conflated.

A Worked Example¶

Figure 1 asks which country contains the lake shown in an image. Direct answering identifies it as Lake Köyceğiz in Turkey, while the first retrieved evidence points to the Bay of Kotor. The agent notices a conflict between the image and the returned description, continues searching, obtains information about Lake Orestiada, and answers Greece. The example illustrates that correction requires allowing new evidence to change an entity hypothesis, not simply producing a longer explanation of the first result.

However, the illustrated final reasoning still describes the water body as a coastal inlet or lagoon, which is not fully consistent with the retrieved lake description. The example demonstrates correction of the country answer, not that every entity identification and explanation is correct. Figure 4 separately shows a correct entity appearing at rank ten. With a default image-search budget of three calls returning three pages each, that visualization does not establish that the default budget necessarily reaches this candidate.

Loss & Training¶

SFT samples 3,000 instances from the E-VQA and InfoSeek training sets, freezes the visual encoder and multimodal projector, and fine-tunes only the language model for three epochs at a learning rate of \(10^{-5}\) using LLaMA-Factory. RL samples 15,000 instances and trains with veRL for three epochs, with global batch size 128 and learning rate \(10^{-6}\). These are the reported sampling sizes, not necessarily the number of trajectories retained after rejection sampling.

Generation is capped at 4,096 tokens per step and 16,384 tokens per trajectory. Training uses eight A800 GPUs. Policy backbones include Qwen2.5-VL-7B and Qwen3-VL-2B/8B. Generated length, summary length, and tool-call limits are distinct constraints; the paper does not combine them into a directly comparable total-FLOPs measure.

Key Experimental Results¶

Main Results¶

E-VQA contains approximately 221K question-answer pairs covering 16.7K fine-grained entities, with a knowledge base of about two million pages. Evaluation uses BERT-based Matching (BEM) for semantic answer agreement. InfoSeek contains approximately 1.3M image-question-answer triplets; the experiments follow prior work in using a 100K-page retrieval subset rather than the full knowledge base of six million entities. Its evaluation uses VQA Accuracy or Relaxed Accuracy according to question type. The table retains the paper's reported score scale; the two datasets should not be treated as having identical metrics.

Method	Policy / generator backbone	E-VQA single-hop	E-VQA all	InfoSeek unseen-Q	InfoSeek unseen-E	InfoSeek all
Qwen3-VL zero-shot	Qwen3-VL-8B	25.3	24.8	25.9	25.5	25.7
MMSearch-R1, local retrieval reproduction	Qwen2.5-VL-7B	40.6	40.7	40.3	39.3	39.7
DeepEyesV2, local retrieval reproduction	Qwen2.5-VL-7B	40.1	39.5	41.6	40.8	41.3
CC-VQA	Qwen2.5-VL-7B	41.4	36.1	44.7	46.1	45.1
REAL	Qwen3-VL-8B	45.5	41.4	43.1	45.1	44.1
ProMSA	Qwen2.5-VL-7B	50.0	49.7	48.8	49.6	49.2
ProMSA	Qwen3-VL-2B	42.4	41.2	44.1	43.0	43.6
ProMSA	Qwen3-VL-8B	52.2	52.6	53.6	53.3	53.4

These values come from Table 1. MMSearch-R1 and DeepEyesV2 replace their original SerpApi retrieval service with local BGE + EVA-CLIP retrieval, so these are not the original online-service configurations. The 8B ProMSA improves over same-backbone REAL by 11.2 and 9.3 overall points, respectively. The 7B ProMSA improves over same-backbone MMSearch-R1 by 9.0 and 9.5 points. These comparisons nevertheless combine changes in search workflow and training rather than isolating a single component.

Ablation Study¶

The following table combines selected rows from Tables 2, 3, and 4 under the main Qwen3-VL-8B configuration. An asterisk in the original optimizer table denotes asymmetric clipping, which is written explicitly here.

Experiment group	Configuration	E-VQA all	InfoSeek all
Training stage	Base: untrained search framework	32.8	36.4
Training stage	Rejection-sampling SFT	38.6	42.1
Optimizer	GRPO	44.2	43.7
Optimizer	GRPO + asymmetric clipping	49.7	50.2
Optimizer	GSPO + asymmetric clipping	49.3	49.6
Full model	TN-GSPO + asymmetric clipping	52.6	53.4
Tool availability	Text search only	27.6	36.8
Tool availability	Image search only	34.7	21.4

Base already uses the search framework but has not been trained; it is not the retrieval-free zero-shot model in the main table. TN-GSPO improves by 3.3 and 3.8 points over GSPO with the same clipping scheme, a more informative comparison for the added normalization than comparison with vanilla GRPO alone. Tool ablations retain the original per-tool call limit, so a single-tool setup also has fewer total available calls. Its entire performance drop cannot be attributed solely to the missing modality.

Key Findings¶

RL adds 14.0 and 11.3 points after SFT, showing that learning valid calls is not sufficient to learn effective search. Asymmetric clipping also contributes substantially on its own.
Increasing each tool's budget from two to three calls changes the scores from 48.2/48.7 to 52.6/53.4; four calls give 52.4/54.3. Increasing per-call top-k from three to four gives 52.1/54.1. More retrieval does not produce monotonic improvement.
Table 9 defines retrieval accuracy as retrieving knowledge containing at least one correct document. The three rounds score 39.1, 48.8, and 52.7, excluding samples answered without retrieval. The corresponding shares labeled "Incorrect Retrieval & Stop" are 14.5, 43.8, and 44.1. Improved retrieval must not obscure failed stopping decisions, and these joint-category shares should not be presented as conditional error rates.
On the additional OK-VQA evaluation, ProMSA 7B/8B scores 82.7/85.6, compared with 78.8 for CC-VQA 7B. This supports applicability across benchmarks but does not by itself establish generalization to all open-domain knowledge tasks.
Table 5 reports 1.8 seconds per sample and an E-VQA score of 49.7 for 7B ProMSA, versus 1.7 seconds/40.7 for MMSearch-R1 and 2.4 seconds/39.5 for DeepEyesV2. These are end-to-end reports under the local experimental setup, not latency estimates for online search services.

Highlights & Insights¶

Separating "wrong entity" from "missing attribute" gives tool selection a concrete semantic purpose. For fine-grained product or landmark retrieval, a similar system could verify identity before querying attributes rather than unconditionally concatenating more documents.
Deduplication makes later calls obtain new information instead of repeating the same retrieval result. It also suggests evaluating what evidence each call adds, not merely counting calls.
TN-GSPO brings tool-decision depth into sequence update scaling without requiring correctness labels for every intermediate action. Its contribution is a compact normalization term that can be added to sequence-level RL, rather than a new retrieval backbone.

Limitations & Future Work¶

The authors identify tool-cost constraints and uncertainty about long-tail entities as reasons for stopping while retrieved evidence is still incorrect; Table 9 exposes this behavior. Evidence-aware abstention and confidence calibration are natural follow-ups, rather than simply increasing budgets.
In terms of experimental coverage, the main paper does not independently remove deduplication or the summarization service, and it does not report variance across multiple random seeds. The experiments support the complete system but do not precisely separate every component's contribution.
A 2B policy depends on an 8B summarization service, while the retrieval indexes and retrievers also incur costs. The timing table does not provide a complete cost breakdown. A successful small policy therefore does not automatically imply the lowest total deployment cost at a given quality level.
InfoSeek uses a 100K-page subset and fixed retrievers. Performance under dynamic web noise, index updates, and full-scale knowledge bases remains to be tested. Whether exclusions prevent useful revisits, or summaries discard contradictory evidence, also deserves dedicated evaluation.
The entity-attribute inconsistency in Figure 1's final explanation illustrates that answer correctness and evidence-chain faithfulness are different properties. Future evaluation should jointly inspect entity alignment, evidential support, and the final answer rather than score only the short answer.

Compared with EchoSight, Wiki-PRF, and ReAG: these methods emphasize fixed retrieval-generation, multimodal processing before retrieval, and candidate relevance assessment, respectively. ProMSA places retrieval timing, retrieval choice, and termination under one trainable policy. Better preprocessing and filtering could still be incorporated on the tool side.
Compared with MMSearch-R1 and DeepEyesV2: these are also multimodal search agents, so ProMSA does not originate tool-based interaction. Its focus is long-tail entity correction, budgets and deduplication, and sequence optimization that accounts for tool depth. The main table's local-retrieval substitution must remain part of the comparison.
Compared with GSPO and GRPO: the change concerns how to scale updates for whole tool-interaction trajectories, not how to generate longer chains of thought. Because asymmetric clipping already produces substantial gains, the contribution of TN-GSPO should be assessed against a baseline with the same clipping scheme.

Rating¶

Novelty: 4/5. Budgeted multimodal search and deduplication are not entirely new, but tool-horizon normalization provides a concrete, ablatable contribution.
Experimental Thoroughness: 4/5. Multiple backbones, two main benchmarks, an additional benchmark, and training/tool-budget ablations provide broad evidence, while isolated deduplication tests, variance, and complete cost breakdowns are missing.
Writing Quality: 3/5. The main argument and system structure are clear, but example faithfulness and some implementation granularity could be explained more precisely.
Value: 4/5. The work offers a reusable search-training approach for long-tail knowledge-based VQA, subject to summarizer costs and the risk of stopping with incorrect evidence.