Skip to content

v retrver evidence driven agentic reasoning for universal multimodal retrieval

content_hash: 5306c31e4c7aaa87 ---# V-Retrver: Evidence-Driven Agentic Reasoning for Universal Multimodal Retrieval

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/chendy25/V-Retrver
Area: Multimodal VLM
Keywords: Universal Multimodal Retrieval, Multimodal Agent, Multimodal Chain-of-Thought, Reinforcement Learning, Active Visual Inspection

TL;DR

V-Retrver reformulates universal multimodal retrieval as an active agentic reasoning process grounded in visual inspection, enabling an MLLM to dynamically invoke image selection and zoom-in tools during reasoning and optimizing the policy via a three-stage curriculum with contrastive NDCG and tool-use rewards to eliminate speculative language hallucination.

Background & Motivation

Multimodal Large Language Models (MLLMs) have recently demonstrated promising potential in universal multimodal retrieval, unifying disparate tasks such as text-to-image, image-to-text, and interleaved multimodal search. To enhance candidate discrimination and ranking interpretability, recent literature increasingly incorporates Chain-of-Thought (CoT) reasoning into candidate reranking. Nevertheless, prevailing CoT-based retrieval systems remain fundamentally language-driven: candidate images are typically compressed upfront into static feature embeddings or coarse textual summaries, and the subsequent reasoning process unfolds strictly within the language space without any mechanism to actively re-inspect visual details.

This language-centric bottleneck becomes severely detrimental in fine-grained or visually ambiguous retrieval scenarios. In settings such as composed image retrieval or dense catalog search, candidate images often share broad categorical semantics while differing only in minute visual details—such as specific embroidery, subtle texture sheen, button counts, or localized background objects. Compressing these visual inputs into fixed embeddings irrecoverably strips away fine-grained nuances, forcing the model to rely on speculative language reasoning to bridge visual ambiguities, which frequently leads to visual hallucination. Even recent reinforcement-learning-enhanced frameworks such as Retrv-R1, while deepening textual reasoning, remain bound to single-pass visual representations and lack the agency to "look again" when resolving ambiguous hypotheses.

Real-world human visual inspection naturally resolves such dilemmas through iterative hypothesis testing and active closer observation. Core idea: reformulate universal multimodal retrieval as an evidence-grounded agentic reasoning process that tightly interleaves hypothesis generation with targeted visual inspection (MIER), dynamically acquiring uncompressed visual evidence via external perceptual tools and steering the policy via a three-stage curriculum reinforcement learning objective (EAPO) that balances ranking accuracy with tool economy.

Method

Overall Architecture

V-Retrver adopts a coarse-to-fine two-stage retrieval pipeline, cleanly decoupling high-throughput candidate proposal from compute-intensive evidence-based agentic reasoning. In the first stage, a dual-encoder multimodal embedding model \(\phi\) (constructed following LamRA) projects the query \(q\) and all candidate items \(c_n\) into a shared representation space, retrieving a reduced top-\(K\) candidate set \(\mathcal{C} = \{c_k\}_{k=1}^K\) based on cosine similarity (\(K=50\) for M-BEIR evaluation). In the second stage, an evidence-gathering reranking agent \(\theta\) performs listwise reranking over \(\mathcal{C}\) through Multimodal Interleaved Evidence Reasoning (MIER). The agent iteratively outputs textual reasoning thoughts, issues tool invocation requests, and incorporates high-resolution perceptual feedback until it terminates with a definitive ranked list.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Query q and Candidate Pool Ω"] --> B["Coarse Embedding Retrieval<br/>Dual-encoder proposes Top-K candidate pool"]
    B --> C["Multimodal Interleaved Evidence Reasoning MIER<br/>Initial hypothesis generation and pre-screening"]
    C --> D["External Visual Perception Tools<br/>SELECT-IMAGE focus / ZOOM-IN localized cropping"]
    D --> E["Dynamic Evidence Feedback & Revision<br/>Append verified observations to context history"]
    E --> F["Evidence-Aligned Policy Optimization EAPO<br/>Three-stage curriculum: SFT → RSFT → GRPO"]
    F --> G["Output Final Candidate Ranking List answer"]

Key Designs

1. Multimodal Interleaved Evidence Reasoning (MIER): Upgrading one-way reasoning into a perception-action loop

Standard language-based CoT operates unidirectionally over frozen visual encodings, making later reasoning steps susceptible to drift and hallucination. MIER restructures this process into a multi-turn grounded trajectory \(\tau = \{T_1, C_1, V_1, T_2, C_2, V_2, \ldots, T_n, A_n\}\). At step \(k\), the MLLM produces a textual reasoning thought \(T_k\) alongside a tool invocation request \(C_k\). An execution parser intercepts \(C_k\), executes the external visual operator, and returns fresh, uncompressed visual evidence \(V_k\), which is dynamically appended to the reasoning context for subsequent deduction. This ensures that every high-stakes comparative deduction (e.g., verifying whether a garment features metallic sheen or specific pocket counts) is anchored directly in newly verified visual observations.

2. Complementary External Visual Toolset: Decoupling subset comparison and localized high-resolution inspection

To prevent token explosion while granting precise perceptual agency, V-Retrver equips the agent with two minimal yet orthogonal perceptual tools: - SELECT-IMAGE: Designed for scenarios where multiple candidate images exhibit high semantic overlap. The agent selects a focused subset of 1 to 4 candidate images from the top-\(K\) pool for direct, side-by-side contrastive analysis, eliminating distraction from irrelevant candidates. - ZOOM-IN: Allows the agent to zoom in on specific regions of a query or candidate image by specifying a bounding box \([x_{\min}, y_{\min}, x_{\max}, y_{\max}]\). It supplies high-resolution crops of localized regions, allowing the model to inspect subtle attributes (such as fabric patterns, logos, and structural accessories) that global visual tokens discard. The toolset is naturally extensible, permitting seamless integration of OCR tools (such as EXTRACT-TEXT) for text-rich multimodal document search.

3. Evidence-Aligned Policy Optimization (EAPO): Harmonizing ranking discrimination with tool-use parsimony

Transforming a base MLLM into an effective retrieval agent that reasons accurately without calling tools redundantly requires structured policy guidance. V-Retrver implements a three-stage curriculum: - Stage I (SFT Activation): Cold-start supervised fine-tuning on synthesized multimodal CoT trajectories generated by Qwen2.5-VL-72B-Instruct, filtered by rule-based consistency checks to establish basic reasoning syntax and tool invocation formats. - Stage II (Rejection Sampling Fine-Tuning - RSFT): Multiple trajectories are sampled per training instance; only trajectories satisfying strict formatting and yielding completely correct ranking predictions are retained for fine-tuning, dramatically suppressing rollout variance. - Stage III (EAPO Reinforcement Learning): Instantiated via Group Relative Policy Optimization (GRPO) to optimize a composite trajectory-level reward: $\(R_i = \alpha r_{\text{format}}(o_i) + \beta r_{\text{rank}}(o_i) + r_{\text{tool}}(o_i)\)$ The format reward \(r_{\text{format}}\) ensures strict compliance with <think>, <tool_call>, and <answer> formatting. The contrastive NDCG reward \(r_{\text{rank}}\) addresses reward sparsity and penalizes hard negative confusions: given ground-truth candidate \(j^*\) and the top-\(m\) hard negatives \(H\) (highest query embedding similarities), relevance at rank \(k\) is graded as: $\(\text{rel}_{\hat{\pi}}(k) = \begin{cases} 2 & \hat{\pi}(k) = j^* \\ 1 & \hat{\pi}(k) \in H \text{ and ranked after } j^* \\ 0 & \text{otherwise} \end{cases}\)$ Placing a hard negative above \(j^*\) incurs zero reward, providing dense and severe gradient penalties against catastrophic ranking errors. Meanwhile, the tool reward \(r_{\text{tool}}(o_i) = \eta \cdot \mathbb{I}_{\{k=1\}} \cdot \mathbb{I}_{\{N_{\text{tool}}>0\}} - \rho \cdot \max(0, N_{\text{tool}} - \tau)\) awards bonus \(\eta=0.2\) only when tool usage leads to a rank-1 retrieval success, and penalizes invocations exceeding the tolerance threshold \(\tau=1\) with penalty \(\rho=0.1\). This guides the agent to converge on an economical average of 1.6 to 1.7 tool calls per query.

A Worked Example

Consider a composed image retrieval query: "White sofa with mottled pillows instead of a dark brown sofa with white pillows and books down". The initial retrieval stage returns 5 candidate images. 1. Initial Pre-screening & Selection: In the <think> block, the agent notes Candidate 2 depicts the negative prompt (dark brown sofa with books) and dismisses it; Candidate 3 is plain white without mottled textures. Candidates 1 and 4 both display white sofas with patterned pillows. The agent issues <tool_call>{"name": "select_images", "arguments": {"target_images": [1, 4]}}</tool_call>. 2. Targeted Zoom-in Verification: Observing the side-by-side selected pair, the agent observes that Candidate 1 might feature integrated shelving under the cushions, issuing <tool_call>{"name": "crop_image", "arguments": {"bbox_2d": [290, 80, 380, 200], "target_image": 1}}</tool_call>. 3. Decision Synthesis: The crop confirms the detailed structural properties aligning with the transformation specifications, while Candidate 4 lacks integrated shelving. The agent finalizes its reasoning and emits the listwise ranking <answer>[1, 4, 3, 5, 2]</answer>.

Loss & Training

During Stage III, EAPO employs a critic-free GRPO objective. For each query \(q\), the model samples \(G=8\) rollouts \(\{o_1, \ldots, o_G\}\), computing normalized group advantages \(A_i = (R_i - \text{mean}(R)) / \text{std}(R)\). The policy optimization objective is: $\(J_{\text{EAPO}}(\theta) = \mathbb{E}_q \left[ \frac{1}{G} \sum_{i=1}^G \min\left(r_i(\theta) A_i, \text{clip}(r_i(\theta), 1 \pm \epsilon) A_i\right) \right] - \lambda \mathbb{E}_q [\text{KL}(\pi_\theta \parallel \pi_{\text{ref}})]\)$ where \(r_i(\theta) = \pi_\theta(o_i|q) / \pi_{\theta_{\text{old}}}(o_i|q)\), clipping hyperparameter \(\epsilon=0.2\), KL penalty coefficient \(\lambda=0\) (using the RSFT policy checkpoint as reference), and learning rate \(1 \times 10^{-6}\) on 8 A800 GPUs.

Key Experimental Results

Main Results

On the universal multimodal retrieval benchmark M-BEIR (encompassing 8 cross-modal retrieval task types across 10 datasets, 190K test queries, and 5.6M candidate pool), main results are summarized below:

Models Architecture / Type qt→ci (COCO R@5) qi→ci (NIGHTS R@5) (qi, qt)→ci (CIRR R@5) (qi, qt)→ct (InfoSeek R@5) M-BEIR Avg. R@K
CLIP-L Dual-encoder 61.1 26.1 13.2 26.4 32.5
UniIR-CLIPSF Fine-tuned Retriever 81.1 32.0 44.6 48.9 50.6
Qwen2.5-VL-7B Base MLLM 71.9 25.5 55.1 54.9 46.7
Vision-R1-7B Reasoning-enhanced 75.0 25.9 57.7 59.0 50.2
MM-Embed-7B Embedding Retriever 71.3 32.4 50.0 57.7 52.7
LamRA-7B Agentic Reranker 85.2 33.5 63.3 78.3 63.7
U-MARVEL-7B Universal Embedding 85.6 34.7 63.2 78.9 64.8
Retrv-R1-7B Text-only CoT RL 86.7 39.5 72.3 83.7 69.2
V-Retrver-7B Active Evidence Agent (Ours) 87.5 39.8 73.5 85.0 69.7

On unseen out-of-domain evaluation, V-Retrver sets new zero-shot state-of-the-art results, achieving 48.2 MAP@5 on CIRCO (vs. LamRA 42.8 and MM-Embed 35.5) and 30.7 R@1 on GeneCIS (vs. LamRA 24.8).

Ablation Study

Systematic ablations isolate the contribution of each curriculum stage and the visual tool mechanism:

Ablation Configuration COCO R@5 (qt→ci) F200K R@10 (qi→ct) CIRR R@5 ((qi, qt)→ci) OVEN R@5 ((qi, qt)→ct) Avg. (4 Tasks) Note
Qwen2.5-VL-7B (Base) 71.9 19.4 55.1 42.4 47.2 Zero-shot ranking without training
Zero-shot Tool (w/o SFT & RSFT & RL) 71.5 18.1 53.4 40.2 45.8 Untrained tool prompting degrades performance
Stage I only (w/o RSFT & RL) 83.2 31.6 63.7 59.0 59.4 Activates syntax and tool invocation
Stage I + II (w/o RL) 83.9 32.8 65.3 61.5 60.9 Rejection sampling boosts consistency
Stage I + III (w/o RSFT) 87.2 37.3 72.4 68.3 66.3 Slower RL exploration without RSFT anchor
Text-only RL (RL w/o visual tool) 84.1 33.2 66.5 63.2 61.8 Identical RL but restricted from calling tools
Full Model (V-Retrver-7B) 87.5 37.8 73.5 69.8 67.2 Full 3-stage curriculum with visual tools

In reward component ablations, eliminating the contrastive ranking reward \(r_{\text{rank}}\) causes the most significant performance degradation of -2.4% (to 64.8%), validating the necessity of dense gradient signals from CNDCG. Dropping \(r_{\text{tool}}\) causes a -1.1% decline (to 66.1%), and dropping \(r_{\text{format}}\) drops performance by -1.7% (to 65.5%).

Key Findings

  • Active visual inspection decisively outperforms language-only reasoning: Comparing identical reinforcement learning optimization regimes, empowering the agent with visual tools yields a +5.4% average recall improvement over the text-only RL counterpart (67.2% vs. 61.8%), with a +7.0% boost on CIRR (73.5% vs. 66.5%), confirming that active zoom-in captures crucial discriminatory details lost in compressed static encodings.
  • Curriculum learning prevents policy collapse: Naive zero-shot tool usage harms performance (45.8% vs. 47.2%). SFT instills syntactic capability, RSFT stabilizes trajectory quality, and EAPO converges valid tool calls to total calls while naturally settling tool invocation frequency to 1.6–1.7 calls per query.
  • Listwise agentic interaction preserves competitive efficiency: Unlike pointwise rerankers (e.g. LamRA-Rank-Pointwise, which calls the model 50 times at \(K=50\), resulting in an inference time ratio of 4.41×), V-Retrver operates in a single multi-turn listwise session (ITR 1.0×), balancing high visual fidelity with manageable computational cost.

Highlights & Insights

  • Paradigm shift from passive feature matching to active evidence verification: Instead of endlessly expanding visual token budgets or letting language models hallucinate across compressed embeddings, V-Retrver enables MLLMs to inspect details on demand via lightweight external tool calls, mimicking human visual problem-solving.
  • Contrastive NDCG reward resolves sparse ranking feedback: Conventional RL often relies on sparse binary hit signals. EAPO assigns asymmetric rewards based on the relative positioning between ground-truth items and top hard negatives, penalizing deceptive hard negatives that outrank the target and establishing dense optimization gradients.
  • Extensible perception toolkit: Incorporating an OCR tool (EXTRACT-TEXT) yields an instant +0.4% recall gain on knowledge-dense retrieval tasks (OVEN) without modifying the underlying reward architecture or training regimen, demonstrating strong plug-and-play potential for richer vision tools.

Limitations & Future Work

  • Author-admitted limitations: The multi-turn tool interaction poses credit assignment challenges in long trajectories, where sequence-level RL cannot easily disentangle the marginal value of individual tool calls. In addition, multi-turn MLLM inference over uncompressed visual inputs incurs non-negligible computational overhead compared to aggressive token-compression methods.
  • Additional limitations: The candidate reranking pool is bounded (e.g., \(K=10\sim 50\)). If the upstream dual-encoder fails to recall the correct candidate into the top-\(K\), the downstream agentic reasoning stage cannot recover it.
  • Future directions: Integrating token-level generalized advantage estimation (GAE) across multi-turn reasoning steps to solve credit assignment; exploring adaptive early-exit mechanisms and dynamic tool scheduling to bypass inspection on easily separable queries.
  • vs Retrv-R1: While Retrv-R1 also enhances MLLM retrieval via reinforcement learning, it relies solely on text-based CoT over heavily compressed visual tokens; V-Retrver preserves uncompressed visual representations and provides active tools (SELECT-IMAGE and ZOOM-IN), securing significant performance gains on fine-grained visual benchmarks.
  • vs LamRA: LamRA performs pointwise scoring that scales linearly with candidate pool size (4.41× latency at \(K=50\)) and evaluates candidates in isolation; V-Retrver executes listwise agentic reasoning with global comparative context and active perceptual inspection at much lower relative latency.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneers active evidence-driven visual tool interaction for universal multimodal retrieval]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations spanning M-BEIR 8 tasks, 5 unseen datasets, held-out tasks, and ablation studies]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, rigorous mathematical formulation of CNDCG and EAPO, and well-designed qualitative figures]
  • Value: ⭐⭐⭐⭐⭐ [Provides a foundational methodology for agentic MLLMs in retrieval and downstream multimodal RAG systems]