MMAgent-R2: Learning to Rerank and Reject for Agentic mRAG

Conference: ECCV 2026
Paper: ECCV Official Poster
Area: Information Retrieval & RAG
Keywords: Multimodal RAG, Knowledge-based VQA, Visual Agent, Reinforcement Learning, Visual Reranking

TL;DR

MMAgent-R2 unifies visual reranking and active rejection into an internal verification mechanism for agentic mRAG, optimized via GRPO with step-level verification rewards to break free from the constraints of static candidate pools and noisy global visual retrieval.

Background & Motivation

Knowledge-based Visual Question Answering (KB-VQA) tasks demand that a vision-language model recognize visual entities from query images and retrieve relevant knowledge from massive encyclopedic knowledge bases to deduce answers. Predominant multimodal Retrieval-Augmented Generation (mRAG) approaches follow a "Retrieve-then-Postprocess" paradigm: an initial visual retriever recalls a small, static set of Top-K candidates using global feature vectors, and subsequent post-processing modules filter irrelevant noise from this fixed pool before reasoning and answer generation.

However, existing mRAG pipelines encounter a severe entity identification bottleneck. First, retrieval stages rely primarily on global visual embeddings (e.g., coarse CLIP representations), which struggle to capture fine-grained visual differences across highly similar entities (such as subtle dome configurations or roof tiers of wooden churches), causing the candidate set to be swamped by visually similar yet factually mismatched distractors. While existing post-processing reranks textual passages linked to candidate entities, text descriptions inherently cannot resolve visual subtleties. Second and more critically, downstream reasoning is confined to the static initial candidate pool. If the true entity is missed in the initial retrieval, the model is compelled to hallucinate over incorrect entities, inevitably propagating errors down to the final response. Furthermore, conventional pipelines rely on rigid multi-stage heuristics or disparate critic modules, severely hindering joint optimization and stumbling on multi-image, multi-hop reasoning.

To overcome these fundamental limitations, the authors re-architect multimodal retrieval and verification as an autonomous agentic decision process. Core idea: integrate visual reranking and active rejection as native internal verification actions within a multi-turn agent loop, enabling the model to directly compare visual details (Rerank) or discard current distractors to fetch new candidates (Reject), jointly optimized via GRPO with step-level verification rewards.

Method

Overall Architecture

MMAgent-R2 formulates KB-VQA as a multi-turn agentic decision process. Driven by a policy \(\pi_\theta\), the VLM operates over an action space spanning external retrieval, internal verification, and answer generation. At each step, conditioned on its interaction trajectory, the agent generates reasoning thoughts and dynamically selects whether to query external knowledge tools, run fine-grained visual verification over candidate images, or synthesize evidence to conclude with a final answer.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Query Images {I_q} & Question Q"] --> B["Autonomous Multi-turn Reasoning"]
    B --> C["External Retrieval<br/>Multimodal Evidence Acquisition"]
    C -->|Return non-overlapping candidate batch| D["Internal Verification"]
    D --> E["Active Rejection Reject<br/>Discard distractors & fetch next batch"]
    D --> F["Visual Reranking Rerank<br/>Fine-grained image comparison"]
    E -->|Under Rej_max threshold| B
    F --> G["Text Evidence Search SearchText<br/>Parallel query aggregation"]
    G --> B
    B --> H["Answer Generation Answer<br/>Synthesize evidence & terminate"]

Key Designs

1. Unified Agent Action Space: Decoupling Acquisition, Verification, and Termination Unlike prior pipelines that splice together ad-hoc filtering scripts or external critic models, MMAgent-R2 houses all capabilities within a single unified action space \(\mathcal{A}\). External Retrieval includes image retrieval SearchImage(j) and text retrieval SearchText({qi}): the former supports dynamic candidate expansion by fetching non-overlapping subsequent batches of Top-K candidates upon repeated calls for the same query image; the latter allows parallel submission of multiple textual queries in one turn to gather multi-faceted facts. Internal Verification consists of visual reranking Rerank(ek) and active rejection Reject(): the former establishes the verified entity as a premise for subsequent reasoning, while the latter signals the environment to supply a fresh batch. Finally, Answer(y) terminates the episode. This clean action interface enables the model to autonomously plan its investigation without relying on external heuristics.

2. Visual-Level Verification Mechanism: Dual Defense with Reranking and Rejection Prior post-processing approaches operated almost exclusively on text passages, remaining blind to fine-grained visual differences. MMAgent-R2 elevates verification directly to the image feature and pixel comparison level. Upon receiving a candidate batch \(\mathcal{B}_t^{(j)} = \{(I_k, e_k)\}_{k=1}^K\), the agent directly contrasts the query image against each candidate image. When a candidate exhibits unmistakable visual alignment in shape, parts, or markings, the agent triggers Rerank(ek). Conversely, when all items in the batch are visually similar distractors, the agent executes Reject(). To prevent unbounded rejection loops and context window degradation, the environment enforces a maximum rejection threshold \(Rej_{\max}\) (set to 2 by default), injecting a boundary prompt when reached to balance thoroughness with execution cost.

3. Composite RL Supervision: Dense Step-Level Verification Rewards via GRPO Supervising multi-turn agentic KB-VQA solely through a sparse binary outcome reward leads to severe credit assignment challenges, leaving the policy unable to distinguish between valid rejections and hesitant guessing. MMAgent-R2 constructs a composite reward \(R = R_{\text{outcome}} + R_{\text{format}} + R_{\text{ver}}\). While \(R_{\text{format}}\) ensures strict syntactic execution, the core step-level verification reward \(R_{\text{ver}}\) provides dense feedback for every reranking and rejection decision across all candidate batches. Given candidate batch \(\mathcal{B}\) and ground-truth entity \(e^*\), the step reward \(\rho\) is defined as:

\[ \rho(a, \mathcal{B}, e^*) = \begin{cases} \delta, & \text{if } a = \texttt{Reject} \text{ and } e^* \notin \mathcal{B} \\ 1, & \text{if } a = \texttt{Rerank}(e^*) \\ 0, & \text{otherwise} \end{cases} \]

where \(\delta=0.2\) is the reward weight for successful rejection when the target entity is genuinely absent. Training is executed via Group Relative Policy Optimization (GRPO) on the verl framework, sampling \(G=8\) rollouts per instance. Crucially, tokens from environment feedback (retrieved image tokens and text passages) are masked out during loss calculation, ensuring that policy gradients update only the model's generated reasoning thoughts and actions.

A Worked Example

Consider a complex comparison question from MMhops: "How many more hull-loss accidents have involved the airplane in the second image compared to the first image?" 1. The agent calls SearchImage(1). The returned Top-5 candidates include the Lockheed L-1011 TriStar and Boeing 767. By examining the vertical stabilizer and engine arrangement, the agent rejects incorrect candidates and executes Rerank(McDonnell Douglas DC-10). 2. The agent calls SearchImage(2). The initial batch returns the Boeing 707 and DC-8. By analyzing the wing pylons and engine pods, the model performs Rerank(Douglas DC-8). 3. The agent triggers parallel text retrieval SearchText(["DC-10 hull loss accidents count", "Douglas DC-8 hull loss accidents count"]), receiving facts stating that the DC-10 was involved in 32 hull-loss accidents and the DC-8 in 84. 4. The agent performs mathematical reasoning \(84 - 32 = 52\) and outputs Answer(52). In contrast, a pipeline without reranking mistakenly selects the Top-1 candidates (L-1011 with 10 accidents and Boeing 707 with 174 accidents), computing an erroneous result of 164.

Key Experimental Results

Main Results

On E-VQA (featuring a massive 2M-page knowledge base), InfoSeek (evaluating unseen entities and questions), and MMhops (evaluating multi-image multi-hop chains), MMAgent-R2 consistently sets new state-of-the-art benchmarks across the board.

Dataset	Metric / Split	MMAgent-R2-8B	MMAgent-R2-7B	Previous SOTA (Baseline)	Absolute Gain
E-VQA (2M KB)	Single-Hop Acc (%)	55.9	55.4	53.7 (QKVQA-7B)	+2.2%
E-VQA (2M KB)	All Acc (%)	54.2	53.6	47.0 (ReAG-7B)	+7.2%
InfoSeek (100K KB)	All Acc (%)	49.1	50.2	47.2 (ReAG-7B)	+3.0%
InfoSeek (100K KB)	Unseen-Question (%)	49.0	50.7	48.3 (ReAG-7B)	+2.4%
InfoSeek (100K KB)	Unseen-Entity (%)	49.2	49.7	46.2 (ReAG-7B)	+3.5%
MMhops (Multi-hop)	Bridging Acc (%)	67.2	65.9	54.0 (Gemini-2.5-pro)	+13.2%
MMhops (Multi-hop)	Comparison Acc (%)	39.8	34.9	29.4 (Gemini-2.5-pro)	+10.4%

Ablation Study

Ablation experiments conducted on InfoSeek with MMAgent-R2-8B highlight the quantitative contributions of each design choice:

Configuration Variant	Entity Ident. Acc (%)	VQA Acc (All %)	Core Mechanism & Takeaway
Full Model (Rerank + Reject)	58.6	45.5	Dual synergy: visual reranking drives precision, rejection expands coverage
w/o Reject (Rerank only)	57.5	44.3	Confined to initial pool; early misses drop identification by 1.1%
w/o Rerank, w/o Reject (Top-1 default)	53.7	40.2	Reverts to classic mRAG; distracted by false positives, dropping 5.3%
Batch size \(K=3\)	58.0	44.7	Insufficient coverage per step causes excessive rejections (Avg. 0.63)
Batch size \(K=7\)	58.7	45.7	Marginal accuracy gain (+0.2%) at substantial context token cost
Max rejection \(Rej_{\max}=0\)	57.5	44.3	Equivalent to running without active rejection
Max rejection \(Rej_{\max}=3\)	58.1	44.8	Redundant candidates lengthen context and degrade reasoning quality
Rejection reward weight \(\delta=0.1\)	57.9	44.9	Weak incentive leaves model guessing over uncertain batches
Rejection reward weight \(\delta=0.3\)	58.4	45.2	Over-incentivization triggers unnecessary rejections and inflated context

Key Findings

• Disproportionate Gains on Challenging Retrieval: On E-VQA's 2M-entry knowledge base where retriever Recall@1 is merely 15.4%, MMAgent-R2-8B elevates entity identification accuracy to 29.5% (+14.1%), boosting final accuracy by 7.2%. Internal visual verification is most impactful when raw retriever accuracy is low.
• Pure RL Outperforms Complex Multi-Stage SFT: Previous top-performing pipelines like ReAG depended on multi-stage SFT cold-starts and separate critic classifiers. In contrast, MMAgent-R2 achieves superior performance using end-to-end GRPO with step-level verification rewards alone.
• Adaptive Rejection Frequency: At \(Rej_{\max}=2\), the empirical average rejection count is only 0.59 per query. The agent naturally learns to reject only when warranted, rather than mindlessly exhausting its rejection quota.

Highlights & Insights

• Internalizing Verification as Native Agent Actions: Rather than relying on rigid post-hoc text filters, MMAgent-R2 empowers the VLM to perform direct visual comparisons over image batches, bridging the gap between global representation retrieval and fine-grained visual reasoning.
• Dynamic Candidate Expansion: Consecutive retrieval queries for the same entity stream non-overlapping candidate batches. Coupled with active rejection, this completely dissolves the error-propagation bottleneck of static candidate pools.
• Step-Level Dense Credit Assignment: Providing explicit intermediate rewards for correct rejection (\(\delta=0.2\)) and correct reranking (\(1.0\)) guides policy discovery far more effectively than sparse downstream task accuracy alone.

Limitations & Future Work

• Upper Bound Imposed by Retriever Recall: As shown in Table 4, the R@15 ceiling remains 37.8% on E-VQA and 80.2% on InfoSeek. Entities absent from all 15 recalled candidates cannot be recovered. Joint end-to-end training of the visual retriever alongside the agent policy represents a vital next step.
• Token Overhead from Multi-Image Batches: Injecting batches of candidate images across multiple turns incurs substantial token overhead and computational latency. Developing visual token compression or compact semantic summarization for candidates is an important future direction.

Related Work & Insights

• vs. Classical Post-Processing mRAG (e.g., EchoSight, mR2AG): Traditional methods rely on multimodal LLMs to score associated textual passages from a static candidate pool. MMAgent-R2 performs direct cross-image visual comparison and can actively expand candidates via rejection.
• vs. Multi-Stage Pipeline Baselines (e.g., ReAG, VLM-PRF): Frameworks like ReAG rely on separately trained discriminator critic models and staged pipeline warmups. MMAgent-R2 integrates retrieval, verification, and answering inside a unified policy trained end-to-end via GRPO.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Pioneers the unification of visual reranking and active rejection as native agent actions with step-level verification RL]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Comprehensive evaluations across E-VQA, InfoSeek, and MMhops with extensive identification and parameter ablations]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical framing, lucid narrative progression, and informative qualitative illustrations]
• Value: ⭐⭐⭐⭐⭐ [Provides a robust, generalizable paradigm for robust multimodal RAG in open-domain knowledge-intensive settings]

Related Papers

• [ACL 2026] Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learning
• [ICML 2026] Graph-R1: Towards Agentic GraphRAG Framework via End-to-end Reinforcement Learning
• [ECCV 2026] What Images Cannot Say: Language-Guided Olfactory Representation Learning
• [ICLR 2026] HiPRAG: Hierarchical Process Rewards for Efficient Agentic Retrieval Augmented Generation
• [ACL 2026] Learning to Extract Rational Evidence via Reinforcement Learning for Retrieval-Augmented Generation