MCite-RL: Towards Reliable Multimodal RAG via Citation-enhanced Agentic Reinforcement Learning¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: Multimodal RAG, Visual Citation, Agentic Reinforcement Learning, Process-Level Reward, Fine-Grained Evidence Grounding
TL;DR¶
MCite-RL establishes a citation-enhanced agentic reinforcement learning framework for multimodal RAG, which resolves spatial citation drift and answer-citation decoupling by transforming static prediction into a dynamic "retrieval-recursive cropping-joint answer & citation" trajectory optimized via multi-tier rewards (process-level IoGT and outcome-level IoU) and GRPO.
Background & Motivation¶
With the extensive deployment of Multimodal Large Language Models (MLLMs) in complex document comprehension, financial reporting analysis, and visually rich chart question answering, Multimodal Retrieval-Augmented Generation (Multimodal RAG) has emerged as a cornerstone architecture for curbing hallucinations and ensuring the traceability and verifiability of model outputs. While traditional text-based RAG directly attaches textual spans or document chunks as citations, multimodal environments demand that models anchor their generated statements to precise visual regionsโsuch as a specific cell in an extensive financial spreadsheet, an inflection point on a graph, or an isolated paragraph within a dense multi-column layout. Although initial benchmarks like VISA and FinRAGBench-V have pioneered visual source attribution, most existing techniques rely on vanilla single-step generation or superficial supervised fine-tuning (SFT), exhibiting severe fragility when confronted with information-dense real-world document images.
In practice, this fragility manifests as two systematic modes of failure. The first is severe visual citation imprecision: during single-step inference, models routinely suffer from spatial drift, producing bounding boxes that are either excessively coarse-grained (encapsulating entire pages or giant tables) or excessively fine-grained (cropping out essential headers or row labels required to interpret the values), thus failing to provide a verifiable evidential basis. The second failure mode is the decoupling between visual citations and generated answers: a model may accurately generate a target numeric fact, yet its cited bounding box points to an entirely different visual region or an unrelated context. This "answering correctly while citing wrongly" phenomenon fundamentally undermines user trust, stemming directly from the lack of a dynamic reasoning mechanism capable of progressively narrowing the search space while preserving evidence consistency.
To overcome these intertwined challenges, static single-step prompting and coarse-grained task feedback are inadequate to provide the necessary supervisory gradients. Core idea: reformulate multimodal visual citation as a dynamic agentic trajectory of iterative retrieval and recursive cropping, and design a citation-enhanced reward combining process-level (IoGT) and outcome-level (IoU) feedback to jointly optimize answer correctness and source traceability under GRPO reinforcement learning.
Method¶
Overall Architecture¶
MCite-RL reframes standard black-box multimodal QA and bounding-box detection into a sequential decision-making trajectory \(\tau = \{(s_0, a_0), (s_1, a_1), \dots, (s_T, a_T)\}\). Given a user query \(q\) and a document image corpus, the agent interacts across three distinct actions: image retrieval (search), visual recursive cropping (crop), and joint generation of the textual answer with bounding box visual citations (answer & citation). To address the cold-start challenge in multimodal agentic reinforcement learning, MCite-RL introduces a disciplined two-stage pipeline: first, high-quality agentic reasoning trajectories synthesized by a multimodal teacher model are strictly filtered and utilized for SFT cold-start; subsequently, Group Relative Policy Optimization (GRPO) samples diverse rollouts and updates the policy using a multi-component citation-enhanced reward.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["User Multimodal Query q and Document Image Corpus"] --> B["Quality-Controlled Trajectory Synthesis & SFT Cold-Start<br/>Teacher rollouts filtered via 4 strict criteria at 8.6% acceptance"]
B --> C["Agentic Recursive Cropping & Evidence Refinement<br/>Iterative retrieval and coarse-to-fine visual cropping"]
C --> D["Joint Answer and Citation Generation in Shared Context<br/>Terminal visual observation produces textual answer and bounding box"]
D --> E["Process- and Outcome-Level Citation-Enhanced Rewards<br/>Hybrid reward combining Retrieval NDCG, Answer EM, IoGT, and IoU"]
E --> F["GRPO Policy Optimization Update<br/>Group relative advantage normalization for answer and citation joint gain"]
Key Designs¶
1. Agentic Recursive Cropping & Evidence Refinement: progressively narrowing the search space against single-step spatial drift
To combat spatial drift where models fail to localize bounding boxes on full-resolution cluttered documents, MCite-RL decomposes visual evidence localization into an iterative multi-turn reasoning chain. At each interaction step \(t\), the agent evaluates its historical trajectory \(h_t\) and current visual observation \(v_{c_t}\). If the evidence remains incomplete or the field of view is overly expansive, the model invokes a <bbox> cropping action, progressing hierarchically through large semantic regions (e.g., entire sections or tables), intermediate candidate areas, and near-final evidence regions. This coarse-to-fine visual pruning not only mitigates visual attention dispersion across dense documents but also progressively discards visual noise, allowing the true supporting evidence to emerge naturally from an iterative reasoning loop.
2. Joint Answer and Citation Generation in Shared Context: eliminating answer-citation decoupling at the architectural root
To eliminate the disconnect where models produce correct factual answers while citing irrelevant visual fragments, MCite-RL avoids detached post-hoc citation detection modules. Instead, the textual answer \(\hat{y}\) and its supporting visual citation \(\hat{c}_{final}\) are generated jointly conditioned strictly upon the final converged visual observation \(v_{c_T}\): $\((\hat{y}, \hat{c}_{final}) = \arg\max_{y, c_{final}} P(y, c_{final} \mid v_{c_T}, q)\)$ Because both the linguistic response and visual spatial coordinates are derived directly from the exact same local high-resolution visual context, cross-modal attention is enforced between generated textual entities and their corresponding visual patches. This structural coupling guarantees mutual grounding, eliminating hallucinations where answers and citations contradict each other.
3. Process- and Outcome-Level Citation-Enhanced Rewards: dual-anchor supervision for reinforcement learning
Existing agentic RAG frameworks (such as VRAG-RL) predominantly optimize final answer accuracy, leaving visual localization as an unguided secondary byproduct. MCite-RL introduces an explicit multi-tiered citation reward function: $\(R(\tau) = I_v(\tau) \cdot \left(\alpha R_{ret} + \beta R_{ans} + \gamma_1 R_{cit}^{out} + \gamma_2 R_{cit}^{proc}\right)\)$ where \(I_v(\tau) \in \{0, 1\}\) acts as a strict format validation penalty. Crucially, the process-level reward evaluates the intermediate cropping quality using Intersection-over-Ground-truth: \(R_{cit}^{proc} = \text{IoGT}(c_T, c^*) = \frac{|c_T \cap c^*|}{|c^*|}\), heavily penalizing any premature cropping that prunes away critical ground-truth evidence and guaranteeing high-recall evidence preservation. Simultaneously, the outcome-level reward measures the precision of the final predicted bounding box using Intersection-over-Union: \(R_{cit}^{out} = \text{IoU}(c_{final}, c^*) = \frac{|c_{final} \cap c^*|}{|c_{final} \cup c^*|}\), enforcing strict spatial compactness and accuracy. This dual-anchor configuration prevents the model from gaming individual metrics (such as generating oversized boxes to maximize recall or over-cropping early).
A Worked Example¶
Consider a query from Wiki-VISA asking who played the character Preacher in the movie Uncle Drew:
1. Initial Search: The model outputs <search>Uncle Drew Preacher actor</search>, and the environment returns the full-page Wikipedia screenshot.
2. Coarse Crop: Observing the dense page layout, the model recognizes that cast members reside in the lower section and issues a coarse crop <bbox>[22, 895, 531, 1119]</bbox>. This crop encompasses the target table section, satisfying \(\text{IoGT} = 1.0\).
3. Fine Crop: Within the coarse cropped image, the model localizes the specific row discussing Preacher, generating <bbox>[6, 257, 1164, 412]</bbox>.
4. Final Response: In the refined visual field containing clear text "Chris Webber as Preacher", the model simultaneously outputs <answer>Chris Webber</answer> and exact citation coordinates <citation>[9, 123, 2117, 185]</citation>, securing full Exact Match and high IoU.
Loss & Training¶
During the SFT cold-start stage, 28,175 candidate trajectories are synthesized by a multimodal teacher and filtered against four strict criteria: non-zero retrieval NDCG, exact match answer (\(EM = 1\)), perfect citation IoU (\(\text{IoU} = 1\)), and an initial crop that is smaller than the full image while covering the entire ground-truth (\(\text{IoGT} = 1\)). Only 2,417 trajectories (8.6% acceptance rate) are retained to fine-tune the backbone. In the RL stage, the policy \(\pi_\theta\) is trained using GRPO over 1,000 samples, calculating advantages via group normalization: $\(\hat{A}_i = \frac{R(\tau_i) - \text{mean}(\{R(\tau_j)\}_{j=1}^G)}{\text{std}(\{R(\tau_j)\}_{j=1}^G)}\)$ Reward weights are configured as: retrieval reward \(R_{ret}\) with \(\alpha = 0.15\) (nDCG), answer reward \(R_{ans}\) with \(\beta = 0.40\) (Exact Match), outcome citation reward \(R_{cit}^{out}\) with \(\gamma_1 = 0.35\) (IoU), and process citation reward \(R_{cit}^{proc}\) with \(\gamma_2 = 0.10\) (IoGT).
Key Experimental Results¶
Main Results¶
Evaluation across open-domain documents (Wiki-VISA), dense financial charts (FinRAGBench-V), and long-context documents (MMLongBench-Doc) demonstrates significant gains in both Answer Accuracy (Ans.) and Citation Precision (Cit., measured in IoU %):
| Method | Wiki-VISA Ans. (%) |
Wiki-VISA Cit. (%) |
FinRAGBench-V Ans. (%) |
FinRAGBench-V Cit. (%) |
MMLongBench-Doc Ans. (%) |
MMLongBench-Doc Cit. (%) |
|---|---|---|---|---|---|---|
| Proprietary Models (Zero-Shot) | ||||||
| GPT-4o | 31.40 | 3.40 | 21.23 | 19.91 | 25.51 | 10.80 |
| Gemini-2.0-Flash | 39.60 | 0.08 | 20.44 | 6.31 | 25.56 | 2.83 |
| Claude-3.5-Sonnet | 12.80 | 0.53 | 34.58 | 15.49 | 30.35 | 16.97 |
| Qwen2.5-VL-3B-Instruct | ||||||
| Vanilla RAG | 41.20 | 0.46 | 15.21 | 0.33 | 27.77 | 2.26 |
| ReAct | 14.80 | 0.00 | 4.59 | 0.72 | 5.42 | 0.27 |
| Search-R1 | 7.80 | 1.69 | 1.43 | 1.09 | 3.32 | 3.36 |
| VRAG-RL (w/ SFT) | 48.80 | 27.97 | 10.76 | 25.16 | 24.72 | 19.03 |
| MCite-RL (3B) | 51.60 | 33.45 | 13.27 | 27.04 | 27.64 | 22.22 |
| Qwen2.5-VL-7B-Instruct | ||||||
| Vanilla RAG | 40.80 | 1.62 | 17.29 | 6.78 | 31.74 | 12.91 |
| ReAct | 17.80 | 2.30 | 12.98 | 15.29 | 10.21 | 12.07 |
| Search-R1 | 48.40 | 1.30 | 10.83 | 12.77 | 29.59 | 13.14 |
| VRAG-RL (w/ SFT) | 54.20 | 20.56 | 17.65 | 15.40 | 31.58 | 12.97 |
| MCite-RL (7B) | 60.00 | 36.05 | 21.81 | 37.22 | 35.04 | 30.12 |
Ablation Study¶
Component-wise ablation of the reward function and training stages on Qwen2.5-VL-7B-Instruct using Wiki-VISA (Table 2 in the paper):
| Configuration | Ans. (%) | Cit. (%) | Empirical Analysis / Mechanism |
|---|---|---|---|
| MCite-RL (Full) | 60.00 | 36.05 | Full model with dual-tier citation rewards |
| w/o Process-level Cit. (\(R_{cit}^{proc}\)) | 57.60 | 34.50 | Omitting IoGT leads to intermediate evidence clipping |
| w/o Outcome-level Cit. (\(R_{cit}^{out}\)) | 54.00 | 13.15 | Without final IoU anchor, model predicts overly expansive boxes (-22.90%) |
| w/o All Cit. Rewards (VRAG-RL setup) | 54.20 | 20.56 | Ans. drops by 5.80% and Cit. drops by 15.49% without explicit grounding |
| w/o Ans. Reward (\(R_{ans}\)) | 53.20 | 41.65 | Over-optimizes spatial overlap at the cost of linguistic semantics |
| w/o Search Reward (\(R_{ret}\)) | 54.80 | 35.95 | Degrades multi-step retrieval efficiency without early-turn incentives |
| w/o Format Constraints (\(I_v\)) | 55.20 | 33.41 | Unstructured rollouts dilute reinforcement learning policy optimization |
| w/o RL Stage (SFT only) | 45.80 | 18.66 | Pure behavioral cloning lacks exploration; performance lags significantly |
| w/o Training (Base Model) | 9.60 | 0.04 | Off-the-shelf base MLLMs lack innate visual citation capabilities |
Key Findings¶
- Visual citation precision directly drives answer correctness: When removing all citation rewards, answer accuracy falls from 60.00% to 54.20%. Explicit visual localization forces the model to base answers on factual visual evidence rather than hallucinating from parametric memory.
- Outcome IoU anchors precision while process IoGT preserves recall: The outcome-level reward provides the strongest localization signal (a 22.90% drop when omitted), whereas the process-level reward prevents the model from cutting off boundary context during multi-step zoom-in.
- Lower entropy indicates decreased policy uncertainty: Policy entropy curves reveal that MCite-RL converges to a significantly lower entropy level than VRAG-RL, demonstrating that citation rewards act as an effective structural regularizer narrowing the search space.
- Semantic answer-citation consistency surges: Evaluated via an LLM-as-a-judge protocol assessing whether citations provide sufficient, non-redundant evidence, MCite-RL 7B achieves 44.80% on Wiki-VISA, 44.83% on FinRAGBench-V, and 49.26% on MMLongBench-Doc, vastly outperforming VRAG-RL (22.80%, 12.43%, and 18.73%).
Highlights & Insights¶
- From passive bounding-box tagging to proactive evidence search: Conventional pipelines treat citation as an optional post-hoc annotation. MCite-RL transforms citation into the core upstream driver of reasoning, mimicking human visual cognitionโscan overall document, zoom in on key region, and extract factual answers.
- IoGT and IoU dual-objective synergy: Employing Intersection-over-Ground-truth (IoGT) during intermediate steps allows exploratory tolerance for large boxes, while final Intersection-over-Union (IoU) enforces pinpoint precision, elegantly resolving the conflict between intermediate recall and terminal compactness.
- Efficient actor-only RL alignment via GRPO: Combining an 8.6% acceptance rate cold-start dataset with critic-free GRPO establishes a lightweight, highly reproducible pipeline for training grounded vision-language reasoning agents.
Limitations & Future Work¶
- Single-image visual context constraint: The current formulation focuses on recursive cropping within single document images, leaving multi-page cross-document visual reasoning (e.g., cross-referencing multi-year balance sheets across pages) for future exploration.
- Rigid rectangular bounding-box representation: Rectangular boxes struggle to tightly bound irregular or non-contiguous evidence, such as curved textual flows, diagonal chart annotations, or multi-column text wrap-arounds.
- Rollout computation overhead: Multi-step rollouts involving dynamic search and repeated image re-cropping incur non-trivial inference latency and GPU memory requirements during reinforcement learning exploration.
Related Work & Insights¶
- vs VISA / FinRAGBench-V: VISA introduced visual source attribution but relied on static supervised fine-tuning; MCite-RL incorporates dynamic agentic actions and multi-level RL rewards to prevent spatial drift and answer decoupling.
- vs VRAG-RL / Search-R1: While VRAG-RL applied RL to visual cropping, its reward was tied exclusively to answer correctness, treating citations as an implicit side-effect; MCite-RL directly rewards citation precision, yielding simultaneous double-digit gains in both evidence localization and answer quality.
- vs Vanilla RAG / ReAct: Standard RAG without iterative visual zooming suffers from resolution loss and attention dispersion on dense documents, where proprietary leaders like GPT-4o achieve a dismal 3.40% citation precision.
Rating¶
- Novelty: โญโญโญโญโญ (Pioneering integration of process- and outcome-level reinforcement learning rewards for visual citation in multimodal RAG)
- Experimental Thoroughness: โญโญโญโญโญ (Evaluated across open-domain, financial, and long-form document benchmarks against closed- and open-source models with comprehensive ablations)
- Writing Quality: โญโญโญโญโญ (Clear problem definition, rigorous mathematical formulation, and intuitive empirical case studies)
- Value: โญโญโญโญโญ (Provides an actionable and reliable paradigm for verifiable, hallucination-resistant multimodal document agents)