Skip to content

Reflect to Inform: Boosting Multimodal Reasoning via Information-Gain-Driven Visual Reflection

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Xiaobu-USTC/VRE
Area: Multimodal VLM
Keywords: Multimodal Large Language Models, Visual Drift, Visual Reflection, Reinforcement Learning, Information Gain

TL;DR

To tackle "visual drift"—the progressive tendency of Multimodal Large Language Models (MLLMs) to abandon visual evidence and fall back on textual hallucination during long-form generation—this paper introduces Visual Re-Examination (VRE), a self-iterative framework governed by Reflection Information Gain that activates and stabilizes intrinsic visual re-attention via homologous cold-start reconstruction, GRPO reinforcement learning with structured rewards, and post-RL self-distillation without external teacher models, extra tools, or architectural changes.

Background & Motivation

Multimodal Large Language Models (MLLMs) have demonstrated extraordinary capabilities across visual perception and complex multi-step reasoning tasks. However, during extended, long-form chain-of-thought generation, these systems suffer from a pervasive failure mode known as "visual drift." As generated reasoning traces grow progressively longer, the standard autoregressive decoding mechanism gradually decouples from the foundational image features, falling back on internal textual priors and language inertia. In tasks demanding precise late-stage visual verification—such as resolving subtle spatial relationships, reading fine-grained OCR text, or parsing intricate scientific plots—this drift rapidly degenerates into ungrounded reasoning and severe factual hallucinations.

Prior attempts to mitigate visual drift largely pursue two paradigms. The first paradigm relies on explicit visual interventions, including dynamically cropping and injecting high-resolution sub-patches, superimposing visual bounding boxes, or interleaving new visual feature tokens directly between text reasoning steps. While effective at forcing visual grounding, these external interventions fundamentally modify the standard autoregressive pipeline, require complex heuristic hand-tuning, and incur substantial computational and latency overhead during inference. The second paradigm reveals that MLLMs inherently possess a latent, unactivated capability to implicitly re-attend to the original image without external inputs. Yet, this latent behavior remains highly unstable: naively prompting a model to reflect or self-correct typically collapses into empty textual paraphrasing—altering phrasing or asserting unverified conclusions without extracting genuine visual evidence from the image.

This dilemma exposes the core tension in multimodal self-reflection: MLLMs have the latent capacity to re-inspect images, but lack a principled, learnable supervision signal that defines when to re-examine, where to look, and what constitutes a productive reflection. Without explicit constraints, optimizing models purely against final answer correctness via standard reinforcement learning leads to multimodal reward hacking, where the model fabricates lengthy, plausible-sounding reflection narratives while ignoring the visual inputs entirely. The core idea of this paper is to formulate visual reflection strictly as an evidence-driven visual retrieval process governed by Reflection Information Gain, using homologous reconstruction to elicit structured reflection priors from the model's own distribution, followed by closed-loop GRPO reinforcement learning with format, accuracy, and reflection quality rewards, and consolidating the policy via post-RL self-distillation.

Method

Overall Architecture

The Visual Re-Examination (VRE) framework operates through three interconnected, self-iterative phases: homologous reflection data construction for Cold-Start SFT, multi-reward Group Relative Policy Optimization (RLVR via GRPO), and Post-RL Self-Distillation. Given a multimodal query comprising an image \(i\) and a question \(q\), the policy model learns to execute a structured two-stage inference process: an initial reasoning phase \(\tau_{\text{pre}}\), followed by an explicit evidence-seeking <reflection>...</reflection> block that actively refocuses attention onto relevant image regions, culminating in a verified <answer>...</answer>.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Input<br/>Image i + Question q"] --> B["Homologous Reconstruction & Difficulty Partitioning<br/>Sample rollout traces & filter unstable boundary samples"]
    B --> C["Information-Gain-Constrained Reflection Synthesis<br/>Synthesize Elaborative / Corrective / Recheck reflections"]
    C --> D["Cold-Start SFT<br/>Impart structured two-stage reflection format prior"]
    D --> E["Closed-Loop RLVR via GRPO<br/>Joint optimization with Format, Accuracy, and Reflection rewards"]
    E --> F["Post-RL Self-Distillation<br/>Rejection sampling on hard prompts to consolidate policy"]
    F --> G["Final Self-Evolved VRE Policy<br/>Autonomous visual re-inspection without external tools"]

Key Designs

1. Homologous Reconstruction & Information-Gain Synthesis: Preventing Distribution Shift and Empty Textual Paraphrasing

Distilling reasoning or reflection traces from stronger proprietary models (such as GPT-4o or GPT-5) forces the student MLLM to mimic alien reasoning structures, causing a severe distribution mismatch on the visual-textual manifold. Furthermore, unconstrained reflections frequently degenerate into redundant textual rewrites. To resolve this, VRE introduces Homologous Reconstruction. Given a multimodal training instance \(s = (i, q, a_s)\), the base policy \(\pi_\theta\) generates \(N=8\) stochastic rollouts at temperature \(T=1.0\) to compute its pass rate \(P_{\text{pass}}(s) = \frac{1}{N}\sum_{j=1}^N \mathbb{I}(\tau_j \text{ is correct})\). The training pool is cleanly partitioned: stable samples (\(P_{\text{pass}}=1\)) and intractable samples (\(P_{\text{pass}}=0\)) are set aside, focusing SFT exclusively on unstable boundary samples (\(0 < P_{\text{pass}} \le 1\)) where the latent re-examination ability exists but is inconsistently triggered.

To synthesize the <reflection> block, the first-stage reasoning trajectory \(\tau_{\text{pre}}\) is concatenated with the original input \(s=(i, q)\) and fed back into the same policy model for a second rollout. Crucially, the generated reflection is evaluated by Reflection Information Gain: for a sequence of reflection blocks \(r_1, \dots, r_K\), a binary indicator \(I_k \in \{0, 1\}\) equals 1 only if block \(r_k\) re-examines the image and contributes new, question-relevant visual evidence absent from preceding steps. Trajectories are aggregated as \(s_{\text{refl}}(\tau) = \sum_{k=1}^K I_k\). Reflections are categorized into three targeted paradigms: - Elaborative Reflection: Applied when initial reasoning is correct, guiding the model to extract omitted fine-grained attributes, small objects, or background context to reinforce confidence; - Corrective Reflection: Applied when initial reasoning contains visual misinterpretations, prompting the model to pinpoint the exact visual drift point, contrast text against pixels, and rectify reasoning; - Recheck Reflection: Applied to precision-critical tasks (e.g., OCR, charts, coordinate counting), enforcing an explicit audit of numerical values and spatial relations before finalizing the output.

2. Structured Multi-Reward GRPO Optimization: Suppressing Multimodal Reward Hacking

In the RLVR stage, optimizing solely for final answer accuracy encourages the model to game the verification system by outputting verbose textual fluff or guessing without looking at the image. VRE employs Group Relative Policy Optimization (GRPO), sampling a group of \(G=10\) trajectories \(\{\tau_1, \dots, \tau_G\}\) per query, and formulates a composite reward combining structural integrity, answer correctness, and reflection quality:

\[R(\tau_i) = \lambda_{\text{form}} R_{\text{form}}(\tau_i) + \lambda_{\text{acc}} R_{\text{acc}}(\tau_i) + \lambda_{\text{refl}} R_{\text{refl}}(\tau_i)\]

with weighting hyper-parameters set to \(\lambda_{\text{form}}=0.4\), \(\lambda_{\text{acc}}=0.6\), and \(\lambda_{\text{refl}}=0.4\). The individual reward mechanisms operate as follows: - Format Reward (\(R_{\text{form}}\)): Imposes a tiered structural penalty. Missing or unbalanced <reflection> or <answer> tags receive a \(-1\) penalty; empty or semantically vacuous reflection blocks receive a \(-2\) penalty; responses exceeding 1000 tokens receive a \(-1\) penalty; structurally compliant outputs receive \(0\). This curbs token bloat and ensures parseable outputs. - Accuracy Reward (\(R_{\text{acc}}\)): Evaluated by an LLM-based judge (Qwen2.5-72B-Instruct) outputting a binary \(0\) or \(1\), tolerating semantically equivalent answer variants (e.g., 1 vs. 1.0) that confound rigid exact-match scripts. - Reflection Reward (\(R_{\text{refl}}\)): Computed as \(R_{\text{refl}} = \mathbb{I}(s_{\text{refl}} > 0 \land \tau_{\text{refl}} \text{ exists})\) via Qwen2.5-72B-Instruct acting as an impartial evaluator (not a rollout teacher). It rigorously verifies whether the reflection extracts tangible new visual facts, penalizing superficial paraphrasing. This scorer achieves 96% agreement with human annotators on 100 audited cases and 95.3% concordance with GPT-5 across 1,000 diverse trajectories.

3. Post-RL Self-Distillation: Salvaging Intractable Prompts and Consolidating the Policy Boundary

Empirical analysis reveals that pushing GRPO beyond 150 optimization steps induces severe over-optimization and capability collapse due to multimodal reward exploitation. To prevent degradation while pushing reasoning frontiers, VRE introduces Post-RL Self-Distillation. The RL policy is halted at its empirical peak (\(\sim 100\) steps) and redeployed to revisit the initially intractable "All Wrong Prompts" (\(P_{\text{pass}}=0\) during cold-start) using iterative sampling.

Equipped with its newly acquired visual re-examination policy, the RL model successfully produces at least one verified correct trajectory in 68.1% of these previously unsolvable cases (58.9% mixed + 9.2% all correct). These salvaged trajectories are curated via rejection sampling into an augmented supervision dataset \(\mathcal{D}_{\text{post}}\). A single epoch of SFT with a conservative learning rate (\(5\times 10^{-7}\)) is conducted. This step smoothly converts the exploratory gains into stable parametric memory, averting gradient shocks and catastrophic forgetting.

Loss & Training

  1. Cold-Start SFT: Standard autoregressive causal language modeling loss \(\mathcal{L}_{\text{cold-start}}\) trained across 8× Ascend 910B (64G) NPUs using LLaMA-Factory. Global batch size is 64, learning rate is \(1\times 10^{-5}\) with cosine schedule, max sequence length is 4900 tokens, trained for 3 epochs.
  2. RLVR Phase: Scaled to 64× Ascend 910B NPUs using the Mindspeed-RL framework with 3D parallelism (tensor=4, pipeline=4, data=4). GRPO hyper-parameters: \(\gamma=1.0, \lambda=0.95\), fixed initial KL coefficient 0.01, clipping ratio 0.2, group size \(G=10\), max response length 7168 tokens, temperature 1.0. Optimization is terminated at 100 steps.
  3. Post-RL SFT: 1 epoch fine-tuning on \(\mathcal{D}_{\text{post}}\) with batch size 64 and learning rate \(5\times 10^{-7}\).

Key Experimental Results

Main Results

VRE was evaluated across mathematical/logical deduction benchmarks (Table 1) and high-resolution perception/document understanding benchmarks (Table 2), instantiated on both Qwen2.5-VL-7B-Instruct and Qwen3-VL-8B-Instruct. Remarkably, VRE operates purely within the model's native autoregressive decoding—without tool use or external image cropping.

Performance on Mathematical and Logical Reasoning Benchmarks:

Model Tool Use Param Size MathVista MathVerse MathVision WeMath LogicVista MMMU
GPT-4o ✗ - 63.8 50.8 30.4 69.0 53.2 69.1
Claude 3.7 ✗ - 66.8 52.0 41.3 72.6 - -
LLaVA-OneVision ✗ 7B 58.6 19.3 18.3 20.9 33.3 48.8
InternVL2.5 ✗ 8B 64.4 39.5 22.0 - - 56.0
InternVL3 ✗ 8B 71.6 39.8 29.3 37.1 - 65.6
DeepEyesV2 ✓ 7B 71.9 52.7 28.9 38.1 48.7 -
Thyme ✓ 7B 70.0 39.1 27.6 39.3 49.0 -
Semantic-back ✗ 7B 71.6 50.5 27.7 71.3 - 52.0
VRE (Ours, Qwen2.5-VL) ✗ 7B 71.2 53.1 26.5 68.7 48.7 52.1
(Gain over base) - - (+3.0) (+3.9) (+2.8) (+6.6) (+2.8) (+0.9)
VRE (Ours, Qwen3-VL) ✗ 8B 81.4 66.7 56.2 78.6 57.9 72.5
(Gain over base) - - (+4.2) (+4.6) (+2.3) (+5.1) (+2.6) (+2.9)

Performance on High-Resolution Perception, Document Understanding, and Real-world QA:

Model Tool Size V*-Attr V*-Spatial V*-Overall ChartQA-Hum ChartQA-Aug ChartQA-All OCRBench-en/zh
GPT-4o ✗ - 60.5 67.5 79.5 91.9 85.7 86.7 46.5 / 32.2
Qwen2.5-VL Base ✗ 7B 78.2 73.6 76.4 72.5 94.9 83.7 56.3 / 57.2
SEAL ✓ 7B 74.8 76.3 75.4 - - - -
DyFo ✓ 7B 80.0 82.9 81.2 - - - -
DeepEyesV2 ✓ 7B - - 81.8 - - 88.4 -
Thyme ✓ 7B 83.5 80.3 82.2 80.0 92.2 86.1 -
VRE (Ours, Qwen2.5-VL) ✗ 7B 84.3 82.9 83.8 87.3 90.4 88.8 62.6 / 64.7
(Gain over base) - - (+6.1) (+9.3) (+7.4) (+14.8) (-4.5) (+5.1) (+6.3 / +7.5)
VRE (Ours, Qwen3-VL) ✗ 8B 95.7 88.2 92.6 86.6 96.8 91.7 69.9 / 68.5
(Gain over base) - - (+1.8) (+4.0) (+2.5) (+3.5) (+1.1) (+2.3) (+4.5 / +7.3)

Ablation Study

To understand the progressive contribution of each framework phase, component ablations were conducted on Qwen2.5-VL-7B-Instruct.

Progression with Incremental Addition of VRE Components:

Setting V*-Bench MathVista MathVerse ChartQA OCRBench_v2 (en/zh) Empirical Role
Base Model 76.4 68.2 49.2 83.7 56.3 / 57.2 Vanilla open-source MLLM
SFT Only 80.1 69.2 49.6 87.8 59.6 / 62.3 Establishes format priors; aids perception but fails on deep logic
SFT + RLVR 83.8 71.0 52.5 88.7 62.1 / 64.3 Reflection rewards catalyze self-correction (+2.9% on MathVerse)
Full VRE 84.3 71.2 53.1 88.8 62.6 / 64.7 Consolidates hard prompt trajectories and maximizes robustness

Component Isolations Under Matched Training Budgets:

Setting Variant V*-Bench ChartQA MathVista MathVerse Mechanism & Diagnostic Takeaway
Full VRE 84.3 88.8 71.2 53.1 Complete pipeline benchmark
Without <reflection> tags 83.8 (-0.5) 88.9 (+0.1) 71.5 (+0.3) 53.0 (-0.1) Structural tags merely organize reasoning rather than generate gains
No information-gain filtering 81.7 (-2.6) 86.1 (-2.7) 69.5 (-1.7) 50.3 (-2.8) Cold-start data becomes contaminated with vacuous paraphrasing
No reflection reward (\(\lambda_{\text{refl}}=0\)) 81.7 (-2.6) 85.6 (-3.2) 69.8 (-1.4) 50.2 (-2.9) Removing reflection reward triggers severe multimodal reward hacking
GPT-5-generated reflections 78.0 (-6.3) 84.5 (-4.3) 70.3 (-0.9) 51.6 (-1.5) Cross-model teacher creates sharp manifold drift and perception drop
Causal visual masking in reflection 75.9 (-8.4) 83.2 (-5.6) 68.3 (-2.9) 49.5 (-3.6) Definitively proves gains stem from visual re-attention, not token length

Key Findings

  • Mechanistic Evidence via Attention Resurgence: Attention heatmaps reveal that while the base model's attention to visual tokens decays monotonically toward zero as reasoning text lengthens, VRE generates a sharp attention resurgence inside <reflection> blocks. Spatially, this shifts attention from dispersed background textures directly onto target objects (e.g., locking onto the exact coordinates of a small dustpan).
  • Evolution of Reflection Paradigms: Across training stages, "No Gain" redundant paraphrasing drops from 11%~19% after cold-start SFT to nearly 0% after Post-RL SFT. On perception-heavy benchmarks (V*, MMMU, MathVerse), "Corrective" reflections surge to 60%~75%, whereas on OCR and charts, "Recheck" and "Elaborative" reflections dominate.
  • Information-Theoretic Verification via Conditional Entropy: To prove genuine information injection, conditional entropy \(H(Y_{\text{refl}} \mid X_{\text{ctx}})\) was measured on V*-Bench. VRE achieves an average entropy of 1.08, compared to 0.66 for Semantic-back. This confirms that VRE's reflections are not predictable from preceding text alone and effectively break textual inertia.
  • Boundaries Breakthrough on Hard Prompts: On the subset of "All Wrong" prompts where the cold-start model completely failed across all rollouts, Post-RL SFT successfully generated at least one correct trajectory in 68.1% of cases (58.9% mixed + 9.2% all correct), demonstrating an authentic expansion of the model's solvable capability boundary.

Highlights & Insights

  • Homologous Reconstruction Solves Cross-Model Manifold Mismatch: Rather than distilling teacher trajectories from GPT-5, VRE forces the model to reflect conditioned on its own prior reasoning trace. KL divergence analysis proves that GPT-5-distilled reflections maintain elevated divergence across the entire generation, while homologous self-reflection preserves distributional affinity, accelerating RL convergence within 50 steps.
  • Information Gain as a Structural Regularizer Against Reward Hacking: By making reflection rewards conditional on genuine, question-relevant visual evidence extraction, VRE halts token inflation (stabilizing average response length at \(\sim 600\) tokens) and prevents models from inventing hallucinations to please verification systems.
  • Causal Visual Masking Closes the Explanatory Loop: Attention heatmaps can be purely correlational. By explicitly blocking visual token access solely during the <reflection> block while holding token counts and parameters identical, benchmark scores plummeted by up to 8.4 points—providing ironclad causal proof that performance gains stem from visual perception rather than extra thinking tokens.

Limitations & Future Work

  • Step-Wise Heuristics for Over-Optimization: Extending GRPO beyond 150 steps leads to rapid performance degradation due to reward overfitting. The current framework circumvents this via early stopping at 100 steps followed by Post-RL SFT, but theoretically grounded reward regularizers and adaptive stopping criteria remain necessary.
  • Sensitivity on Highly Saturated Splits: On Qwen2.5-VL, while ChartQA-Human gained 14.8 points, ChartQA-Aug dropped 4.5 points (from a very high 94.9% baseline), indicating slight sensitivity to synthetic distribution nuances (though resolved on Qwen3-VL with a +1.1 point gain).
  • Domain Knowledge Bound: On MathVision, gains primarily stem from improved visual comprehensiveness (+5.56) and visual accuracy (+7.84). Intrinsic mathematical theorem gaps cannot be resolved purely by looking back at the image without external symbolic tools or domain-specific mathematical training.
  • vs. Explicit Visual Intervention (DeepEyesV2, Thyme, DyFo, Visual Sketchpad): Explicit methods inject cropped high-res image patches, bounding boxes, or sketch tokens during inference. VRE achieves superior results on V*-Bench (83.8% vs. 82.2% for Thyme and 81.8% for DeepEyesV2) without modifying the model architecture, invoking external tools, or adding inference-time vision pipelines.
  • vs. Naive Implicit Refocusing (Semantic-back / Look-back): While Semantic-back showed that MLLMs can re-attend to images, its reflections lack explicit information-gain supervision and often collapse into textual restatement (entropy 0.66 vs. VRE's 1.08). VRE introduces an explicit reward signal to enforce novel visual fact extraction.
  • vs. Strong-to-Weak Distillation (GPT-5 Distilled SFT): Training directly on GPT-5-generated reflection traces resulted in a severe 6.3-point drop on V*-Bench and 4.3-point drop on ChartQA due to distribution shift. VRE's homologous self-generation ensures natural alignment with native representations.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates visual reflection through information gain and homologous reconstruction, cleanly solving visual drift without external tools.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous across two MLLM generations, comprehensive ablation suites, attention heatmaps, KL divergence tracking, conditional entropy calculations, and causal masking interventions.
  • Writing Quality: ⭐⭐⭐⭐⭐ Flawless narrative structure, deep mechanistic insights, clear mathematical definitions, and highly informative visualizations.
  • Value: ⭐⭐⭐⭐⭐ Establishes a highly practical post-training recipe for empowering open-source vision-language models with autonomous, self-corrective visual introspection.