Skip to content

Reflect to Inform: Boosting Multimodal Reasoning via Autonomous Visual Reflection

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/Xiaobu-USTC/VRE
Area: Multimodal VLM
Keywords: Visual Reflection, Multimodal Large Language Models, Visual Drift, Information Gain, Reinforcement Learning

TL;DR

Addressing the "visual drift" phenomenon where MLLMs progressively detach from visual inputs and succumb to textual hallucinations during extended reasoning, Visual Re-Examination (VRE) activates the model's latent implicit re-attention without external tools or extra visual inputs. Through difficulty-aware data curation, information-gain-constrained homologous reflection synthesis, and a closed-loop SFT–RLVR–Post-RL-SFT training pipeline, VRE substantially elevates multimodal reasoning accuracy and perceptual fidelity across diverse benchmarks.

Background & Motivation

Multimodal Large Language Models (MLLMs) have demonstrated impressive breakthroughs across multimodal understanding and complex reasoning tasks. Nevertheless, when deployed in long-form generation and extended chain-of-thought (CoT) reasoning, they consistently suffer from a pervasive failure mode known as "visual drift": as the generated sequence lengthens, the model's attention to image features progressively decays, causing it to fall back on internal language priors. Consequently, intermediate reasoning becomes ungrounded, precipitating severe factual errors and hallucinations especially when resolving fine-grained spatial relations, deciphering dense text in charts and documents, or following multi-step geometric logic.

Prior literature attempts to mitigate visual drift primarily via two diverging avenues. The first family adopts explicit visual interventions—such as dynamically injecting high-resolution cropped patches, superimposing Set-of-Mark visual prompts, or interleaving token-level visual representations directly into intermediate reasoning steps. While effective at enforcing visual grounding, these external interventions distort the standard autoregressive inference pipeline, depend heavily on complex heuristic engineering, and impose non-trivial computational overhead. Conversely, the second line of research observes that MLLMs inherently harbor a latent capability to re-attend to visual tokens without auxiliary visual inputs. However, this latent introspection remains highly erratic: naive prompt nudges frequently degenerate into empty textual paraphrasing or superficial restatements rather than retrieving actual missing visual evidence.

This fundamental tension underscores that awakening intrinsic visual introspection requires more than structural prompting or brute-force decoding interventions; it requires an evidence-driven supervisory mechanism grounded in tangible information gain. Core idea: Propose Visual Re-Examination (VRE), a self-iterative framework that leverages Reflection Information Gain to transform sporadic re-attention into a purposeful, learnable policy through a closed-loop pipeline spanning difficulty-aware homologous SFT, reinforcement learning with verifiable rewards (RLVR), and post-RL self-distillation.

Method

Overall Architecture

The objective of VRE is to institutionalize purposeful visual re-examination within the standard autoregressive generation pipeline, avoiding architectural modifications, external visual encoders, or auxiliary tools. Given a multimodal input comprising an image and a question, the model produces a unified sequence consisting of an initial reasoning trace, a structured <reflection> introspection block, and a definitive <answer> segment.

The overall framework operates through a closed-loop evolutionary pipeline organized into two primary phases: "Reflection Data Construction" and "Model Optimization". In the data construction phase, the base model evaluates sample difficulty by sampling rollout pass rates to isolate unsettled boundary samples, subsequently synthesizing information-gain-constrained reflection traces (categorized into elaborative, corrective, and recheck reflections) via homologous reconstruction for cold-start SFT. In the optimization phase, the cold-start model is continually refined through Group Relative Policy Optimization (GRPO) using a multi-component reward scheme (format integrity, answer accuracy, and reflection information gain). Finally, the policy revisits previously intractable samples via rejection sampling to execute Post-RL SFT, stabilizing the acquired policy and preventing reward hacking and perceptual degradation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Input<br/>Image + Query"] --> B["Difficulty-Aware Sample Partitioning<br/>Rollout pass rates isolate cognitive boundary"]
    B --> C["Information-Gain Homologous Reflection Synthesis<br/>Elaborative / Corrective / Recheck reflection注入"]
    C --> D["Cold-Start Supervised Fine-Tuning<br/>SFT establishes reflection format and visual priors"]
    D --> E["Closed-Loop RLVR via GRPO<br/>Joint rewards: Format + Accuracy + Reflection Gain"]
    E --> F["Post-RL Self-Distillation SFT<br/>Revisiting hard prompts to curb reward hacking"]
    F --> G["Autonomous Visual Reflection Output<br/>Initial reasoning → Visual re-examination → Grounded answer"]

Key Designs

1. Difficulty-Aware Sample Partitioning: Pinpointing the model's volatile cognitive boundary Blindly applying reflection imitation across entire training corpora is suboptimal: trivial instances do not necessitate visual re-examination, whereas intractable queries exceed the model's foundational reasoning capacity and induce noisy hallucinations. To resolve this dilemma, VRE probes the model's intrinsic competence on each multimodal sample \(s=(i, q, a_s)\) by drawing \(N=8\) independent rollout trajectories using base policy \(\pi_\theta\) under non-zero temperature, calculating empirical pass rate: $\(P_{\text{pass}}(s) = \frac{1}{N} \sum_{j=1}^N \mathbb{I}(\tau^{(j)} \text{ is correct})\)$ The training corpus is thereby partitioned into three distinct regimes: stable samples (\(P_{\text{pass}}=1\)), intractable samples (\(P_{\text{pass}}=0\)), and unstable boundary samples (\(0 < P_{\text{pass}} < 1\)). The unstable subset constitutes the high-leverage supervision signal: it exposes queries where the model possesses latent perceptual and deductive competence but fails to consistently activate visual verification during extended decoding, making it the ideal testbed for reflection synthesis.

2. Information-Gain Homologous Reflection Synthesis: Eliminating empty textual paraphrasing Unconstrained reflection generation routinely degenerates into redundant self-paraphrasing or superficial semantic loops that introduce zero visual value. To enforce genuine evidence retrieval, VRE defines Reflection Information Gain over each reflection block \(r_k\) via indicator \(I_k \in \{0, 1\}\): \(I_k = 1\) if and only if \(r_k\) re-examines the image and extracts novel, question-critical visual evidence absent from prior reasoning; tautologies, unsupported guess modifications, or irrelevant background descriptions yield \(I_k = 0\). The overall reflection reward is: $\(s_{\text{refl}}(\tau) = \sum_{k=1}^K I_k\)$ Crucially, VRE mandates "homologous reconstruction" instead of distilling reflection rationales from stronger proprietary teachers (such as GPT-5). By concatenating the model's own initial reasoning trace \(\tau_{\text{init}}\) with the original input \(s=(i, q)\) and feeding it back into the same policy model, the synthesized reflection remains fully aligned with the native feature manifold. Three targeted reflection types are synthesized: "Elaborative Reflection" reinforces fine-grained background and attribute grounding when the initial path is correct; "Corrective Reflection" pinpoints visual misinterpretations and forcibly re-grounds the inference on the true visual locus; and "Recheck Reflection" conducts a stringent visual verification on critical coordinates, numbers, or text strings.

3. Closed-Loop RLVR and Post-RL Self-Distillation: Expanding cognitive ceilings while curbing reward hacking While cold-start SFT instills structural reflection priors, pure behavioral cloning cannot rectify deep logical errors; conversely, unconstrained reinforcement learning inevitably precipitates reward hacking, sequence length explosion, and catastrophic forgetting of core visual faculties. To reconcile exploration and stability, VRE establishes a dual-stage evolution loop. During the RLVR phase, the policy samples \(G=10\) rollouts per prompt under GRPO and optimizes a multi-objective reward: $\(R(\tau_i) = \lambda_{\text{form}} R_{\text{form}} + \lambda_{\text{acc}} R_{\text{acc}} + \lambda_{\text{refl}} R_{\text{refl}}\)$ Here, \(R_{\text{form}} \in \{0, -1, -2\}\) penalizes tag formatting errors, sequence length exceeding 1000 characters, or empty content; \(R_{\text{acc}} \in \{0, 1\}\) evaluates final answer accuracy via an LLM judge; and \(R_{\text{refl}} = \mathbb{I}(s_{\text{refl}} > 0 \land \tau_{\text{refl}} \text{ exists})\) strictly rewards validated visual information gain. Upon reaching peak policy performance at ~100 steps, RL training is deliberately halted. The upgraded policy then revisits previously intractable samples, harvesting correct trajectories via rejection sampling to construct an augmented distillation corpus for Post-RL SFT, thereby smoothly consolidating hard-prompt breakthroughs into stable parametric memory.

Loss & Training

During the Cold-Start SFT phase, the base model is optimized over the curated homologous reflection dataset \(\mathcal{D}_{\text{SFT}}\) via standard cross-entropy: $\(\mathcal{L}_{\text{cold-start}}(\theta) = -\mathbb{E}_{(i, q, \tau) \sim \mathcal{D}_{\text{SFT}}} \left[ \sum_{t=1}^{|\tau|} \log \pi_\theta(y_t \mid i, q, y_{<t}) \right]\)$ Hyperparameters for Cold-Start SFT specify a global batch size of 64, learning rate of \(1\times 10^{-5}\) governed by a cosine scheduler over 3 epochs, and maximum sequence length of 4900. In the GRPO reinforcement learning phase, discount factor \(\gamma=1.0\), generalized advantage estimation parameter \(\lambda=0.95\), initial KL penalty coefficient is fixed at 0.01, and clipping threshold is 0.2. Reward weights are set to \(\lambda_{\text{form}}=0.4\), \(\lambda_{\text{acc}}=0.6\), and \(\lambda_{\text{refl}}=0.4\), with rollout sampling temperature at 1.0 and maximum output length at 7168 tokens. In the Post-RL SFT stage, training proceeds over ~12k curated hard-prompt trajectories for 1 epoch with a reduced learning rate of \(5\times 10^{-7}\) to avoid catastrophic forgetting. The infrastructure leverages 64× Ascend 910B (64G) NPUs utilizing a 3D parallelism scheme with tensor, pipeline, and data parallel degrees configured to 4, 4, and 4.

Key Experimental Results

Main Results

VRE was evaluated across mathematical reasoning, high-resolution visual perception, document understanding, and multidisciplinary domains, consistently outperforming general-purpose open-source MLLMs and tool-augmented reasoning baselines.

Model Tool Use Param Size MathVista MathVerse MathVision WeMath LogicVista MMMU V*-Bench Overall ChartQA Overall OCRBench_v2 (en/zh) RealworldQA
GPT-4o ✗ - 63.8 50.8 30.4 69.0 53.2 69.1 79.5 86.7 46.5 / 32.2 75.5
Claude 3.7 ✗ - 66.8 52.0 41.3 72.6 - - - - - -
InternVL2.5 ✗ 8B 64.4 39.5 22.0 - - 56.0 - 84.8 49.8 / 52.1 70.1
Qwen2.5-VL ✗ 7B 68.2 49.2 23.7 62.1 45.9 51.2 76.4 83.7 56.3 / 57.2 68.2
DeepEyesV2 ✓ 7B 71.9 52.7 28.9 38.1 48.7 - 81.8 88.4 - -
Thyme ✓ 7B 70.0 39.1 27.6 39.3 49.0 - 82.2 86.1 - 70.2
Semantic-back ✗ 7B 71.6 50.5 27.7 71.3 - 52.0 79.6 - - -
VRE (Ours, Qwen2.5-VL) ✗ 7B 71.2 53.1 26.5 68.7 48.7 52.1 83.8 88.8 62.6 / 64.7 69.8
Gain vs Qwen2.5-VL-7B - - +3.0 +3.9 +2.8 +6.6 +2.8 +0.9 +7.4 +5.1 +6.3 / +7.5 +1.6
Qwen3-VL ✗ 8B 77.2 62.1 53.9 73.5 55.3 69.6 90.1 89.4 65.4 / 61.2 71.5
VRE (Ours, Qwen3-VL) ✗ 8B 81.4 66.7 56.2 78.6 57.9 72.5 92.6 91.7 69.9 / 68.5 73.8
Gain vs Qwen3-VL-8B - - +4.2 +4.6 +2.3 +5.1 +2.6 +2.9 +2.5 +2.3 +4.5 / +7.3 +2.3

Ablation Study

Ablations systematically decouple component contributions, reinforcement learning reward configurations, and causal visual attention dependencies.

Configuration / Setting V*-Bench MathVista MathVerse ChartQA OCRBench_v2 Description / Finding
Base Model (Qwen2.5-VL-7B) 76.4 68.2 49.2 83.7 56.3 / 57.2 Unmodified baseline model
+ SFT Only 80.1 69.2 49.6 87.8 59.6 / 62.3 Establishes format priors; aids perception but leaves deep logic stagnant
+ SFT + RLVR 83.8 71.0 52.5 88.7 62.1 / 64.3 Unlocks active visual verification; MathVerse leaps by +2.9 points
Full VRE (+ Post-RL SFT) 84.3 71.2 53.1 88.8 62.6 / 64.7 Consolidates hard prompts into parametric memory and stabilizes policy
Ablation: Without <reflection> tags 83.8 (-0.5) 71.5 (+0.3) 53.0 (-0.1) 88.9 (+0.1) - Tags organize the reflection phase rather than driving raw accuracy
Ablation: No information-gain filter 81.7 (-2.6) 69.5 (-1.7) 50.3 (-2.8) 86.1 (-2.7) - Absence of information gain causes reflection to collapse into paraphrasing
Ablation: No reflection reward (\(\lambda_{\text{refl}}=0\)) 81.7 (-2.6) 69.8 (-1.4) 50.2 (-2.9) 85.6 (-3.2) - Disabling reflection rewards triggers severe textual reward hacking
Ablation: GPT-5-generated reflections 78.0 (-6.3) 70.3 (-0.9) 51.6 (-1.5) 84.5 (-4.3) - Cross-model distillation forces alien reasoning styles, inflating KL divergence
Causal Check: Visual masking during reflection 75.9 (-8.4) 68.3 (-2.9) 49.5 (-3.6) 83.2 (-5.6) 55.7 / 55.8 (-6.9/-8.9) Confirms gains stem from active visual retrieval rather than lengthened text

Key Findings

  • Reflection Reward is Essential to Preempt Reward Hacking: Under matched compute and query exposures, removing the reflection reward (\(\lambda_{\text{refl}}=0\)) precipitates a 2.6-point drop on V*-Bench and a 3.2-point drop on ChartQA. Without explicit information-gain supervision, the RL policy rapidly learns to exploit the accuracy judge by generating verbose, hollow self-paraphrasing tokens that ignore the image.
  • Causal Intervention Confirms Authentic Visual Resurgence: Blocking attention to visual tokens during the <reflection> block causes severe performance collapse across all benchmarks (-8.4 on V*-Bench, -6.9/-8.9 on OCRBench_v2). This causally disproves the hypothesis that gains stem merely from extra test-time computational tokens, proving that visual evidence re-acquisition is the definitive mechanism.
  • Superiority of Homologous Reconstruction Over External Distillation: Replacing self-generated reflections with GPT-5 traces reduces V*-Bench performance by 6.3 points and ChartQA by 4.3 points. Cross-model distillation forces the 7B student to imitate alien syntactic patterns, driving token-level KL divergence significantly higher compared to native homologous rollouts.
  • Cognitive Boundary Breakthrough: On the challenging subset of "All Wrong" prompts where the cold-start model completely failed across all rollouts, Post-RL SFT successfully generates at least one correct solution in 68.1% of cases (58.9% mixed + 9.2% fully mastered), proving that VRE genuinely expands the solvable problem envelope.

Highlights & Insights

  • Tool-Free, Native Visual Re-grounding: VRE circumvents the engineering complexity of external patch croppers, visual tool agents, and architectural changes, demonstrating that standard autoregressive MLLMs possess sufficient latent capacity to independently refocus on visual evidence when properly incentivized.
  • Information Gain as a Formal Objective: By formalizing Reflection Information Gain as an explicit indicator metric and confirming it via conditional entropy analysis (1.08 for VRE vs. 0.66 for Semantic-back), the work establishes a rigorous theoretical and empirical boundary separating authentic visual introspection from empty verbal repetition.
  • Three-Stage Anti-Hacking Stabilization: The empirical discovery that MLLM RLVR collapses after ~100–150 steps due to textual reward hacking—and the subsequent remedy of using Post-RL SFT to distill dynamic gains into stable weights—offers a robust, practical blueprint for scaling reinforcement learning in vision-language models.

Limitations & Future Work

  • Vulnerability to Prolonged RL Optimization: Step scaling analysis reveals that continuing GRPO past 150 steps induces substantial capability collapse as the policy exploits the reward judge. Current mitigation relies on early termination at step 100 followed by SFT; theoretically grounded reward regularizers and automated adaptive early-stopping mechanisms remain necessary future directions.
  • Modest Uplift on Pure Symbolic Math Deduction: While perceptual grounding, fine-grained localization, and document parsing witness dramatic improvements, benchmarks dominated by abstract mathematical knowledge (such as MathVision) exhibit smaller margins, showing that visual re-examination cannot substitute for domain-specific symbolic math corpora.
  • Dependence on High-Capacity LLM Evaluators: The offline curation and RLVR stages require a 72B-scale LLM judge and scorer to reliably assess semantic accuracy and reflection information gain, presenting computational overhead for resource-constrained environments.
  • vs Thyme / DeepEyesV2: These methods depend on explicit agentic tool invocation or high-resolution visual cropping to retrieve details, altering the generation pipeline and increasing inference latency. In contrast, VRE operates purely via implicit visual re-attention within standard autoregression, incurring zero architectural overhead.
  • vs Semantic-back: While Semantic-back demonstrates that models can implicitly look back, it lacks rigorous guidance on what evidence to retrieve, causing reflections to often degenerate into redundant self-paraphrasing (entropy 0.66). VRE introduces strict information gain constraints and structured reward incentives (entropy 1.08), turning sporadic looking into a purposeful policy.
  • vs R1-VL / MM-Eureka: Standard multimodal RLVR frequently falls into pure textual shortcuts, inducing severe visual drift and ungrounded hallucinations during extended CoT reasoning. VRE incorporates reflection information gain as an indispensable regularizer, enforcing persistent visual grounding throughout long-chain deduction.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates an information-gain-driven visual reflection paradigm that converts latent, unstable re-attention into a purposeful, learnable policy.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous validation across 10+ benchmarks with attention visualizations, causal masking interventions, KL divergence analyses, and cognitive boundary statistics.
  • Writing Quality: ⭐⭐⭐⭐⭐ Lucid, cohesive narrative flow that clearly articulates the tension of visual drift and methodically justifies every design choice.
  • Value: ⭐⭐⭐⭐⭐ Provides a tool-free, highly scalable post-training recipe for eliminating multimodal hallucinations in long-form generation.