reasoning matters mitigate hallucination in multimodal large reasoning models vi
content_hash: 2253ae5a6c864921 ---# Reasoning Matters: Mitigate Hallucination in Multimodal Large Reasoning Models via Reasoning-Conditioned Preference Optimization
Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/KongJiawei1231/RC-DPO
Area: Multimodal VLM
Keywords: Multimodal Large Reasoning Models, hallucination mitigation, conditioned preference optimization, Chain-of-Thought alignment, Monte Carlo Tree Search
TL;DR¶
Addressing the issue where coarse-grained response-level preference optimization decouples the Chain-of-Thought from final answers and induces severe visual hallucinations in Multimodal Large Reasoning Models, RC-DPO explicitly models the reasoning chain as a condition for answer generation, combining MCTS high-quality reasoning sampling with attention-guided hard negative pruning to achieve fine-grained, answer-supportive CoT alignment and substantial hallucination mitigation across multiple benchmarks.
Background & Motivation¶
Multimodal Large Reasoning Models (MLRMs, such as R1-Onevision and MM-Eureka) extend reasoning capabilities to vision-language scenarios by explicitly generating intermediate Chain-of-Thought (CoT) steps before producing final responses, achieving remarkable progress on complex multimodal tasks including visual mathematics and scientific question answering. However, as reasoning chains extend, models increasingly rely on language priors rather than visual evidence, generating fluent and logically plausible explanations that nonetheless invent non-existent objects, incorrect attributes, or contradictory relations. Unlike conventional MLLMs, hallucinations in MLRMs not only emerge in final answers but also originate within intermediate CoT rationales and propagate downstream.
Existing training-based post-alignment methods (e.g., OPA-DPO, RLAIF-V) typically employ response-level Direct Preference Optimization (DPO), where the CoT and final answer are concatenated as a monolithic sequence and optimized under a single response-level preference signal. Through rigorous analysis of training dynamics and error decomposition, the authors discover that the policy-reference preference gap expands much faster on the final answer segment than on the CoT segment. Furthermore, an answer-only optimization variant (OPA-DPOAns) sharing the identical CoT yields CHAIRS performance comparable to full-response optimization. These observations reveal that standard response-level DPO relies heavily on answer-level shortcuts, leaving intermediate reasoning chains insufficiently supervised and allowing hallucinations to persist in the CoT.
A further conditional probability analysis reveals that final answers remain almost entirely hallucination-free when no hallucinated content appears in the CoT, whereas answer hallucinations surge dramatically once the reasoning chain contains factual errors. Consequently, rectifying the final answer in isolation cannot fundamentally eliminate reasoning-induced visual hallucinations. Core idea: Explicitly formulate the intermediate Chain-of-Thought as a condition for answer generation, contrast the conditional support for the identical preferred answer under different CoT conditions (RC-DPO), and construct fine-grained preference data via MCTS positive exploration coupled with attention-guided hard negative pruning.
Method¶
Overall Architecture¶
RC-DPO advances preference optimization from coarse-grained response-level matching to fine-grained reasoning-chain alignment. The complete methodology comprises three consecutive stages: 1. CoT Format Alignment (SFT Warm-up): Utilizes a curated set of structured demonstrations via LoRA fine-tuning to discipline the model into step-by-step reasoning trajectories partitioned by explicit step delimiters; 2. Reasoning-Enhanced Preference Data Construction: Deploys the SFT-warmed model as an exploration policy to conduct MCTS with an external multimodal verifier for discovering visually grounded positive CoTs, while computing cross-modal image-attention scores on the positive CoT to prune visually salient tokens, creating hard negative CoTs that preserve structural syntax but lack visual grounding; 3. Joint Preference Optimization (RC-DPO Training): Combines response-level DPO with a novel reasoning-conditioned preference loss that fixes the preferred answer and contrasts likelihood ratios across distinct CoT conditions, driving the policy model to derive correct answers strictly upon faithful visual reasoning.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Multimodal Data<br/>Image v + Question q"] --> B["Stage 1: CoT Format Alignment<br/>Few-shot prompt SFT for structured reasoning"]
B --> C["Stage 2: Positive MCTS Sampling<br/>Select / Expand / Simulate / Backpropagate"]
B --> D["Stage 2: Hard Negative CoT Pruning<br/>Image-attention guided salient token pruning"]
C --> E["Stage 3: Reasoning-Conditioned DPO<br/>Fix target answer aw, contrast cw vs cl/c~w"]
D --> E
E --> F["Faithful Output MLRM<br/>Mitigate CoT and answer hallucinations"]
Key Designs¶
1. CoT Format Alignment: Structuring explicit reasoning steps for tree search
To enable effective trajectory search over discrete decision steps, the model is first adapted via few-shot prompting and supervised fine-tuning to produce structured reasoning chains. Each reasoning step is constrained to describe exactly one logical derivation grounded in observable visual clues and must terminate with a special delimiter token <END>. This SFT warm-up converts unconstrained, continuous free-form thoughts into discrete, multi-step tree nodes, allowing the warmed policy \(\pi_{\text{SFT}}\) to expand candidate reasoning steps systematically.
2. Positive MCTS Sampling: Exploring visually grounded reasoning chains Rather than relying on expensive and opaque proprietary APIs like GPT-4V for rewriting, RC-DPO employs an internal Monte Carlo Tree Search framework. Rooted at the input image and question, \(\pi_{\text{SFT}}\) iteratively samples candidate step extensions, balancing high-value path exploitation and unvisited branch exploration via the UCB1 criterion. Upon reaching an unexpanded node, rollout completion generates a full CoT-answer trajectory evaluated by a multimodal verifier \(S(c, a; x)\) checking atomic visual claims against the image. Verifier rewards update the path's visit count \(N(s)\) and value estimate \(Q(s)\), and the trajectory with the highest cumulative reward \((c_w, a_w)\) is selected as the preferred positive sample.
3. Hard Negative CoT Pruning: Decoupling visual grounding via cross-modal attention Standard negative responses from random sampling often exhibit severe syntactic breakdown or blatant semantic degradation, failing to teach the model to distinguish subtle visual misalignments where reasoning remains superficially coherent. RC-DPO computes the cross-modal attention score \(s(c_t)\) for each token \(c_t\) in the preferred CoT \(c_w\) across all selected attention heads and layers towards visual image tokens \(v_k\): $\(s(c_t) = \frac{1}{|L||H|}\sum_{l \in L}\sum_{h \in H}\sum_{k \in I} A_{t,k}^{(l,h)}\)$ By ranking tokens according to visual attention intensity and pruning the top \(r = 20\%\) most salient tokens to yield \(\tilde{c}_w\), the resulting corrupted reasoning chain retains the original linguistic flow and reasoning template while discarding critical visual evidence, establishing a challenging hard-negative condition.
4. Reasoning-Conditioned Preference Optimization: Formulating CoT as answer generation condition To dismantle answer-level shortcuts, RC-DPO factorizes the joint probability of generating a complete response as \(\pi_\theta(c, a | x) = \pi_\theta(c | x)\pi_\theta(a | x, c)\). Given the preferred answer \(a_w\), the preference log-likelihood ratio between a grounded chain \(c_w\) and a flawed chain \(c_l\) (or corrupted chain \(\tilde{c}_w\)) decomposes into CoT preference and answer-support conditional terms. Isolating the answer-support term yields the reasoning-conditioned preference objective: $\(\mathcal{L}_{\text{RC}} = -\log \sigma \left( \beta \log \frac{\pi_\theta(a_w | x, c_w)}{\pi_{\text{ref}}(a_w | x, c_w)} - \beta \log \frac{\pi_\theta(a_w | x, c_l)}{\pi_{\text{ref}}(a_w | x, c_l)} \right)\)$ By holding the target answer identical and contrasting only the CoT conditions against the baseline reference model \(\pi_{\text{ref}}\), this margin explicitly optimizes the policy to assign higher generative likelihood to correct answers when conditioned on faithful CoTs over corrupted ones.
Loss & Training¶
The overall training objective of RC-DPO combines response-level preference optimization with reasoning-conditioned preference learning: $\(\mathcal{L} = \mathcal{L}_{\text{DPO}} + \lambda \mathcal{L}_{\text{RC}}\)$ where \(\mathcal{L}_{\text{DPO}}\) maintains global preference ordering over complete responses \((c_w, a_w) \succ (c_l, a_l)\), while \(\mathcal{L}_{\text{RC}}\) is instantiated separately with ordinarily sampled \(c_l\) and attention-pruned \(\tilde{c}_w\) and summed. The optimization sets \(\beta = 0.1\) and loss balance weight \(\lambda = 0.1\). Training applies LoRA (rank=8, alpha=16) on the LLM backbone while freezing the vision encoder, with a learning rate of \(1\text{e-}6\) for 1 epoch.
Key Experimental Results¶
Main Results¶
The authors evaluate RC-DPO across four representative 7B MLRMs: R1-Onevision-7B, MM-Eureka-7B, ThinkLite-VL-7B, and OpenVLThinker-7B.
Table 1: Performance comparison on Object HalBench (CS: CHAIRS, CI: CHAIRI, lower is better)
| Method | R1-Onevision CS ↓ | R1-Onevision CI ↓ | MM-Eureka CS ↓ | MM-Eureka CI ↓ | ThinkLite-VL CS ↓ | ThinkLite-VL CI ↓ | OpenVLThinker CS ↓ | OpenVLThinker CI ↓ |
|---|---|---|---|---|---|---|---|---|
| Vanilla | 36.7 | 7.6 | 34.0 | 7.5 | 30.7 | 7.5 | 31.7 | 7.3 |
| VCD | 42.3 | 9.4 | 28.7 | 8.2 | 35.0 | 8.2 | 27.0 | 8.1 |
| SID | 43.7 | 9.2 | 35.3 | 8.3 | 34.7 | 8.4 | 35.7 | 8.9 |
| RLAIF-V | 38.0 | 7.9 | 33.3 | 7.3 | 33.0 | 7.8 | 27.3 | 6.8 |
| OPA-DPO | 40.0 | 8.6 | 29.0 | 5.4 | 30.3 | 7.6 | 27.0 | 7.1 |
| RC-DPO (Ours) | 21.7 | 6.2 | 21.3 | 5.4 | 18.7 | 4.9 | 24.3 | 6.4 |
Table 2: Comparison on the POPE benchmark across three splits (Accuracy / F1 Score %)
| Model | Method | Random Acc / F1 | Popular Acc / F1 | Adversarial Acc / F1 |
|---|---|---|---|---|
| MM-Eureka | Vanilla | 86.43 / 84.55 | 85.30 / 83.48 | 85.03 / 83.23 |
| MM-Eureka | VCD | 87.73 / 86.15 | 86.43 / 84.91 | 85.67 / 84.21 |
| MM-Eureka | SID | 87.50 / 85.88 | 85.93 / 84.35 | 85.40 / 83.85 |
| MM-Eureka | RLAIF-V | 86.30 / 84.28 | 84.87 / 82.92 | 84.50 / 82.57 |
| MM-Eureka | OPA-DPO | 87.33 / 85.63 | 86.13 / 84.49 | 85.30 / 83.71 |
| MM-Eureka | RC-DPO | 89.17 / 87.95 | 86.80 / 85.69 | 86.07 / 85.02 |
| R1-Onevision | Vanilla | 76.83 / 76.46 | 75.93 / 75.76 | 76.47 / 76.16 |
| R1-Onevision | VCD | 76.57 / 76.90 | 76.27 / 76.38 | 76.80 / 76.97 |
| R1-Onevision | SID | 76.77 / 76.82 | 77.63 / 77.63 | 77.03 / 77.24 |
| R1-Onevision | RLAIF-V | 79.73 / 78.86 | 79.07 / 78.36 | 78.60 / 77.98 |
| R1-Onevision | OPA-DPO | 85.53 / 83.58 | 83.60 / 81.83 | 83.20 / 81.47 |
| R1-Onevision | RC-DPO | 87.27 / 85.56 | 85.50 / 83.88 | 84.80 / 83.24 |
Ablation Study¶
Ablations on Object HalBench systematically confirm each module's contribution and parameter sensitivity:
| Configuration / Variant | CHAIRS ↓ Trend & Observation | Note |
|---|---|---|
| Full RC-DPO | Lowest hallucination rate (~21.3 on MM-Eureka) | Optimal overall synergy |
| w/o Reasoning loss \(\mathcal{L}_{\text{RC}}\) | Jumps sharply to >28.0 | Removing fine-grained alignment causes largest drop |
| w/o Attention pruning \(\tilde{c}_w\) | Increases by ~3.5 points | Hard negative conditions are vital for subtle hallucination detection |
| w/o MCTS sampling (Standard sampling) | Increases by ~2.8 points | Suboptimal positive demonstrations cap optimization potential |
| Loss weight \(\lambda = 0.1\) | Performance sweet spot | Smaller weights under-supervise; larger weights hinder general learning |
| Pruning ratio \(r = 20\%\) | Optimal visual corruption level | \(r=10\%\) provides weak negatives; \(r=40\%\) disrupts logical CoT syntax |
Key Findings¶
- Simultaneous reduction in CoT and answer hallucination: Segment-level CHAIRS demonstrates that while baseline OPA-DPO leaves CoT hallucinations near 30%, RC-DPO concurrently drives down both CoT and answer hallucination rates, disproving the assumption that longer reasoning chains inevitably amplify hallucinations.
- Enhanced general multimodal reasoning: On MME, RC-DPO boosts MM-Eureka perception score from 1595.77 to 1613.03 (notably OCR improves from 170.00 to 192.50) and cognition score from 706.79 to 718.93; on VMCBench, overall accuracy improves from 80.88% to 81.55%, proving that grounded reasoning reinforces visual understanding.
- Cross-scale generalizability: Evaluated on Vision-R1-32B, RC-DPO reduces Object HalBench CS from 46.3 to 25.7 and CI from 9.1 to 6.5, showing that the algorithm scales seamlessly to larger model architectures.
Highlights & Insights¶
- Exposing the "Answer Shortcut" flaw: By crafting controlled ablations that only contrast answer spans while sharing CoTs, the paper empirically proves that response-level DPO under-optimizes reasoning chains, pinpointing why MLRMs hallucinate despite RLHF.
- Surgical visual decoupling via attention pruning: Pruning tokens with the highest cross-modal image attention strips visual grounding while preserving fluent reasoning structure, generating challenging and realistic hard negatives.
- Clean mathematical formulation: Decomposing conditional likelihoods via Bayes' chain rule provides direct CoT-level supervision without requiring auxiliary discriminator networks or external proprietary API loops.
Limitations & Future Work¶
- CoT trajectory-level granularity: RC-DPO currently treats the entire CoT as a holistic condition, lacking step-by-step credit assignment to identify the exact inflection point where visual deviation first occurred.
- Dependence on visual attention distributions: If the vision-language projector exhibits diffused or spurious attention patterns, attention-guided pruning might occasionally remove functional words rather than true visual grounded tokens.
Related Work & Insights¶
- vs OPA-DPO (Yang et al., 2025b): OPA-DPO conducts on-policy response-level DPO but falls prey to answer shortcuts; RC-DPO isolates reasoning chains into conditional likelihood terms for fine-grained calibration.
- vs RLAIF-V (Yu et al., 2025): RLAIF-V depends on expensive external frontier models (e.g., GPT-4V) to generate correction feedback; RC-DPO achieves a self-contained data generation pipeline via internal MCTS and verifier scoring.
- vs VCD / SID (Leng et al., 2024; Huo et al., 2025): Decoding-time contrastive methods induce substantial inference latency and fail to address multi-step reasoning hallucination propagation; RC-DPO internalizes visual grounding into model weights with zero inference overhead.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Formulates a pioneering conditional preference framework for reasoning chains in MLRMs.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across four 7B models, one 32B model, and nine diverse hallucination and reasoning benchmarks.
- Writing Quality: ⭐⭐⭐⭐⭐ Cohesive narrative with thorough motivating empirical evidence, clear mathematical derivations, and clean experimental verification.
- Value: ⭐⭐⭐⭐⭐ Delivers an actionable, effective blueprint for stabilizing visual grounding in reasoning-centric multimodal foundation models.