Rationale-Enhanced Decoding for Multi-modal Chain-of-Thought

Conference: CVPR 2026 arXiv: 2507.07685 Code: None Area: LLM Reasoning Keywords: Chain-of-Thought reasoning, multimodal large language models, decoding strategy, rationale grounding, plug-and-play

TL;DR

This work identifies that existing LVLMs effectively ignore intermediate rationale content during CoT reasoning, and proposes RED (Rationale-Enhanced Decoding)—multiplying the image-conditioned and rationale-conditioned next-token distributions at the logit level. This approach is theoretically equivalent to the optimal solution of KL-constrained reward maximization, and significantly improves multimodal reasoning accuracy without any training.

Background & Motivation

Background: Large vision-language models (LVLMs) adopt chain-of-thought (CoT) methods from LLMs, first generating intermediate reasoning (rationale), then producing the final answer conditioned on image, rationale, and question. CoT is widely believed to enhance grounding and accuracy in multimodal reasoning.

Limitations of Prior Work: The authors reveal a surprising finding through two key experiments—LVLMs effectively ignore rationale content during CoT reasoning. (1) Attention contribution analysis: when image and rationale are both provided, the attention contribution from rationale tokens drops significantly, with image tokens dominating prediction; (2) Rationale substitution experiment: replacing the correct rationale with a completely irrelevant one leaves model performance nearly unchanged, demonstrating that the model does not utilize the semantic content of the rationale at all.

Key Challenge: The joint conditional probability \(p_\theta(y_i|\mathbf{y}_{<i}, x, r, q)\) fails in practice to leverage the information in \(r\)—image tokens exert far greater "attraction" than rationale tokens. Yet dropping the image and using only \(p_\theta(y_i|\mathbf{y}_{<i}, r, q)\) discards visual information.

Goal: Design a training-free decoding strategy that enables LVLMs to genuinely exploit both image and rationale information during CoT reasoning.

Key Insight: Decouple image conditioning and rationale conditioning into two independent distributions and compose them at the logit level, bypassing the problem of rationale being ignored under joint conditioning.

Core Idea: By reformulating CoT reasoning as KL-constrained maximization with rationale-conditioned log-likelihood as the reward, the optimal decoding strategy is derived as the image-conditioned probability multiplied by the \(\lambda\)-th power of the rationale-conditioned probability.

Method

Overall Architecture

A standard two-step CoT pipeline is employed: (1) given image \(x\) and question \(q\), generate rationale \(r\); (2) given \(x\), \(r\), \(q\), generate the final answer. RED modifies the decoding strategy in step (2) without altering model parameters or the rationale generation procedure. RED can be combined with any rationale generation method.

Key Designs

1. KL-Constrained Reward Maximization Formulation:

2. Function: Reformulates CoT decoding as a theoretically grounded optimization problem.

3. Mechanism: Introduces a new next-token distribution \(\pi\) that maximizes: \(\max_\pi \mathbb{E}_\pi[R] - \beta \mathbb{D}_{\text{KL}}[\pi || \pi_{\text{ref}}]\), where the reward is \(R = \log p_\theta(y_i | \mathbf{y}_{<i}, r, q)\) (rationale-grounding reward) and the reference policy is \(\pi_{\text{ref}} = p_\theta(y_i | \mathbf{y}_{<i}, x, q)\) (image-conditioned probability).

4. Design Motivation: Maximizing the rationale-conditioned log-likelihood ensures the model utilizes rationale information; the KL constraint prevents excessive deviation from the image-conditioned distribution, thereby preserving visual information. This avoids the failure mode of rationale being ignored in the direct \(p(y|x,r,q)\) formulation.

5. RED Optimal Decoding Formula:

6. Function: Provides a closed-form optimal solution requiring no training.

7. Mechanism: Applying the known optimal policy form for KL-constrained reward maximization to this specific setting yields \(\hat{p}_\theta(y_i) = \frac{1}{Z_\theta} p_\theta(y_i|\mathbf{y}_{<i}, x, q) \times p_\theta(y_i|\mathbf{y}_{<i}, r, q)^\lambda\). This is a product-of-experts distribution that emphasizes the intersection of the image-conditioned and rationale-conditioned probability regions.

8. Design Motivation: Theorem 4.1 rigorously proves that this formula is the optimal solution to Eq. (7). The parameter \(\lambda = 1/\beta\) controls the influence weight of rationale information.

9. Practical Implementation (Logit-Level Weighted Summation):

10. Function: Translates RED into a simple logit operation.

11. Mechanism: \(\widehat{\text{logits}}_\theta(y_i) = \log\text{softmax}(\text{logits}_\theta(y_i|\mathbf{y}_{<i}, x, q)) + \lambda \cdot \log\text{softmax}(\text{logits}_\theta(y_i|\mathbf{y}_{<i}, r, q))\), then \(\hat{p}_\theta(y_i) = \text{softmax}(\widehat{\text{logits}}_\theta(y_i))\). The two logits can be computed in parallel, avoiding additional latency.
12. Design Motivation: Weighted summation of log-softmax values is the log-space equivalent of multiplication, yielding a simple and efficient implementation.

Loss & Training

RED is a purely inference-time method requiring zero training. It requires only two forward passes through the existing LVLM (one image-conditioned, one rationale-conditioned), followed by logit-level composition. The sole hyperparameter is \(\lambda\), which controls the degree of rationale influence.

Key Experimental Results

Main Results

GQA Dataset Accuracy (%)

Method	Gemma-3-4B	Gemma-3-12B
Direct (no CoT)	40.00	45.34
CoT (standard)	41.08	41.76 (decrease!)
CCoT (scene graph)	44.54	44.50
RED + CoT	significant gain	significant gain
RED + CCoT	significant gain	significant gain

Key Finding: Irrelevant Rationale Substitution

Input	Gemma-3-4B	Gemma-3-12B
\((x, r_{\text{CoT}}, q)\)	41.08	41.76
\((x, r'_{\text{CoT}}, q)\) irrelevant rationale	41.88	41.75
\((r_{\text{CoT}}, q)\) rationale only	40.15	37.87
\((r'_{\text{CoT}}, q)\) irrelevant rationale only	7.40	16.21

Ablation Study

Configuration	Effect	Note
Standard CoT decoding	baseline	$p(y
Rationale-only conditioning	degraded	lacks visual information
RED (appropriate \(\lambda\))	optimal	balances image and rationale
High-quality rationale (GPT-4) + RED	further improved	RED gains scale with rationale quality

Key Findings

• Standard CoT frequently underperforms direct answering: on Gemma-3-12B, CoT drops accuracy from 45.34 to 41.76, as the model ignores the rationale while being exposed to additional noise.
• The rationale substitution experiment is a killer piece of evidence: replacing the correct rationale with a random one leaves performance nearly unchanged (±0.1%), yet removing the image and relying solely on the rationale produces a dramatic difference (40.15 vs. 7.40), demonstrating that LVLMs entirely disregard rationale when an image is present.
• RED yields larger gains when combined with higher-quality rationales (e.g., GPT-4 generated), confirming that RED genuinely enables the model to "use" the rationale.
• RED is plug-and-play and can be stacked with other contrastive decoding methods (VCD, LCD).

Highlights & Insights

• Identifying the problem is more valuable than solving it: the discovery that "LVLMs ignore rationale in multimodal CoT" is rigorously supported by two elegant experiments—attention contribution analysis and rationale substitution. This finding challenges the widely held assumption that CoT is always beneficial.
• Theoretical elegance: deriving the decoding strategy as the optimal solution to KL-constrained reward maximization provides rigorous theoretical grounding for what might otherwise appear to be an ad hoc logit multiplication. The RLHF-flavored derivation framework is transferable to other multi-source decoding problems.
• Minimal implementation: the approach reduces to two lines of code (log-softmax weighted summation), requiring zero training, zero architectural modification, and zero additional models—a genuinely plug-and-play method.

Limitations & Future Work

• Two forward passes are required (image-conditioned and rationale-conditioned), doubling inference cost (though parallelizable in a batch).
• The rationale generation step itself still uses standard decoding with no quality guarantee; RED's gains are contingent on rationale quality.
• \(\lambda\) requires dataset-level tuning, and the optimal value may vary across tasks.
• The paper does not deeply analyze why LVLMs ignore rationale (the authors mention positional bias, attention sink, and overfitting during visual instruction tuning as possible causes, but do not empirically verify them).
• Validation is limited to VQA-type tasks; open-ended generation tasks are not explored.

Related Work & Insights

• vs. VCD (Visual Contrastive Decoding): VCD contrasts normal and corrupted images to mitigate hallucination; RED contrasts image-conditioned and rationale-conditioned distributions to enhance reasoning grounding. The two approaches are orthogonal and can be stacked.
• vs. LCD (Language Contrastive Decoding): LCD contrasts with/without image to reduce language priors; RED enhances rationale utilization. The two are equally orthogonal and complementary.
• vs. CCoT (Compositional CoT): CCoT improves rationale quality by generating scene graphs (optimizing Eq. 5); RED optimizes the decoding strategy for Eq. 6. The two are composable: high-quality rationales from CCoT can be combined with RED decoding.
• The "decouple input sources → compose at logit level" framework is generalizable to any multi-source reasoning scenario (e.g., fusing query-conditioned and context-conditioned distributions in RAG).

Rating

• Novelty: ⭐⭐⭐⭐⭐ A perfect combination of discovery and solution; the motivating experiments are highly convincing.
• Experimental Thoroughness: ⭐⭐⭐⭐ Validated across multiple models and datasets, though task diversity is limited (primarily VQA).
• Writing Quality: ⭐⭐⭐⭐⭐ The narrative flows smoothly from problem identification to theoretical modeling to practical algorithm.
• Value: ⭐⭐⭐⭐⭐ A plug-and-play inference enhancement method that exposes an important limitation of LVLMs' use of CoT.

Related Papers

• [CVPR 2026] Harnessing Chain-of-Thought Reasoning in Multimodal Large Language Models for Face Anti-Spoofing
• [CVPR 2026] Understanding and Mitigating Hallucinations in Multimodal Chain-of-Thought Models
• [CVPR 2026] VisRef: Visual Refocusing while Thinking Improves Test-Time Scaling in Multi-Modal Large Reasoning Models
• [CVPR 2026] Reinforcing Structured Chain-of-Thought for Video Understanding
• [CVPR 2026] Latent Chain-of-Thought World Modeling for End-to-End Autonomous Driving