HERO: Enhancing Multimodal Faithfulness via Dynamic Entropy-Aware Reinforcement Learning¶

Conference: ECCV 2026
Paper: ECCV Official
Area: Reinforcement Learning
Keywords: Multimodal Large Language Models, Hallucination Mitigation, Chain-of-Thought Reasoning, Reinforcement Learning, Entropy-Aware Optimization

TL;DR¶

Proposes HERO, an on-policy reinforcement learning framework that identifies the "Confidence Trap" in CoT-tuned LVLMs—where models retreat into language priors under visual ambiguity to produce low-entropy confident errors—and overcomes it using dynamic entropy-aware weighting and variance-gated hard-negative sampling.

Background & Motivation¶

Large Vision-Language Models (LVLMs) demonstrate remarkable abilities in parsing complex visual scenes and engaging in multimodal dialogue. However, persistent hallucinations continue to undermine their reliability in high-stakes domains like autonomous navigation and medical imaging. Recent research has widely adopted Chain-of-Thought (CoT) fine-tuning to unlock slow thinking and complex logical reasoning in multimodal models. Yet, empirical investigation reveals a counter-intuitive reasoning-faithfulness trade-off: optimizing for multi-step reasoning chains often comes at the direct cost of visual grounding, exacerbating visual fabrications.

Through controlled image-degradation experiments with Gaussian noise, this work uncovers the key bottleneck behind this phenomenon: "Overconfidence under Degradation" and the resulting "Confidence Trap." Intuitively, degrading visual evidence should reduce model confidence and increase output predictive entropy. Instead, CoT-tuned models retreat into strong linguistic priors to preserve narrative coherence regardless of visual clarity. Becoming "blind reasoners," they generate low-entropy, high-confidence errors. This finding exposes the flaw in conventional test-time methods like Visual Contrastive Decoding (VCD), which assume hallucinations correlate with high uncertainty. Under the Confidence Trap, both original and corrupted inputs favor the same hallucinated token with low entropy. Furthermore, offline preference optimization methods (such as DPO or HA-DPO) rely on static datasets and apply uniform penalties across error types, leaving confident hallucinations uncorrected.

To resolve this critical pathology where the most dangerous errors are mathematically confident, an on-policy optimization mechanism is required to explicitly align confidence with factual grounding. Core idea: formulate multimodal alignment as an on-policy reinforcement learning framework named HERO, which employs variance-gated sampling to filter uninformative plateaued groups and applies dynamic entropy-aware weighting to exponentially penalize low-entropy, high-confidence hallucinations while adaptively arbitrating between reward offense and KL defense.

Method¶

Overall Architecture¶

HERO (Hallucination-Entropy Regulated Optimization) builds on Group Relative Policy Optimization (GRPO). The pipeline consists of candidate response generation, variance-gated filtering, dual evaluation (NLI-based factuality reward and token-level predictive entropy), dynamic entropy weighting, and adaptive offense-defense arbitration. Given a multimodal prompt, the model samples a group of candidate responses. Uninformative groups with near-zero reward variance are filtered out. For retained responses, an NLI model computes factual rewards while average token-level Shannon entropy is evaluated; dynamic entropy-aware weights scale the policy gradients, and an adaptive balancing mechanism switches between reward exploration and reference policy regularization.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Input<br/>Image and text prompt"] --> B["Policy Sampling<br/>Generate candidate group of G outputs"]
    B --> C["Variance-Gated Sampling<br/>Compute reward variance & prune plateaued groups"]
    C --> D["Dual Evaluation Branch<br/>NLI reward model r and token-level entropy H"]
    D --> E["Dynamic Entropy-Aware Weighting<br/>Amplify penalties on low-entropy errors"]
    E --> F["Dynamic Offense-Defense Balancing<br/>Mean reward guides exploration vs. KL anchoring"]
    F --> G["GRPO Policy Update<br/>Realign predictive confidence with visual facts"]

Key Designs¶

1. Variance-Gated Sampling: Pruning Plateaued Groups to Focus on Hard Negatives Post-trained LVLMs already possess strong base capabilities, causing standard online RL to waste significant computation on prompts where all sampled responses are either uniformly correct or completely incorrect (reward variance near zero). To eliminate this computational waste, HERO evaluates the scalar reward variance across the \(G\) sampled outputs for prompt \(x\): \(\sigma_R^2(x) = \text{Var}(\{r_1, \dots, r_G\})\). Prompts exhibiting near-zero variance are treated as plateaued groups and discarded before backward policy updates. Serving as an efficiency-oriented hard-negative mining strategy, this selection avoids uninformative backward passes and reduces wall-clock training time from 22 hours to 16 hours (reducing compute from 274 to 185 GPU hours) without degrading final accuracy.

2. Dynamic Entropy-Aware Weighting: Breaking the Confidence Trap Standard GRPO assigns uniform optimization weights to all samples, ignoring the model's intrinsic predictive uncertainty and leaving low-entropy overconfident errors insufficiently penalized. To directly rectify this failure mode, HERO computes the average token-level Shannon entropy \(H(x)\) of each response and constructs an inverse sigmoid modulating weight \(w(H(x))\):

\[w(H(x)) = 1 + \sigma\left(-k \cdot (H(x) - H_0)\right)\]

where \(H_0\) is an empirical baseline representing the mean entropy of the reference model, \(k\) sets the steepness of the transition, and \(\sigma\) denotes the sigmoid function. When the model falls into a hallucination driven by strong language priors, its entropy drops significantly (\(H(x) \ll H_0\)), driving \(w(H(x))\) toward its maximum value of 2.0. Consequently, negative advantage updates for low-reward outputs are heavily amplified, forcing the model to unlearn pathological confidence. Conversely, when the model exhibits legitimate uncertainty (\(H(x) \ge H_0\)), the weight smoothly decays to 1.0, preserving normal GRPO optimization and avoiding penalizing appropriate hesitation.

3. Dynamic Offense-Defense Balancing: Arbitrating Exploration and Regularization Online reinforcement learning easily falls victim to reward hacking or catastrophic forgetting, which degrades general multimodal representations. To balance aggressive error correction with representational stability, HERO introduces an adaptive arbitration mechanism driven by the group's mean reward \(\bar{r}(x) = \frac{1}{G}\sum_{i=1}^G r_i\) relative to the baseline SFT reward expectation \(\theta_{\text{base}}\):

\[p_{\text{offense}}(x) = \text{sigmoid}\left(-(\bar{r}(x) - \theta_{\text{base}})\right), \quad p_{\text{defense}}(x) = 1 - p_{\text{offense}}(x)\]

When response quality lags behind the baseline expectation (\(\bar{r}(x) < \theta_{\text{base}}\)), the system enters "Offensive Mode" (\(p_{\text{offense}} \to 1\)) to aggressively explore faithful solutions via entropy-weighted advantage gradients. When group performance satisfies expectations (\(\bar{r}(x) > \theta_{\text{base}}\)), it smoothly transitions into "Defensive Mode" (\(p_{\text{defense}} \to 1\)), enforcing strong KL divergence regularization against \(\pi_{\text{ref}}\) to prevent distribution drift.

4. Multi-Component Reward Model: Fine-Grained NLI Proxy for Factuality Open-ended multimodal generation lacks natural scalar supervision. HERO utilizes a pre-trained DeBERTa-v3 NLI model to decompose ground-truth image annotations into atomic visual propositions as premises, verifying whether generated captions entail them. The holistic reward decomposes into format compliance \(S_{\text{format}}\), object existence \(S_{\text{obj}}\), and fine-grained attribute accuracy \(S_{\text{attr}}\):

\[R(C) = S_{\text{obj}} + S_{\text{attr}} + \lambda S_{\text{format}}\]

Here, \(S_{\text{obj}}\) measures the proportion of ground-truth objects logically entailed by the caption. Crucially, the attribute score \(S_{\text{attr}}\) is aggregated strictly over the subset of confirmed objects (\(E_o = 1\)), computing average entailment across atomic attribute propositions. This hierarchical structure avoids penalizing attributes of absent objects and provides clean, dense factuality supervision.

Loss & Training¶

The overall HERO optimization objective combines the entropy-weighted policy gradient with adaptive KL regularization:

\[\mathcal{J}_{\mathrm{HERO}}(\theta) = \mathbb{E}_{x \sim \mathcal{B}}\left[ p_{\mathrm{offense}}(x) \cdot w(H(x)) \cdot \mathcal{J}_{\mathrm{GRPO}}(x) - p_{\mathrm{defense}}(x) \cdot \mathcal{L}_{\mathrm{KL}}(x) \right]\]

The group relative advantage \(\hat{A}_i = \frac{r_i - \text{mean}(\{r\})}{\text{std}(\{r\}) + \epsilon}\) normalizes rewards within each sampled group, eliminating the critic network. The framework is evaluated on Qwen2.5-VL (3B and 7B). Models undergo SFT on LLaVA-CoT-100k to acquire baseline reasoning abilities, followed by on-policy HERO alignment on the Visual Attributes in the Wild (VAW) dataset. Deterministic greedy decoding (temperature = 0) is applied during inference.

Key Experimental Results¶

Main Results¶

Evaluation across standard hallucination benchmarks THRONE, POPE, and AMBER demonstrates comprehensive gains over base models and contemporary baselines:

Base Model	Optimization Method	THRONE Acc	THRONE F0.5	POPE Acc	POPE F1	AMBER-G CHAIR↓	AMBER-G Cover↑
Qwen2.5-3B	Base	78.6	85.7	85.0	83.8	7.5	58.5
Qwen2.5-3B	+ CoT-SFT	78.3	85.6	84.1	82.8	8.1	55.6
Qwen2.5-3B	+ VCD (Inference)	78.8	86.1	84.8	83.2	7.8	58.5
Qwen2.5-3B	+ HA-DPO (Offline)	78.5	85.5	84.2	82.0	7.5	57.5
Qwen2.5-3B	+ GRPO (Online)	82.2	88.2	85.5	83.8	7.3	61.5
Qwen2.5-3B	+ HERO (Ours)	82.8	88.6	86.5	85.4	7.2	62.3
Qwen2.5-7B	Base	77.9	84.8	86.5	84.6	6.6	60.8
Qwen2.5-7B	+ CoT-SFT	77.3	84.9	85.9	84.2	6.2	54.1
Qwen2.5-7B	+ VCD (Inference)	78.5	85.5	87.0	85.0	6.2	56.5
Qwen2.5-7B	+ HA-DPO (Offline)	79.5	86.0	87.2	86.0	6.2	55.5
Qwen2.5-7B	+ GRPO (Online)	81.9	88.5	89.2	90.0	6.0	59.0
Qwen2.5-7B	+ HERO (Ours)	82.9	89.0	90.8	90.4	6.3	61.2

Ablation Study¶

Ablations on optimization components and sampling strategies evaluated on the lowest-entropy 30% sample subset and general benchmarks:

Configuration / Strategy	THRONE Acc (%)	THRONE F1 (%)	MMBench Acc (%)	Note
Vanilla GRPO	80.6	77.7	65.5	Standard unweighted GRPO baseline
+ Variance-Gated Sampling	80.6	77.8	65.4	Prunes plateaued samples; saves significant compute
+ Entropy-Aware Weighting	82.0	78.7	65.7	Penalizes overconfident errors (+1.4% Acc)
+ Dynamic Balancing (Full HERO)	82.8	79.3	65.9	Optimal performance across factuality and general reasoning
Comparison: Full Data	80.6	77.7	65.5	Requires 22h wall-clock time (274 GPU·h)
Comparison: Random Sampling (60%)	79.2	76.5	64.9	Notable accuracy drop (-1.4%) under fixed budget
Comparison: Variance-Gated Sampling (~60%)	80.6	77.8	65.4	Requires only 16h wall-clock time (185 GPU·h) without drop

Key Findings¶

Recovery of Critical Low-Entropy Zones: In the ultra-low-entropy regime (\(H(x) \le 0.3\)), the CoT-SFT baseline suffers a truthfulness collapse to approximately 50.0% (nearly half of its most confident predictions are wrong). HERO recovers truthfulness in this zone to over 92.0%, achieving an 84% relative improvement.
Core Contribution of Dynamic Entropy Weighting: Entropy-aware weighting contributes a direct gain of +1.4% accuracy and +1.0% F1 on THRONE over vanilla GRPO, demonstrating that differentiating optimization updates by predictive uncertainty is far superior to uniform penalties.
Zero Alignment Tax: On general reasoning benchmarks including MMBench (65.6% \(\to\) 65.9%), MME (73.04 \(\to\) 73.42), and MMMU (46.56% \(\to\) 45.67%), HERO preserves the reasoning capability of CoT-SFT, proving that hallucination mitigation does not require sacrificing general competence.

Highlights & Insights¶

Counter-Intuitive Empirical Discovery: The paper overturns the standard assumption that hallucinations stem from high uncertainty, showing that visual ambiguity drives CoT models to retreat into language priors and produce pathological low-entropy fabrications.
Inverse Sigmoid Modulation: Modulating policy gradients via an inverse sigmoid function of predictive entropy provides an elegant, plug-and-play mechanism to severely punish confident fabrications while forgiving legitimate hesitation.
Efficiency-Driven Variance Gating: Identifying plateaued groups via empirical reward variance eliminates 32% of redundant backward computation while concentrating gradients on high-leverage decision boundaries.

Limitations & Future Work¶

Static Empirical Entropy Baseline: The threshold \(H_0\) is a static scalar derived from reference models. Since intrinsic uncertainty varies considerably across diverse visual tasks (e.g., OCR vs. open-ended scene captioning), an input-adaptive baseline could improve calibration.
Variance Gating Failure under Severe Domain Shift: If all sampled outputs fail uniformly under extreme distribution shift (yielding zero reward and zero variance), variance gating may mistakenly discard these groups as uninformative plateaus.
NLI Atomic Limitations for Complex Spatial Topologies: While atomic NLI propositions effectively verify object and attribute presence, they struggle to model dense multi-object geometric layouts and fine-grained spatial relationships.

vs. Inference-Time Contrastive Decoding (e.g., VCD, CDAR): VCD relies on contrasting logits between original and distorted inputs. In the Confidence Trap, models assign high confidence to hallucinated tokens under both branches, collapsing contrastive signals. HERO rectifies model parameters directly during training.
vs. Offline Preference Optimization (e.g., DPO, HA-DPO): Offline methods rely on static datasets and apply uniform loss across errors. HERO uses on-policy generation to continuously detect and penalize newly emerging overconfident blind spots.
vs. Standard GRPO: Standard GRPO is uncertainty-blind and computes gradients across all sampled groups. HERO incorporates variance gating and dynamic entropy-aware weighting, achieving faster convergence and targeted faithfulness.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Identifies the counter-intuitive low-entropy "Confidence Trap" in multimodal CoT reasoning and introduces an elegant entropy-regulated RL solution.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous noise-degradation experiments, entropy bin slicing, multiple hallucination benchmarks, and general capability checks.
Writing Quality: ⭐⭐⭐⭐⭐ Lucid problem formulation, clear mathematical exposition, and highly compelling visualizations.
Value: ⭐⭐⭐⭐⭐ Provides crucial insights and practical methods for reconciling chain-of-thought reasoning with factual faithfulness in multimodal models.