HIVE: Understanding Post Hallucination Reasoning in Vision Language Models

Conference: ECCV2026
Paper: ECCV Paper
Area: Hallucination / Multimodal VLM
Keywords: post-hallucination reasoning, semantic intervention, caption faithfulness, paired evaluation, reasoning stability

TL;DR

HIVE uses paired faithful and hallucinated captions to study how vision-language models continue reasoning after hallucinations enter their context, finding substantial accuracy gains on some visual tasks without establishing either universal benefits or the trustworthiness of fabricated evidence.

Background & Motivation

Vision-language models (VLMs) often receive incomplete visual evidence: occlusion, blur, and low resolution leave some attributes unobservable. A model may fill these gaps using prior knowledge, producing plausible content that cannot be verified from the current input. Conventional hallucination research mainly asks whether such content can be detected, suppressed, or corrected, often treating an unfaithful description as the endpoint of analysis. In a multistage system, however, that description can subsequently enter a classifier, question-answering model, or decision module; its influence does not end with the preceding stage. Caption faithfulness alone cannot reveal whether the downstream model will ignore the content, be misled by it, or happen to reach the correct answer through it.

This question relates to chain-of-thought faithfulness but is not equivalent to checking an explanation for logical errors. Prior language-model studies suggest that incorrect intermediate text can coexist with a correct final answer, making answer correctness and evidential correctness distinct evaluation targets. For visual tasks, a conservative description may merely repeat visible colors and shapes without supplying a semantic cue that activates a useful diagnostic or recognition hypothesis. An unverified cue can instead trigger a useful task prior, but it can just as easily trigger the wrong one. The gap addressed here is therefore evaluative: how can the downstream influence of hallucination be measured once it is already present in the input context, without assuming every effect must be negative?

The authors call this stage Post-Hallucination Reasoning (PHR) and introduce HIVE, the Hallucination Inference and Verification Engine, to organize controlled comparisons. The aim is not to train a model to fabricate more effectively, but to obtain captions with different faithfulness labels from the same generation process and feed them to the same task solver. Experiments span textual and vision-language tasks to test whether the phenomenon depends on modality and task constraints, rather than selecting only successful examples. This design still requires caution about causality: matching prompts and decoding budgets reduces confounding but cannot ensure that two natural-language captions differ in no semantic dimension except truthfulness. Core Idea: separate the occurrence of hallucination from its subsequent utility, compare downstream predictions using matched faithful and hallucinated captions, and examine how semantic expansion and reasoning stability relate to outcomes.

Method

Overall Architecture

HIVE is evaluation infrastructure, not a new VLM architecture or a training algorithm for improving model parameters. Its input is an original example with a task label, potentially text, an image, or a structured record; intermediate outputs are natural-language captions and their faithfulness labels. The Caption Generator produces candidates, the Caption Discriminator retains contrasted faithful/hallucinated pairs, and the Task Solver receives the original input with each caption type separately. Outputs include predictions under each condition, accuracy differences, and reasoning text and embedding statistics for analysis. A Raw condition adds no caption, forming a control alongside the two augmented paths; the main tables primarily report the faithful path F and hallucinated path H.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Original input and task"] --> Generator["Caption Generator"]
    Generator --> Discriminator["Caption Discriminator"]
    Discriminator -->|Faithful F / Hallucinated H| Solver["Task Solver"]
    Input -->|Raw; augmented paths retain original input| Solver
    Solver --> Output["Paired predictions and<br/>reasoning statistics"]

Key Designs

1. Caption Generator: obtain both caption types from one stochastic generation process

The generator produces a set of free-form descriptions for each example rather than using one prompt for faithful content and another to deliberately manufacture errors. Within each paired comparison, generation source, prompt, sampling temperature, and token budget are held fixed, allowing stochastic generation to produce candidates with different faithfulness levels. This reduces decoding-configuration confounds such as the hallucinated group simply receiving a larger budget, a higher temperature, or special task guidance. The generation prompt is task-agnostic and descriptive; the downstream task instruction specifies the eventual decision. Consequently, the caption is neither the ground-truth answer label nor a reasoning policy optimized through solver feedback. The authors denote the number of candidate captions per input by \(N\), but the available main text does not provide a verifiable fixed value, so it should not be invented as an implementation setting.

Hallucination here includes speculative semantics that cannot be verified from the input, not only object names that visibly contradict an image. With incomplete images, an additional attribute can direct a model toward a task-relevant hypothesis even when the attribute lacks reliable visual support. Faithful and hallucinated captions remain different texts and may also differ in informativeness, specificity, and implicit class cues. Matched generation therefore provides a stronger starting point for comparison, not proof that an abstract truthfulness property has been fully isolated. The subsequent token-masking and random-checker controls further examine whether benefits depend on particular semantics rather than merely adding arbitrary text.

2. Caption Discriminator: establish evidence faithfulness before constructing contrasted pairs

Multiple independent detectors assess whether each candidate is faithful to its original input, and majority voting determines the final label. Only candidates that form a faithful/hallucinated contrast with majority-supported labels are retained for the matched downstream evaluation. The intermediate label answers whether the input supports the caption, not whether the caption helps the task solver answer correctly. Keeping these judgments separate matters: choosing captions by downstream success would mix the outcome being studied into sample construction. Majority voting is intended to reduce individual-detector noise, but detectors may share blind spots, and agreement does not guarantee factual labeling. Detector identities, their number, and ensemble configuration are delegated to Appendix S7, which is absent from the available cache; these implementation details cannot be claimed as verified.

Section 4.2 reports two reliability checks: 81.76% discriminator accuracy on TruthfulQA and 83.72% on a manually labeled cross-domain caption collection. The latter collection is described as 20 captions from each of 9 tasks, totaling 180; 83.72% does not directly correspond to an integer number of correct decisions in a single evaluation of all 180 captions. An unspecified averaging procedure or effective-sample convention could explain this, but the main text is insufficient to decide, so the reported percentage is retained without inferring a correct-count numerator. The authors also replace factual discrimination with a random checker while keeping decoding controls fixed; none of the task differences in Table 3 is significant. This weakens the explanation that arbitrary filtering would produce the same gains, but does not remove every influence of caption length, task informativeness, or the retained sample distribution.

3. Task Solver: compare matched predictions and trace semantic changes

The solver is a unified interface to existing models, not a newly trained network; its prompt combines a task instruction, the original input, and an optional caption. Raw uses only the original input, F adds a faithful caption, and H adds a hallucinated caption; augmented paths do not replace the original visual evidence with a description. The solver and its decoding conditions are matched within each comparison to avoid attributing changes in model capability or sampling configuration to caption faithfulness. The central metric is the H-minus-F accuracy difference, measured in percentage points rather than a percentage increase relative to F. The following is simplified notation for the stated metric, not a character-by-character reconstruction of the damaged extracted equation:

\[ \Delta(H-F)=\operatorname{Acc}_{H}-\operatorname{Acc}_{F}. \]

When both accuracies are expressed as percentages, their difference is in percentage points; a positive value only establishes H over F, not necessarily H over Raw. The main experiments compare this quantity across tasks and models, while semantically equivalent prompt paraphrases test dependence on a particular wording. The authors additionally select examples where H succeeds and F fails, locate hallucinated tokens serving as core evidence, replace them with a neutral placeholder, and rerun the solver. This intervention tests reliance on those cues, but near-perfect H-before accuracy follows from subset selection and cannot be compared directly with full-test-set accuracy. A drop after masking shows that the cues matter for these successful cases; it does not establish their truth or a benefit from hallucination on every example.

At the input level, the authors compare faithful and hallucinated caption embeddings to examine semantic coverage rather than merely counting additional words. At the process level, reasoning chains are embedded to study trajectory entropy, step-to-final cosine similarity, and similarity between chains from different sampling runs. For outcome analysis, captions are grouped by whether their associated answers are correct, and their semantic entropy is compared; this is an outcome-conditioned association. The reported pattern is lower trajectory entropy on some tightly constrained reasoning tasks and higher entropy on more open-ended tasks, so it cannot be reduced to a universal rule that higher entropy is better. Embedding models, entropy estimation, and statistical details are referred to Appendices S8 and S9, which are absent from the cache; this note retains conceptual definitions without guessing estimator formulas. These analyses support an observation of changed yet potentially stable semantic trajectories, but text-embedding similarity alone does not reveal the model's actual internal causal reasoning process.

Loss & Training

The paper proposes no new loss, fine-tuning stage, or reinforcement learning strategy; experiments use inference calls to existing models. The main adjustable factors concern caption generation: temperature, maximum token budget, and hallucination intensity; F/H pairs are compared under fixed settings before those settings are varied in ablations. Temperature settings are 0.0, 0.3, 0.6, and 0.9; length settings are 128, 256, 512, and 1024 tokens, with temperature 0.6 used for the length comparison. The intensity analysis uses GPT-4o to generate captions of different strengths, interpolates between faithful and hallucinated captions and between strong and weak hallucinations, and re-projects them into SBERT space for analysis. The main text does not fully explain how interpolation becomes executable textual input, so Figure 5 supports reporting an inverted-U trend, not describing a fully specified deployment algorithm. The discussion suggests falling back to the faithful path to manage risk, but provides no validated automatic fallback policy or safety threshold.

Key Experimental Results

Main Results

The study covers 9 tasks and 9 distinct models overall, not a complete matrix in which every model is evaluated on every task. Table 1 contains 5 text-only tasks and 7 models; Table 2 contains 4 vision-language tasks and 4 models, with overlap between the model groups. The selection below includes substantial gains, small gains, and negative cases; both F and H use caption augmentation, accuracies are in %, and H-F is in percentage points.

Source	Dataset	Model	F accuracy	H accuracy	H-F
Table 2, p. 8	PlantVillage	GPT-4o	62.73	77.41	+14.68
Table 2, p. 8	PlantVillage	Claude 3 Sonnet	55.28	72.50	+17.22
Table 2, p. 8	ISIC	GPT-4o	63.88	75.64	+11.76
Table 2, p. 8	ISIC	Qwen VL-Max	58.71	75.62	+16.91
Table 2, p. 8	Dex-Net	GPT-4o	53.25	55.76	+2.51
Table 2, p. 8	GQA	Qwen VL-Max	69.88	67.90	-1.98
Table 1, p. 7	BBBP	O3	73.27	59.47	-13.80
Table 1, p. 7	ProofWriter	O3	97.76	98.45	+0.69

Among vision-language results, all four models improve on PlantVillage, whereas Claude 3 Sonnet and Qwen VL-Max both decline on GQA. Text-only effects are not uniformly absent: their sign and magnitude depend on the model, and the large O3 decline on BBBP should not be hidden by a broad description of small benefits. The more defensible conclusion is that gains are more frequent on visual tasks, not that hallucination reliably helps every VLM or every task.

Ablation Study

The first ablation table reproduces the selected analysis from Table 5, p. 10: hallucinated evidence tokens are masked only on the subset where H succeeds and F fails. Values retain the paper's 0-to-1 accuracy scale and accompanying dispersion terms; the main text does not identify those terms as standard deviations, standard errors, or another statistic. The table also does not provide per-task sample counts or solver identities, so these details are not inferred.

Dataset	Domain	Downstream task	H accuracy after masking
AntiCP2	Biomedicine	Peptide classification	0.244 ± 0.085
PlantVillage	Agriculture	Disease recognition	0.700 ± 0.111
Dex-Net	Robotics	Grasping prediction	0.364 ± 0.061
ISIC	Dermatology	Lesion diagnosis	0.380 ± 0.062

The second ablation table selects the temperature portion of Table 6, p. 14; all entries are H-F differences in percentage points. Its caption calls the quantity relative gain but defines it as H-F, so it is not interpreted here as a relative percentage increase. The main text and caption do not explicitly identify the solver for this table or the token budget held fixed during the temperature sweep; matching reference values are not used to infer either detail.

Dataset	Temperature 0.0	Temperature 0.3	Temperature 0.6	Temperature 0.9
AntiCP2	-4.27	+0.10	+3.76	-5.00
PlantVillage	+2.30	-4.46	+14.68	+1.51
Dex-Net	+0.07	+1.88	+2.51	+0.14
ISIC	+9.26	-5.05	+11.76	+3.70

All four tasks peak at 0.6 within this temperature grid, which is not a guarantee of a universal optimum across models. In the same table's length experiment, AntiCP2 reaches +4.48 at 512 tokens versus +3.76 at 256 tokens, so relatively stable performance at 256 tokens does not mean it is optimal for every task.

Key Findings

• The prompt-paraphrase experiments in Table 4, p. 9 preserve positive gains for the listed combinations, but PlantVillage / GPT-4o gains are +14.68, +28.00, and +9.89, showing substantial magnitude sensitivity to wording.
• Figure 3, p. 10 shows step-to-final convergence and highly overlapping inter-chain distributions on PlantVillage; the text reports an inter-chain mean around 0.97 and p > 0.6 for the difference. This measures semantic consistency, not factual accuracy.
• Figure 2, p. 8 and the discussion on p. 12 associate higher caption semantic entropy with correct predictions, while the direction of trajectory-entropy change depends on the task; these are not interchangeable entropy measures.

Highlights & Insights

• Separately evaluating an error's occurrence and its downstream effect is the most reusable design choice. The discriminator measures evidence faithfulness while the solver measures answer correctness, avoiding a single label that conflates both objectives.
• In the ISIC example on p. 13 and Figure 6 on p. 14, a spurious vascular cue anchors a seborrheic-keratosis interpretation and yields the correct benign answer, while the faithful path incorrectly predicts malignancy. This shows that a semantic cue can activate a correct hypothesis, not that the vascular evidence exists.
• A reader takeaway is to represent candidate hypotheses separately from verified evidence, then filter them against the original input or external verification. This could preserve semantic exploration without promoting unverified descriptions into facts; it is not a module implemented by the paper.

Limitations & Future Work

• The authors acknowledge limited benchmarks and interventions concentrated at the caption level, leaving hallucinations within reasoning steps or latent representations incompletely covered. The main text references a hallucination-position analysis in Appendix S4, but that appendix is absent from the cache and its detailed results cannot be verified.
• Reader assessment: matched generation and pairing cannot eliminate all semantic confounds, and majority voting remains fallible; further controls should match caption length, task relevance, and class cues, and report pair-retention rates and human verification.
• Extracted equations are damaged and the cache contains only the main paper and references; prompts, dataset settings, detectors, and statistical implementations in Appendices S2, S6, S7, S8, and S9 cannot be verified. The note explicitly preserves uncertainty about the human-labeled accuracy denominator and dispersion definitions.
• The authors explicitly do not advocate hallucination as a deliberate inference strategy. In medicine and robotics especially, one correct answer or repeatable agreement cannot replace reliable evidence, calibration assessment, and practical safety validation.

Related Work & Insights

• Compared with hallucination detection and mitigation: SelfCheckGPT, factual-consistency evaluation, and contrastive decoding primarily identify or reduce unsupported content; HIVE studies its effect after entering a subsequent stage, making the objectives complementary.
• Compared with multimodal chain-of-thought analysis: MIRAGE and More Thinking, Less Seeing examine hallucination in reasoning chains; HIVE primarily intervenes on paired captions supplied to a task solver rather than only observing already generated chains.
• Compared with retrieval-augmented generation: retrieval seeks traceable evidence, whereas HIVE's speculative cues provide no such guarantee. A possible extension is to propose candidate explanations and retrieve evidence to verify them, instead of treating usefulness as a proxy for trustworthiness.

Rating

• Novelty: 4/5. Isolating the post-hallucination stage contributes more through its research perspective than through new modules.
• Experimental Thoroughness: 4/5. Tasks, models, and controls are varied, but some reproducibility details depend on unavailable appendices and causal attribution remains limited.
• Writing Quality: 3/5. The main argument is clear, but wording that equates stability with reliability needs qualification, and some statistical conventions and gain terminology are unclear.
• Value: 4/5. Useful for evaluating multistage-system reliability, but not a basis for recommending hallucination in applications.