Skip to content

The Cost of Reasoning: Chain-of-Thought Induces Overconfidence in Vision-Language Models

Conference: ECCV2026
Paper: ECCV
Area: VLM Reasoning (Multimodal VLM)
Keywords: uncertainty quantification, chain-of-thought, selective generation, vision-language models, overconfidence

TL;DR

Across four multimodal benchmarks and four open-weight VLMs, this paper shows that chain-of-thought prompting and reasoning-trained models improve task accuracy while substantially degrading the ranking quality of uncertainty estimates that rely on answer-token likelihood (ATL); it attributes this to implicit answer conditioning β€” the reasoning trace commits to a conclusion before the answer is generated, inflating answer-token likelihood regardless of correctness β€” whereas agreement-based consistency stays robust and often improves under reasoning.

Background & Motivation

Vision-language models are moving into high-stakes settings β€” from diagnostic reading of medical images to decisions in autonomous driving β€” where "knowing when it does not know" matters as much as being right. The engineering counterpart is selective generation: compute a confidence score \(g(x)\) for every input, answer only when \(g(x)\geq\tau\) and abstain otherwise, trading coverage against risk by moving the threshold \(\tau\). In this paradigm, the quality of an uncertainty estimate is not whether it is a well-calibrated probability but whether it reliably ranks correct predictions above incorrect ones, which is why ranking-oriented metrics such as PRR, AUGRC and Spearman correlation dominate the evaluation. At the same time, modern VLM pipelines have pushed reasoning to the front: either through chain-of-thought (CoT) prompting that makes the model write intermediate steps, or through reasoning-native "Thinking" models.

The natural intuition is that reasoning, having made the model think longer, should improve both accuracy and the reliability of uncertainty. The results in this paper show that this intuition often fails. Across several VLM families and benchmarks, the authors observe that reasoning does raise task accuracy, yet systematically damages the ranking quality of estimates built on answer-token likelihood β€” maximum sequence probability (MSP), perplexity (PPL), mean token entropy (MTE), and Monte Carlo sequence entropy (MC-SE): PRR falls from 0.5–0.7 to 0.1–0.3, and Spearman correlation drops to nearly zero. The problem is not merely that extra conditioning changes token probabilities, which would be expected; it is that the meaning of the answer-token likelihood changes. It no longer measures whether the answer is correct but increasingly measures how consistent that answer is with the reasoning trace the model has just written.

Multimodal settings make this misalignment especially consequential. In text-only models, reasoning and conclusion share one symbolic system, whereas a VLM must ground its conclusion in visual evidence that may itself be ambiguous: a reasoning trace can converge on a confident conclusion from an image that is simply not legible. Token-level probabilities can express "consistent with my own reasoning" but cannot express "the visual signal is ambiguous." As a result, models abstain less often on hard, visually ambiguous samples, producing more high-confidence errors even as average accuracy improves. The core idea of this paper is to attribute this degradation to "implicit answer conditioning" β€” the reasoning trace semantically commits to a conclusion before the final answer is generated, making answer tokens highly predictable and systematically inflating their likelihood, independently of whether the answer is correct; consequently, reasoning-enabled VLMs should switch to agreement-based consistency, which does not depend on token likelihood.

Method

Overall Architecture

This paper introduces no new model or training objective; its "method" is a diagnostic protocol that fixes the model and the data so that whether the model reasons becomes the only variable, then observes how each family of uncertainty estimators behaves and uses correlation analysis plus controlled interventions to localize the mechanism. The protocol has four layers. First, reasoning conditions are constructed: the same instruction-tuned model with and without CoT prompting, plus a matched comparison between an Instruct model and a Thinking model on the same base. Second, nine uncertainty estimators from three families are computed in parallel on every output. Third, ranking quality is measured inside the selective-generation framework with PRR, Spearman correlation and AUGRC. Finally, the mechanism is probed β€” confidence shifts grouped by correctness transitions, partial correlations between reasoning length and confidence controlling for correctness, the frequency with which the answer appears in the trace, and a [MASK] intervention on the trace. Evaluation covers OK-VQA, MathVista, MMMU-Pro-Vision and Oxford-IIIT Pet (a pure visual classification control task where reasoning brings little benefit), with Gemma3-4B-IT, InternVL3-9B and Qwen3-VL-8B/32B-IT.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["4 benchmarks: OK-VQA / MathVista<br/>MMMU-Pro-Vision / Oxford-IIIT Pet"] --> B["Three estimator families x three reasoning conditions<br/>ATL / self-probing / consistency"]
    B --> C["Selective-generation evaluation<br/>PRR / Spearman / AUGRC"]
    C -->|ATL ranking degrades| D["Implicit answer conditioning<br/>correctness-agnostic confidence shift"]
    D --> E["Answer frequency & [MASK] intervention"]
    E --> F["Conclusion: use consistency<br/>for reasoning-enabled VLMs"]

Key Designs

1. Three estimator families Γ— three reasoning conditions: making reasoning the only variable

To ask whether reasoning hurts uncertainty, the families of uncertainty estimates must be separated first, otherwise the effects of different mechanisms cancel out. The paper groups nine estimators into three categories. The answer-token likelihood (ATL) family compresses token probabilities over the answer span into one scalar: MSP takes the joint probability of the answer sequence directly (a product over tokens, hence exponentially decaying with answer length and sensitive to it); PPL normalizes the log-likelihood by sequence length to mitigate that bias; MTE averages the Shannon entropy of the next-token distribution over the answer positions; and MC-SE averages the negative log-likelihood over \(K\) sampled sequences (with a length-normalized MC-NSE variant). One easily missed detail: MC-SE and MC-NSE still aggregate token probabilities within each sampled sequence independently, so they never explicitly measure disagreement across generations β€” several samples can all have low entropy as long as each is individually likely. The self-probing family uses only SRC: the question, the image and the model's own answer are composed into a new context, the model is asked "Is the above answer correct? Answer 'yes' or 'no'", and confidence is the normalized probability of the yes token,

\[P_{\text{SRC}}=\frac{P(\text{yes}\mid X)}{P(\text{yes}\mid X)+P(\text{no}\mid X)},\quad X=\{Q,I,A\}\]

The agreement family follows Self-Consistency: sample \(K\) answers and take the fraction agreeing with the majority vote,

\[\text{Consistency}=\frac{1}{K}\sum_{k=1}^{K}\mathbb{1}\!\left[a^{(k)}=A^{\ast}\right]\]

Consistency is the only black-box estimator of the three: it needs just the final answer text, not access to token probabilities.

On this grid of estimators, the reasoning condition is designed with three settings to separate prompt-induced reasoning from a model's intrinsic reasoning ability. The same instruction-tuned model is evaluated with and without CoT instructions (dataset-specific instructions for MathVista and OK-VQA, plus four text-only in-context examples for OK-VQA). In addition, Qwen3-VL-8B-IT is compared against Qwen3-VL-8B-Thinking, which produces reasoning traces on its own under exactly the same prompts. All multi-sample estimators use \(K=10\), and every confidence score is computed over the final answer span only, never over the intermediate reasoning trace β€” otherwise a longer trace would itself contaminate the score.

2. Selective-generation evaluation: using PRR to decouple ranking quality from accuracy

Reporting AUGRC alone leads to the wrong conclusion, and this is the paper's key methodological move. In selective generation the model abstains below a threshold \(\tau\); sweeping \(\tau\) traces out a risk–coverage curve. The general AURC β€” and the generalized AUGRC adopted here β€” is influenced by both model accuracy and the ranking quality of the uncertainty estimate: raise accuracy and AUGRC improves even while ranking deteriorates. The experiments walk straight into this trap: on reasoning-heavy benchmarks such as MathVista and MMMU-Pro-Vision, CoT makes AUGRC look better, but only because accuracy rose (MathVista 63.6β†’70.4), while PRR and Spearman in the very same cells consistently get worse. The authors therefore make PRR the primary metric: it sandwiches the estimate between a random ranking and an oracle ranking and keeps only the ranking gain over chance,

\[\text{PRR}=\frac{AR_{\text{unc}}}{AR_{\text{orc}}}\]

where \(AR_{\text{unc}}\) is the area between the rejection curve of the uncertainty estimate and the random baseline, and \(AR_{\text{orc}}\) is the area between the oracle (perfect-ranking) curve and the same baseline. Because numerator and denominator share the same accuracy background, PRR is largely invariant to overall model accuracy, which is exactly what is needed to ask "what else did reasoning change besides accuracy?" A non-parametric Spearman rank correlation is reported alongside as corroboration. The paper explicitly avoids ECE as the primary metric: many effective estimators in selective generation are not calibrated probabilities in the first place, and good calibration does not imply good ranking (ECE results are in Appendix A).

3. Implicit answer conditioning: casting confidence inflation as a correctness-agnostic shift

This is the paper's explanation of the phenomenon. A reasoning trace is autoregressively generated text; once it has written a sentence like "therefore rounding to the nearest inch gives 3 inches", the answer tokens that follow are already strongly constrained by the model's own preceding text β€” the answer becomes highly predictable given the trace. At the token level this predictability is numerically indistinguishable from genuine confidence, so answer likelihood is inflated, and what it reflects is the model's semantic commitment to its own trace rather than a grasp of whether the answer is right. To make this testable, let \(s(\hat y\mid x,r)\) be the ATL confidence of the predicted answer \(\hat y\) conditioned on input \(x\) and reasoning trace \(r\), and \(s(\hat y\mid x)\) the same score without reasoning. The reasoning-induced confidence shift is

\[\Delta s=s(\hat y\mid x,r)-s(\hat y\mid x)\]

Reasoning raises ATL confidence on average (\(\mathbb{E}[\Delta s]>0\)). The signature of implicit answer conditioning is that this inflation is correctness-agnostic:

\[\mathbb{E}[\Delta s\mid \hat y=y^{\ast}]\approx\mathbb{E}[\Delta s\mid \hat y\neq y^{\ast}]\]

Because the extra confidence is not tied to correctness, it cannot help separate correct from incorrect predictions, so ATL becomes a less faithful indicator of correctness β€” the same statement as the observed PRR / Spearman degradation.

This account must also rule out an obvious competing hypothesis: perhaps longer, more verbose reasoning simply flattens or lifts the probability distribution. Controlling for prediction correctness, the authors compute the partial Spearman correlation between the number of reasoning tokens and each estimate's confidence and find a negative correlation: longer traces correspond to lower confidence. That matches prior work linking longer reasoning to harder problems and genuinely reduced confidence (Devic et al. propose trace length itself as an uncertainty signal), and it points in the opposite direction from inflation β€” if length were the driver, long traces would be more confident. The degradation therefore comes from the conditioning effect of the reasoning context, not from length as a surface property.

4. Answer frequency and the [MASK] intervention: from correlational to interventional evidence

Showing "longer β†’ less confident" is not enough; positive evidence for conditioning is needed. The paper first uses how often the predicted answer appears in the reasoning trace as a proxy for conditioning strength: the more often a model repeats a conclusion, the stronger its contextual commitment to it. On OK-VQA with Qwen3-VL-32B-IT under CoT, confidence rises monotonically with answer frequency. Extending the correlation to all datasets and models, ATL estimates show a stable positive correlation with answer frequency, and β€” crucially β€” the correlation survives when recomputed on incorrect samples only, which is the direct manifestation of correctness-agnostic inflation. As a control, consistency and SRC are essentially uncorrelated with answer frequency, so they are not subject to this conditioning.

Correlation is not causation, so a controlled intervention follows: for each generated CoT trace, every exact occurrence of the predicted answer is replaced with [MASK] while all other tokens are untouched, and the fixed answer sequence is then re-scored under the masked trace. Because the answer itself is held fixed, accuracy is identical across the CoT and Masked-CoT conditions by construction, so any change in PRR can only be attributed to the confidence distribution. A Random Mask control β€” masking an equal number of randomly chosen reasoning tokens β€” rules out the trivial explanation that deleting tokens helps by itself. The result (OK-VQA, \(n=3900\), restricted to the subset where the answer really occurs in the trace) is that MSP's PRR goes from 0.141 to 0.475, more than tripling, with Spearman improving by 0.17–0.28, while random masking produces only moderate change (PRR 0.210). Masking does not fully restore the no-CoT level (0.475 vs. 0.498), which indicates that explicit mentions are only the most blatant part of the conditioning and that implicit conditioning persists β€” a boundary the authors themselves stress.

A Worked Example: confidence climbing along a wrong reasoning trace

Fig. 1 of the paper gives a case that makes the mechanism concrete. The input asks the model to move a ruler next to a nail and report its length to the nearest inch. Without CoT the model answers directly with <answer>3</answer>. With CoT it first writes a <thought>: the nail's head aligns with the 0-inch mark and its tip extends past the 3-inch mark but not to the 4-inch mark, so rounding to the nearest inch gives 3 inches, followed by the same <answer>3</answer>. The answer is wrong in both cases (ground truth is 2), yet the interesting part is the trajectory of confidence: as reasoning tokens are generated one by one, the model's probability of its final answer, \(P(\hat a\mid x,r_{\leq t},\theta)\) β€” the probability of answer 3 given only the reasoning prefix produced so far β€” climbs monotonically, and once the trace has converged on "therefore it is 3" it clearly exceeds the no-CoT confidence. The model has been persuaded by its own trace: the wrong conclusion is locked in inside the reasoning before the answer is emitted, and the answer tokens then receive a high likelihood for the wrong reason. This picture is the mirror image of the [MASK] result β€” erase the mentions that lock in the conclusion and confidence falls back, recovering most of the ranking quality.

Key Experimental Results

Main Results

The main experiments use Qwen3-VL-8B-IT as the backbone and compare the same model with and without CoT prompting (Table 1; results for Gemma3-4B-IT, InternVL3-9B and Qwen3-VL-32B-IT are in Appendix F and show the same trends). Lower AUGRC is better, higher PRR is better, and Acc is the accuracy for that setting (No CoT / CoT).

Dataset Acc (No CoT / CoT) Estimator AUGRC↓ No CoT AUGRC↓ CoT PRR↑ No CoT PRR↑ CoT
Oxford-IIIT Pet 79.3 / 76.2 MSP 0.056 0.110 0.710 0.231
Oxford-IIIT Pet 79.3 / 76.2 MTE 0.053 0.112 0.735 0.205
Oxford-IIIT Pet 79.3 / 76.2 SRC 0.060 0.100 0.709 0.314
Oxford-IIIT Pet 79.3 / 76.2 Consistency 0.081 0.100 0.525 0.264
OK-VQA 55.9 / 52.1 MSP 0.167 0.208 0.525 0.241
OK-VQA 55.9 / 52.1 MTE 0.168 0.216 0.510 0.174
OK-VQA 55.9 / 52.1 SRC 0.175 0.194 0.465 0.441
OK-VQA 55.9 / 52.1 Consistency 0.167 0.171 0.436 0.478
MMMU-Pro-Vision 38.1 / 46.0 MSP 0.264 0.256 0.450 0.123
MMMU-Pro-Vision 38.1 / 46.0 MTE 0.263 0.234 0.479 0.376
MMMU-Pro-Vision 38.1 / 46.0 SRC 0.276 0.234 0.380 0.409
MMMU-Pro-Vision 38.1 / 46.0 Consistency 0.285 0.215 0.107 0.434
MathVista 63.6 / 70.4 MSP 0.143 0.126 0.432 0.219
MathVista 63.6 / 70.4 MTE 0.146 0.130 0.413 0.171
MathVista 63.6 / 70.4 SRC 0.143 0.094 0.473 0.557
MathVista 63.6 / 70.4 Consistency 0.148 0.068 0.345 0.684

The second main experiment replaces prompt-induced reasoning with the model's intrinsic reasoning: Qwen3-VL-8B-IT (no CoT instruction) versus Qwen3-VL-8B-Thinking (no CoT instruction, reasons on its own), with Acc as Instruct / Thinking.

Dataset Acc (Instruct / Thinking) Estimator PRR↑ Instruct PRR↑ Thinking Ξ”PRR
Oxford-IIIT Pet 79.3 / 77.2 MSP 0.710 0.120 βˆ’0.590
Oxford-IIIT Pet 79.3 / 77.2 MTE 0.735 0.086 βˆ’0.649
Oxford-IIIT Pet 79.3 / 77.2 MC-SE 0.645 βˆ’0.057 βˆ’0.702
Oxford-IIIT Pet 79.3 / 77.2 Consistency 0.525 0.223 βˆ’0.302
OK-VQA 55.9 / 56.0 MSP 0.525 0.110 βˆ’0.415
OK-VQA 55.9 / 56.0 MTE 0.510 0.065 βˆ’0.446
OK-VQA 55.9 / 56.0 MC-SE 0.515 0.278 βˆ’0.236
OK-VQA 55.9 / 56.0 Consistency 0.436 0.502 +0.066
MMMU-Pro-Vision 38.1 / 56.6 MSP 0.450 0.357 βˆ’0.093
MMMU-Pro-Vision 38.1 / 56.6 MTE 0.479 0.403 βˆ’0.077
MMMU-Pro-Vision 38.1 / 56.6 MC-SE 0.447 0.556 +0.109
MMMU-Pro-Vision 38.1 / 56.6 Consistency 0.107 0.455 +0.348
MathVista 63.6 / 80.3 MSP 0.432 0.011 βˆ’0.421
MathVista 63.6 / 80.3 MTE 0.413 βˆ’0.056 βˆ’0.469
MathVista 63.6 / 80.3 MC-SE 0.440 0.036 βˆ’0.403
MathVista 63.6 / 80.3 Consistency 0.345 0.767 +0.422

Ablation Study

The [MASK] intervention tests the "answer mention" channel of conditioning (OK-VQA, Qwen3-VL-8B-IT, \(n=3900\)). Accuracy is identical by construction across the four conditions (masking never changes the answer), so differences in PRR and Spearman can only come from the confidence distribution itself.

Estimator Metric No CoT (Acc 59.7) CoT (57.7) Masked-CoT (57.7) Random Masked-CoT (57.7)
MSP PRR↑ 0.498 0.141 0.475 0.210
MSP Spearman↑ 0.375 0.115 0.378 0.161
PPL PRR↑ 0.469 0.075 0.436 0.146
PPL Spearman↑ 0.345 0.059 0.340 0.113
MTE PRR↑ 0.481 0.071 0.329 0.137
MTE Spearman↑ 0.364 0.057 0.230 0.102

Key Findings

  • ATL degradation and accuracy gains are two different things. On MMMU-Pro-Vision and MathVista, CoT improves AUGRC (e.g. MathVista MSP 0.143β†’0.126), yet in the same cell PRR falls from 0.432 to 0.219 and Spearman from 0.284 to 0.170. Because AUGRC is driven by both accuracy and ranking, reading it alone would misreport "ranking collapsed while accuracy rose" as better uncertainty β€” which is precisely why the authors insist on PRR.
  • Reasoning-trained models amplify the effect. The PRR drop from Instruct to Thinking is generally larger than from no-CoT to CoT prompting; MTE falls from 0.735 to 0.086 on Oxford-IIIT Pet and from 0.413 to βˆ’0.056 on MathVista (a negative PRR means the ranking is worse than chance). The effect is therefore not an artifact of prompt wording but a general property of reasoning in these models.
  • Consistency is the only estimator that generally benefits from reasoning. PRR goes from 0.345 to 0.684 on MathVista and from 0.107 to 0.434 on MMMU-Pro-Vision (both under CoT and under the Thinking model), with smaller gains on OK-VQA (0.436β†’0.478 / 0.436β†’0.502). The boundary should be stated honestly: on the control task Oxford-IIIT Pet, consistency also degrades (0.525β†’0.264), just less than ATL; Pet is exactly the pure-classification setting where reasoning brings no accuracy gain, and there every estimator gets worse.
  • SRC behaves mixed. It clearly benefits on reasoning-heavy benchmarks (MathVista 0.473β†’0.557, MMMU 0.380β†’0.409), is roughly flat on OK-VQA (0.465β†’0.441), and on Pet, although it falls from 0.709 to 0.314, becomes the best choice in that setting, on par with consistency.
  • The [MASK] intervention shows explicit mentions explain most, but not all, of the effect. Masking answer mentions restores MSP's PRR to 95% of the no-CoT baseline (0.475 vs. 0.498) whereas random masking reaches only 0.210, confirming that the gain comes from the semantic channel of answer mentions rather than from deleting tokens. The residual gap shows implicit conditioning is still present.
  • Masking can be harmful on structured reasoning tasks. On MathVista (Appendix L) random masking causes a substantial drop in UQ ranking quality; the authors explain that masking disrupts the structured tokens mathematical solutions depend on. Interventions of this kind must therefore account for how much the task relies on reasoning coherence.
  • Consistency's cost is compute, and it is highest exactly where consistency helps most. MSP, PPL and MTE need a single forward pass, whereas consistency needs \(K=10\) stochastic generations; MC-SE and MC-NSE carry the same cost yet remain unreliable. Two factors mitigate this: the \(K\) generations are independent and can be batched, so wall-clock latency need not scale linearly with \(K\) given enough throughput (total token cost still does), and \(K\) can be tuned down to trade compute for ranking quality, with graceful degradation (Appendix D). Where multi-sample reasoning is infeasible, SRC is a cheaper alternative that is less consistently reliable but avoids the systematic inflation affecting ATL.

Highlights & Insights

  • Defining uncertainty quality as a ranking problem β€” and spotting the AUGRC trap. The paper first explains why ECE is inapplicable to selective generation (effective estimators need not be calibrated, and calibration does not imply good ranking), then strips accuracy out of the picture by making PRR's numerator and denominator share the same accuracy background. This "define the quantity first, then describe the phenomenon" move is the most transferable methodological lesson here for anyone reproducing UQ evaluations.
  • The mechanism comes with a falsifiable signature rather than a story. Saying "reasoning makes models more confident" would be vacuous; the paper turns implicit answer conditioning into the correctness-agnostic shift \(\mathbb{E}[\Delta s\mid \hat y=y^{\ast}]\approx\mathbb{E}[\Delta s\mid \hat y\neq y^{\ast}]\), rules out the verbosity hypothesis via the negative length correlation, and verifies the signature through the positive answer-frequency correlation on incorrect samples alone. The argument chain is closed.
  • The two-tier evidence design is reusable. Correlation (answer frequency Γ— confidence) followed by a controlled intervention ([MASK] versus Random Mask, with the answer fixed so accuracy is constant by construction) is a template for any question of the form "is factor X inflating model confidence?" β€” for instance, whether format instructions, retrieved context or persona prompts manufacture false confidence.
  • A concrete deployment conclusion: switch to consistency on reasoning VLMs. As long as reasoning is already in the pipeline, continuing to gate abstention on single-pass token likelihood such as MSP or MTE gives the most dangerous confidence on the hardest samples; paying for \(K\) generations buys a usable uncertainty signal rather than the wasted compute of MC-SE.

Limitations & Future Work

  • Limitations acknowledged by the authors: the masking intervention only covers explicit answer mentions, the most blatant form of conditioning, and still fails to fully restore the no-CoT level, so part of the implicit conditioning is untouched; moreover masking requires exact string matching of the answer and is therefore only feasible on open-ended formats such as OK-VQA and MathVista.
  • Limitations I see: the main experiments detail only the Qwen3-VL-8B tier, pushing Gemma3-4B, InternVL3-9B and the 32B model entirely into the appendix, so the cross-model claim rests on thinner evidence; all models are open-weight, with no closed-source strong reasoners (o-series, Gemini Thinking) included. On the control task Oxford-IIIT Pet reasoning actually lowers accuracy (79.3β†’76.2), where "every estimator degrades" cannot be fully separated from a changed risk structure caused by lower accuracy, so the consistency drop on Pet should not be read directly as evidence of conditioning. The masking subset (\(n=3900\)) is filtered by "the answer occurs in the trace", which is a selection bias: these are the most strongly conditioned samples, so the masking gain may be overestimated.
  • At the mechanism level, one piece of internal evidence is missing: every intervention acts on the context text and never touches the model's internal representations, so "answer-token likelihood is contaminated by the model's own trace" remains a behavioural inference. Adding hidden-state probes or a conditionalization measure on internal states would harden the argument.
  • Concrete improvement directions worth trying: (1) turn "suppress the answer's early appearance" into a decoding constraint or a light penalty, and see whether ATL can recover its low-cost advantage in ranking; (2) build an explicit conditioning-strength feature from when the answer first appears in the trace and how often it is repeated afterwards, and use it as a correction term for confidence; (3) train a small calibrator that consumes only trace statistics, as a substitute for consistency when multi-sampling is infeasible; (4) extend the analysis to multi-turn agents β€” Appendix K already shows that reasoning and answering repeatedly within one context amplifies conditioning further, which is exactly the most dangerous failure mode for long-horizon agents.
  • vs Semantic Entropy (Kuhn et al.): They sample multiple generations and cluster them into semantic equivalence classes with an external NLI model, capturing semantic diversity across samples; this paper analyses how token likelihood is distorted within a single conditioned generation and deliberately avoids auxiliary models, computing every signal from the model's own outputs so that multimodal comparisons stay consistent.
  • vs Devic et al. (trace length as an uncertainty signal): They propose reasoning-trace length itself as an uncertainty signal in text-only reasoning models; this paper shows that length correlates negatively with confidence, so length can neither explain confidence inflation nor replace ranking-based uncertainty estimation. The two findings point in opposite directions but are complementary: length may indicate difficulty, yet it cannot capture the failure mode of "the model being persuaded by itself".
  • vs Khan & Fu (answer consistency under prompt rephrasing): They use black-box consistency for selective VQA with non-reasoning VLMs; this paper extends consistency to reasoning-enabled VLMs and explains why it does not degrade β€” it measures agreement between answers, not commitment to the model's own trace, and is therefore intrinsically insensitive to conditioning.
  • vs LM-Polygraph and the large-scale UQ benchmarks that followed: Those works compare white-box and black-box uncertainty methods for LLMs systematically, but evaluate static, non-reasoning settings; this paper adds the dimension that reasoning systematically rewrites the reliability of those methods.
  • vs conformal prediction for VLMs (Kostumov et al.): Conformal methods assume a fixed answer set, cannot handle open-ended generation, and have no access to token-level likelihood; this paper covers open-ended generation, token-likelihood estimates, and introspective estimates.
  • vs Kadavath et al.'s P(True) and Srinivasan et al.'s selective prediction: The SRC measure here follows their introspective probing and normalization scheme, but evaluates it under ranking-based selective-generation metrics and finds mixed behaviour in multimodal, reasoning-enabled settings (benefiting on structured reasoning benchmarks, degrading on pure classification), suggesting no stable verdict yet for introspective signals on reasoning VLMs.

Rating

  • Novelty: ⭐⭐⭐⭐ First systematic account of how reasoning damages uncertainty-ranking reliability in multimodal models, with a falsifiable mechanistic attribution and interventional evidence.
  • Experimental Thoroughness: ⭐⭐⭐⭐ Four benchmarks Γ— four models Γ— nine estimators, plus matched comparisons for prompt-induced and intrinsic reasoning and a masking intervention; loses a point because the main tables only expand the 8B tier, no closed-source models are covered, and the masking subset introduces selection bias.
  • Writing Quality: ⭐⭐⭐⭐ The three contributions map one-to-one onto the experiments, and the mechanism section is organised as "exclude the competing hypothesis β†’ positive evidence β†’ interventional check"; the many figures push a lot of results into the appendix, leaving cross-model trends described only in prose.
  • Value: ⭐⭐⭐⭐ A direct deployment-level warning for any system doing abstention or selective generation on reasoning VLMs, together with a clearly usable alternative (consistency).