The Mirage of Calibrated Confidence: Trajectory-Independence of Verbalized Confidence in Vision-Language Models¶
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: Vision-Language Models, Confidence Calibration, Trajectory-Independence, Trajectory Grounding, TGS-Bench
TL;DR¶
This paper demonstrates that verbalized confidence in Vision-Language Models (VLMs) is largely trajectory-independent (failing to causally reflect visual perception or reasoning steps), and introduces the Trajectory-Grounding Score (TGS-self and TGS-pair) along with TGS-Bench across 10 benchmarks to expose this critical evaluation blind spot.
Background & Motivation¶
Recent Vision-Language Models (VLMs) have attained remarkable milestones across diverse complex visual reasoning tasks, including geometric problem solving, visual mathematical reasoning, and multi-discipline understanding. Nevertheless, these models frequently yield incorrect answers while expressing unwarranted, dangerously high confidence. To bridge this gap and foster trustworthiness, research in verbalized confidence calibration has progressed rapidlyโemploying Brier-score alignment, reinforcement learning with doubt rewarding, and decoupled visual-reasoning confidence schemes. These methods achieve substantial reductions in Expected Calibration Error (ECE) and notable gains in classification discrimination (AUROC), leading the community to assume that low aggregate calibration error signifies genuine self-assessment where models critically inspect their own reasoning before asserting confidence.
However, aggregate population-level metrics obscure the true causal foundations of individual predictions. A model can readily achieve a deceptively low ECE simply by memorizing the empirical base rate of its correctness across broad task difficulty strata, without ever consulting whether its visual perception rationale was accurate or its logical chain valid for a particular query. An alarming, illustrative pathology arises in state-of-the-art calibrated models: during chain-of-thought generation, the model repeatedly expresses explicit self-doubt (e.g., "Wait, I should recheck row 4 because the middle cell is ambiguous"), revisits its calculations, arrives at an erroneous conclusion, and yet proceeds to emit an overwhelming confidence score of 0.89. The self-doubt is plain in the generated text, yet the verbalized confidence remains completely blind to it.
This discrepancy stems from a systemic phenomenon the authors define as trajectory-independence: verbalized confidence is largely uncoupled from the specific visual perception and reasoning trajectory that produced the final prediction. Existing evaluation paradigms (ECE and AUROC) measure aggregate outcome-accuracy alignment and are utterly blind to this failure mode. Even worse, current calibration fine-tuning can paradoxically aggravate this disconnect by conditioning the model on static prior patterns rather than dynamic epistemic signals. Core idea: This paper systematically uncovers and formalizes the trajectory-independence of verbalized confidence in VLMs across content substitution, token masking, and internal hesitation markers, and proposes the Trajectory-Grounding Score (TGS-self and TGS-pair) alongside the 10-benchmark suite TGS-Bench to provide the first rigorous diagnostic of trajectory-level confidence faithfulness.
Method¶
Overall Architecture¶
The framework systematically investigates and quantifies whether a VLM's verbalized confidence is causally informed by its reasoning trajectory. Given a multimodal query \((I, q)\) composed of image \(I\) and question \(q\), the VLM produces a trajectory \(\tau = (v, r, a)\) encompassing visual perception rationale \(v\), reasoning chain \(r\), and final answer \(a\), followed by an 11-way discrete confidence score \(c \in \{0, 1, \dots, 10\}\). The methodology first executes a three-stage causal audit of escalating severity (content variation, attention-level token masking, and intra-trajectory hesitation tracking). It then formalizes the Trajectory-Grounding Score (TGS) in two complementary formulations: TGS-self, which evaluates internal grounding by comparing confidence with and without trajectory access, and TGS-pair, which measures discrimination between externally constructed good and bad trajectories along decoupled visual, reasoning, and answer axes using the newly built TGS-Bench.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
IN["Multimodal Input (I, q) & Generated Trajectory ฯ = (v, r, a)"] --> D1["Stage 1: Multi-Lens Trajectory-Independence Audit<br/>Content Variation / Token Masking / Hesitation Tracking"]
D1 --> D2["Stage 2: Self-Trajectory Grounding (TGS-self)<br/>Confidence margin between full trajectory & 100% attention mask"]
D2 --> D3["Stage 3: Controlled Contrastive Benchmark (TGS-Bench)<br/>72B gold-conditioned good vs. 7B single-axis corrupted bad"]
D3 --> D4["Stage 4: Pairwise Trajectory Discrimination (TGS-pair)<br/>Signed confidence margin across vision, reasoning, answer axes"]
D4 --> OUT["Diagnostic Output: Revealing outcome-level vs. trajectory-level disconnect"]
Key Designs¶
1. Multi-Lens Trajectory-Independence Audit: Revealing confidence insensitivity via content substitution, token masking, and internal hesitation markers To examine whether confidence causally tracks trajectory quality, the authors develop three complementary causal interventions. First, content variation substitutes one trajectory axis (\(v\), \(r\), or \(a\)) with alternative candidates while holding remaining components fixed, measuring the Jensen-Shannon divergence (JSD) between resulting confidence distributions; if confidence were grounded, distributions would shift substantially, yet the vision axis displays near-zero shift with \(P(\text{JSD} \le 0.1)\) spanning 40%โ90%. Second, attention-level token masking physically prevents the confidence-generation tokens from attending to trajectory tokens across nested masking ratios (0%, 25%, 50%, 75%, 100%) without perturbing positional encodings; unexpectedly, confidence shows minimal downward drift before paradoxically surging upward at 100% full masking. Third, intra-trajectory hesitation tracking extracts self-doubt markers (e.g., "wait", "actually", "let me recheck"), comparing expected confidence immediately before the marker (\(c_{\text{pre}}\)) and after the subsequent sentence boundary (\(c_{\text{post}}\)); across eligible events, \(|c_{\text{pre}} - c_{\text{post}}| \le 0.02\) in 78.5% (Base) and 49.0% (calibrated) cases, proving that the model's confidence fails to register even its own explicit self-corrections.
2. Self-Trajectory Grounding (TGS-self): Quantifying causal reliance on self-generated reasoning To formalize the token masking intervention into a standardized diagnostic metric, the authors introduce TGS-self. This metric computes the signed expectation gap between verbalized confidence with full access to the self-generated trajectory \(\tau\) versus confidence under 100% causal attention masking \(\tau_{\text{masked}}\): $$ \text{TGS-self}(M) = \mathbb{E}{(q, I)} \left[ \mu(C \mid \tau) - \mu(C \mid \tau) \right] $$ where expected confidence }\(\mu(C) = \sum_{c=0}^{10} \frac{c}{10} P(C=c)\) is computed via normalized digit-token logits over \([0, 1]\). TGS-self spans \([-1, 1]\): positive values indicate healthy reliance on self-reasoning; near-zero values denote trajectory-blindness; and negative values reveal perverse anti-grounding, where depriving the model of its reasoning trajectory paradoxically boosts its confidence.
3. Controlled Contrastive Benchmark (TGS-Bench): Isolating single-axis flaws across 10 benchmarks While TGS-self tests whether the model inspects its trajectory, it cannot evaluate sensitivity to trajectory correctness. To enable this, the authors construct TGS-Bench across 10 diverse benchmarks spanning mathematical, geometric, logical, and multi-discipline reasoning. High-quality good trajectories \((v_{\text{good}}, r_{\text{good}}, a_{\text{good}})\) are generated by prompting Qwen2.5-VL-72B-Instruct conditioned on the ground-truth label, guaranteeing faithful perception and reasoning. Bad trajectories are synthesized by injecting an isolated defect into exactly one axis while copying the remaining two byte-for-byte: the answer axis (\(a_{\text{bad}}\)) applies deterministic rule perturbations; the vision axis (\(v_{\text{bad}}\)) employs Qwen2.5-VL-7B to falsify a single visual detail (count, coordinate, or color); and the reasoning axis (\(r_{\text{bad}}\)) injects a single logical/arithmetic error while strictly preserving the correct final answer. Preserving the correct answer in \(r_{\text{bad}}\) is crucial: it prevents the model from shortcutting process assessment by simply checking final answer validity.
4. Pairwise Trajectory Discrimination (TGS-pair): Measuring sensitivity to trajectory defects across modalities Leveraging the controlled pairs in TGS-Bench, TGS-pair measures whether substituting a good trajectory component with a defective counterpart yields an appropriate downward confidence shift: $$ \text{TGS-pair}x = \mathbb{E}) \right], \quad x \in {v, r, a} $$ with the overall score averaged across axes } \left[ \mu(C \mid x_{\text{good}}) - \mu(C \mid x_{\text{bad}\(\text{TGS-pair} = \frac{1}{3} \sum_{x \in \{v, r, a\}} \text{TGS-pair}_x\). Unlike symmetric JSD or outcome-level ECE, TGS-pair possesses clear directional polarity: positive scores reflect correct quality discrimination, zero signifies quality blindness, and negative scores uncover anti-grounding (preferring flawed trajectories). By decomposing into \(\text{TGS-pair}_{\text{vis}}\), \(\text{TGS-pair}_{\text{reas}}\), and \(\text{TGS-pair}_{\text{ans}}\), it pinpoints the exact breakdown point in cross-modal confidence grounding.
Benchmark & Evaluation Protocol¶
Evaluation spans 10 rigorous benchmarks: mathematical/geometric reasoning (DynaMath, Geo3K, MathVerse, MathVision, MathVista, WeMath), logical deduction (LogicVista), and multi-discipline visual QA (A-OKVQA, MMK-12, MMMU-Pro), supplemented by MM-Vet and GQA for transfer diagnosis. Evaluated systems include base models (Qwen3-VL-4B and 8B with verbalized prompting) and leading calibration techniques: inference-stage prompting (Verbalize, P(True), SteerConf) and training-stage alignment (ConfTuner, RLCR, and the decoupled state-of-the-art VL-Calibration). Cross-architecture generality is verified on InternVL3.5-8B. All metrics utilize exact 11-way digit-token logit readouts normalized to \([0, 1]\).
Key Experimental Results¶
Main Results: Outcome-Level Calibration vs. Trajectory-Level Grounding¶
Table 1 demonstrates the stark decoupling between conventional outcome-level calibration metrics and trajectory-level grounding across the 10-benchmark suite.
| Model Scale & Method | AUROC (โ) | ECE (โ) | TGS-self (โ) | TGS-pair (โ) |
|---|---|---|---|---|
| 4B Models | ||||
| Qwen3-VL-4B (Base) | 0.537 | 0.432 | +0.043 | +0.059 |
| VL-Calibration-4B | 0.764 | 0.116 | -0.012 | 0.053 |
| 8B Models | ||||
| Qwen3-VL-8B (Base) | 0.561 | 0.402 | +0.079 | 0.084 |
| VL-Calibration-8B | 0.768 | 0.081 | -0.018 | 0.065 |
| Cross-Architecture (8B) | ||||
| InternVL3.5-8B | 0.655 | 0.300 | +0.046 | 0.069 |
Ablation & Generator Sensitivity Analysis¶
Table 2 reports TGS-pair across 10 benchmarks when varying the trajectory generator model (evaluated on 5.3K matched records), alongside per-axis grounding breakdowns.
| Generator / Evaluation Axis | 4B Base | 4B VL-Cal | 8B Base | 8B VL-Cal | Observation |
|---|---|---|---|---|---|
| Qwen2.5-VL-72B Generator | 0.059 | 0.053 | 0.084 | 0.065 | Default generator; avoids capability-gating bias |
| Qwen2.5-VL-32B Generator | 0.057 | 0.040 | 0.081 | 0.043 | Scores consistently remain \(< 0.1\) across scales |
| Qwen2.5-VL-7B Generator | 0.047 | 0.053 | 0.069 | 0.054 | Calibrated model outperforms Base in only 1 of 6 settings |
| Vision Axis TGS-pair (vis) | 0.021 | 0.018 | 0.035 | 0.022 | Consistently the weakest axis across all setups |
| Reasoning Axis TGS-pair (reas) | 0.048 | 0.041 | 0.072 | 0.051 | Moderate sensitivity, but still largely ungrounded |
| Answer Axis TGS-pair (ans) | 0.108 | 0.100 | 0.145 | 0.122 | Grounding is heavily concentrated in answer checking |
Key Findings¶
- The Calibration Mirage & Anti-Grounding: While VL-Calibration dramatically curbs ECE (from 0.432 to 0.116 at 4B, and 0.402 to 0.081 at 8B), TGS-self flips from positive to negative (-0.012 at 4B and -0.018 at 8B). VL-Calibration exhibits negative TGS-self on 8 of 10 benchmarks at 4B and 9 of 10 at 8B. In other words, fully masking the reasoning trajectory actually increases the calibrated model's confidence, exposing a severe anti-grounding pathology where models fit dataset base-rates rather than reflecting genuine reasoning.
- Vision Grounding is the Primary Bottleneck: Across all model families and scales, the vision axis is uniformly the weakest link (\(\text{TGS-pair}_{\text{vis}} \ll \text{TGS-pair}_{\text{ans}}\)). Models are virtually blind to erroneous visual statements in their trajectories, proving that verbalized confidence remains overwhelmingly dominated by language-space priors.
- Internal Epistemic Signals are Ignored: Models show stark insensitivity to their own self-doubt. In trajectories where the model repeatedly hedges or catches errors, pre- and post-hesitation confidence differences are negligible (\(|c_{\text{pre}} - c_{\text{post}}| \le 0.02\) in 78.5% of Base events), and models frequently assign top-tier confidence (9/10) to flawed self-corrected chains that culminate in wrong answers.
Highlights & Insights¶
- Exposing a Critical Community Blind Spot: The paper elegantly proves that aggregate outcome-level metrics (ECE and AUROC) can be gamed via coarse base-rate fitting, establishing that low calibration error does not imply trustworthy, grounded self-reflection.
- Rigorous Causal Control in Benchmark Design: By pairing defective reasoning trajectories with preserved correct final answers in TGS-Bench, the study surgically isolates process-reading capability from answer-matching heuristics.
- Diagnostic Foundation for Next-Generation Calibration: Identifies the failure of existing post-hoc and RL calibration techniques to penalize trajectory-independence, laying the empirical groundwork for process-supervised calibration objectives.
Limitations & Future Work¶
- Diagnostic Rather Than Mitigative: The study identifies, formalizes, and evaluates the trajectory-independence failure mode, but does not propose an explicit training loss or architectural modification to resolve it.
- Task Scope Centered on Verifiable Math/Logic: TGS-Bench predominantly encompasses reasoning benchmarks with objective ground-truth answers; extending single-axis trajectory perturbation to open-ended dialogue and free-form generation remains an open challenge.
- Generator Dependency: The construction of controlled good trajectories relies on an external high-capacity model (Qwen2.5-VL-72B); although sensitivity analyses verify robustness across generator sizes, capability gating remains an inherent factor.
Related Work & Insights¶
- vs. Text-Only Confidence Calibration (Xiong et al., 2024; Li et al., 2025; Damani et al., 2026): While prior work explored Brier alignment and RL doubt rewards in LLMs, this work demonstrates that extending them to VLMs fails to ground confidence in visual perception and reasoning chains.
- vs. Decoupled VLM Calibration (Xiao et al., 2026): VL-Calibration introduces structural tags
<vision>and<reasoning>to decouple confidence heads; this paper proves via causal attention masking that such syntactic separation is superficial and fails to enforce actual trajectory grounding. - vs. ADVICE Answer-Independence (Seo et al., 2026): ADVICE noted that LLM confidence is insensitive to answer swaps; this paper extends this discovery to multimodal models and substantially broadens it to entire visual-perceptual and reasoning trajectories.
Rating¶
- Novelty: โญโญโญโญโญ Pioneering formalization and empirical discovery of trajectory-independence in multimodal confidence estimation.
- Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation across 10 benchmarks, multiple model scales (4B/8B), and three rigorous causal axes.
- Writing Quality: โญโญโญโญโญ Crystal-clear narrative structure with seamless logical progression from empirical anomaly to formal metrics.
- Value: โญโญโญโญโญ High-impact wake-up call for safe and reliable VLM deployment in risk-sensitive applications.