Skip to content

Revisiting Greedy Decoding for Visual Question Answering: A Calibration Perspective

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: Visual Question Answering, Greedy Decoding, Confidence Calibration, Stochastic Sampling, Hallucination Mitigation

TL;DR

This paper revisits decoding strategies in Multimodal LLMs from a calibration perspective, theoretically deriving sufficient conditions under which deterministic greedy decoding strictly outperforms stochastic sampling in closed-ended VQA, and proposing GDRM to decouple stochastic reasoning exploration from deterministic answer generation.

Background & Motivation

In open-ended natural language generation, stochastic sampling strategies such as Top-k, Top-p (nucleus), and Min-p have long served as the default standard for balancing text coherence and linguistic diversity. As Multimodal Large Language Models (MLLMs) rapidly advanced, empirical pipelines routinely adopted these stochastic decoding heuristics without domain- or task-specific justification. However, Visual Question Answering (VQA) fundamentally diverges from open-ended creative generation: the overwhelming majority of VQA queries are closed-ended decision problems that demand exact, factual outputs. Consequently, the true answer posterior is extraordinarily head-heavy, heavily dominated by single tokens representing numbers, colors, or boolean values.

A deeper tension arises from the differing natures of uncertainty in text versus multimodal tasks. While uncertainty in open-ended text generation is predominantly aleatoricโ€”reflecting multiple equally valid narrative trajectoriesโ€”uncertainty in VQA is predominantly epistemic, originating from missing, noisy, or ambiguous visual evidence. Under epistemic uncertainty, expanding candidate token sets through stochastic sampling does not yield plausible creative alternatives; instead, it risks sampling low-probability, poorly calibrated distractors. On ChartQA, for instance, failure analyses reveal that stochastic sampling errors are dominated by severe numeric deviations exceeding 20% (accounting for 43.4% of failures) and answer-type mismatches (8.5%), whereas semantically equivalent near-misses constitute only 2.5%, proving that sampling injects tail noise rather than helpful diversity.

Addressing the open debate on whether MLLMs should retain stochastic decoding heuristics, this paper establishes a formal theoretical bridge between model calibration metrics and predictive task accuracy. Core idea: by extending Expected Calibration Error and Brier Score to strategy-specific distributions, the paper theoretically proves that when top-1 predictions exhibit high confidence and calibration while tail tokens remain poorly calibrated, greedy decoding is strictly optimal, motivating GDRM to combine stochastic reasoning traces with deterministic answer anchoring.

Method

Overall Architecture

The framework provides both a diagnostic theoretical pipeline and a practical decoding mechanism tailored for closed-ended multimodal question answering. Given an image-question pair \((I, x)\), the system characterizes the model's posterior \(q(\cdot|I, x)\) through strategy-parameterized calibration metrics; when the derived sufficient conditions hold, deterministic greedy decoding is applied to eliminate ungrounded tail distractors. For reasoning MLLMs (such as Qwen3-VL-Thinking), the architecture decouples the generation flow: it allows stochastic sampling or beam search during the internal chain-of-thought exploration phase to maintain reasoning search capacity, but strictly switches to greedy decoding for the final answer tokens to enforce grounded convergence.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Input (I, x)<br/>Image & Closed-ended Question"] --> B["Strategy-Specific Calibration Measurement<br/>Compute ECEฮฑ and BSฮฑ Metrics"]
    B --> C{"Greedy Optimality Conditions<br/>Verify Gฮฑ1 โ‰ฅ 0 or Gฮฑ2 โ‰ฅ 0"}
    C -->|Standard MLLMs Certified| D["Deterministic Greedy Decoding<br/>Output Calibrated Top-1 Token"]
    C -->|Reasoning MLLM Deliberation| E["GDRM Decoupled Decoding<br/>Sample Reasoning Trace r then Greedy Answer ลท"]
    D --> F["Accurate Grounded Final Answer<br/>Suppress Low-Probability Hallucinations"]
    E --> F

Key Designs

1. Strategy-Specific Calibration Measurement: Addressing the Unobservable Posterior in Strategy Diagnostics

Measuring the impact of candidate selection parameter \(\alpha\) (such as \(k\) in Top-k or \(p\) in Top-p) is non-trivial because the ground-truth distribution \(p(a|I, x)\) is not continuously observable. The paper generalizes standard classification calibration to strategy-specific metrics \(\text{ECE}_\alpha\) and \(\text{BS}_\alpha\) defined directly over strategy sampling distribution \(q_\alpha\):

\[\text{ECE}_\alpha := \mathbb{E}_{I, x, a \sim q_\alpha(\cdot|I, x)} \left[ |q(a|I, x) - p(a|I, x)| \right]\]

To circumvent numerical instability where the classical Brier Score denominator \(p(a|I, x)\) may vanish on tail candidates, the design substitutes the denominator with the model's own positive likelihood \(q(a|I, x)\), yielding a mathematically tractable formulation:

\[\text{BS}_\alpha := \mathbb{E}_{I, x, a \sim q_\alpha(\cdot|I, x)} \left[ \frac{|q(a|I, x) - p(a|I, x)|^2}{q(a|I, x)} \right]\]

This normalization remains well-defined over truncated candidate spaces and provides the exact algebraic foundation required to establish lower bounds on prediction accuracy.

2. Greedy Optimality Conditions: Formulating Theoretical Guarantees for Deterministic Superiority

To analytically prove when greedy selection surpasses stochastic alternatives, the authors define strategy correctness as \(C_\alpha := \mathbb{E}_{I, x, a \sim q_\alpha}[p(a|I, x)]\), where \(a_1\) represents the top-1 argmax prediction. By bounding \(C_\alpha\) and \(C_1\) via calibration errors, the paper establishes that greedy decoding is provably optimal (\(C_1 \ge C_\alpha\)) if either of the following sufficient conditions holds for all non-greedy \(\alpha\):

\[\mathcal{G}_1^\alpha := \mathbb{E}_{I, x, a \sim q_\alpha} \left[ q(a_1|I, x) - q(a|I, x) \right] - \text{ECE}_1 - \text{ECE}_\alpha \ge 0\]
\[\mathcal{G}_2^\alpha := \mathbb{E}_{I, x, a \sim q_\alpha} \left[ q(a_1|I, x) - \frac{1 + q^2(a|I, x)}{2q(a|I, x)} + \frac{\text{BS}_\alpha}{2} \right] - \text{ECE}_1 \ge 0\]

These conditions show that whenever the model exhibits high confidence in its top candidate while preserving low calibration error on \(a_1\) relative to the tail, greedy decoding cannot be outperformed by sampling. In practical deployment, practitioners can estimate \(\mathcal{G}_1^\alpha\) pre-hoc using a tiny labeled development set (e.g., 200 samples) to verify optimality before running full-scale inference.

3. GDRM Decoupled Decoding: Bridging Generative Reasoning Exploration with Deterministic Answer Anchoring

In thinking MLLMs, fully deterministic decoding throughout the entire context has historically been viewed as harmful because stochastic exploration is believed to facilitate diverse reasoning paths. However, empirical findings show that token confidence naturally rises along valid reasoning trajectories, resulting in strong confidence when models reach their conclusions. To resolve this dilemma, the authors propose Greedy Decoding for Reasoning Models (GDRM):

\[r \sim \text{SAMPLING}\left(p_\theta(\cdot | I, x)\right), \quad \hat{y} = \text{GREEDY}\left(p_\theta(\cdot | I, x + r)\right)\]

GDRM bifurcates generation: the reasoning trace \(r\) is generated under standard stochastic sampling or beam search to explore chain-of-thought deductions, while the final answer tokens \(\hat{y}\) are strictly generated via greedy argmax conditioned on \(I, x + r\). This prevents late-stage stochastic tail drift and eliminates order-of-magnitude slips or erroneous abstentions.

A Worked Example

Consider a numerical chart question on ChartQA requiring trend extraction: 1. Input & Task: An image displaying a multi-line trend chart is queried with "What is the inflation rate in 2021?". 2. Failure of Stochastic Sampling: Under Top-k (\(k=20, \tau=1.0\)) or Top-p (\(p=0.95\)), the model's vocabulary distribution at the answer token assigns probability \(0.86\) to 2.6 and \(0.07\) to 26. Stochastic sampling samples the secondary token, causing an order-of-magnitude hallucination (26000), or drops into an ungrounded categorical mismatch (Yes). 3. Execution under GDRM: Qwen3-VL-Thinking generates reasoning tokens \(r\) via sampling to trace axis bounds and coordinates. Once the thought concludes, GDRM enforces GREEDY decoding on the answer prefix, forcing argmax selection on 2.6, thereby outputting the ground-truth prediction without tail distortion.

Key Experimental Results

Main Results

The method was evaluated across MMMU, ChartQA, and BLINK using Qwen2.5-VL (3B and 7B) and LLaVA-v1.5 (7B). Below are representative accuracy figures at default \(\tau=1.0\) (averaged over 4 random seeds):

Dataset Model Greedy (Ours) Top-p (nucleus) Top-k Min-p Greedy Advantage
MMMU Qwen2.5-VL (3B) 45.44% 41.30 ยฑ 1.14% 41.08 ยฑ 0.87% 40.20 ยฑ 0.96% +4.14%
ChartQA Qwen2.5-VL (3B) 83.12% 78.93 ยฑ 0.47% 76.11 ยฑ 0.26% 76.09 ยฑ 0.49% +4.19%
BLINK Qwen2.5-VL (3B) 35.32% 31.67 ยฑ 0.74% 30.82 ยฑ 0.96% 30.98 ยฑ 0.75% +3.65%
MMMU Qwen2.5-VL (7B) 52.56% 46.33 ยฑ 0.92% 46.00 ยฑ 0.76% 44.11 ยฑ 0.95% +6.23%
ChartQA Qwen2.5-VL (7B) 82.13% 79.69 ยฑ 0.44% 77.92 ยฑ 0.35% 71.56 ยฑ 1.76% +2.44%
BLINK Qwen2.5-VL (7B) 41.56% 37.45 ยฑ 0.68% 35.61 ยฑ 0.82% 30.30 ยฑ 1.08% +4.11%

Across 27 experimental configurations, greedy decoding outperformed the best stochastic strategy in 24 settings, with performance margins exceeding random perturbation standard deviations by up to 18.8ร—.

Ablation Study

The table below showcases the impact of applying GDRM on Qwen3-VL-Thinking (4B), comparing baseline stochastic reasoning against GDRM-enhanced decoding:

Sampling Baseline MMMU (Baseline) MMMU (+GDRM) ChartQA (Baseline) ChartQA (+GDRM) BLINK (Baseline) BLINK (+GDRM)
Temperature Only (\(\tau=1.0\)) 53.22% 54.44% (+1.22) 80.94% 81.09% (+0.15) 46.08% 48.29% (+2.21)
Top-k (\(k=20\)) 54.11% 54.78% (+0.67) 80.78% 81.46% (+0.68) 44.77% 47.71% (+2.94)
Top-p (\(p=0.95\)) 54.56% 54.67% (+0.11) 81.56% 81.82% (+0.26) 45.13% 45.98% (+0.85)
Min-p 53.78% 55.33% (+1.55) 77.71% 78.02% (+0.31) 45.29% 45.40% (+0.11)
Beam Search 59.78% 60.92% (+1.14) 81.51% 81.56% (+0.05) 47.40% 48.39% (+0.99)
Standard Greedy Baseline 54.11% โ€” 81.46% โ€” 45.71% โ€”

Additionally, on the MM-HallBench hallucination benchmark, Qwen2.5-VL (3B) with Greedy decoding achieved an Answer Score of 3.64 and a reduced Hallucination Rate of 34% (versus Top-k at 3.38 and 42%), corroborating that greedy decoding directly suppresses object and attribute hallucinations.

Key Findings

  • Temperature Inversion Curve: Increasing temperature \(\tau\) from 0.3 to 2.0 triggers steep performance drops across all stochastic variants; as \(\tau \to 0\), stochastic methods monotonically converge toward greedy performance.
  • Extreme Top-1 Concentration: Rank-wise analysis on ChartQA shows that normalized confidence at rank 1 averages 87.56% with an empirical correctness of 86.17%. Correctness collapses to 7.39% at rank 2 and 0.72% at rank 5, indicating that non-top-1 candidates are overwhelmingly ungrounded distractors.
  • Pre-hoc Diagnostic Validity: Estimated \(\mathcal{G}_k^1\) on a 200-sample validation split was positive for all \(k \in \{2, 3, 4, 5\}\) (e.g., \(0.0470\) for \(k=2\) and \(0.2752\) for \(k=5\)), accurately predicting greedy superiority on unseen test splits without empirical trial-and-error.

Highlights & Insights

  • Challenging Inherited Assumptions: Dispels the common belief that MLLMs must inherit stochastic decoding from text LLMs, revealing how epistemic uncertainty turns sampling diversity into noisy distraction.
  • Actionable Pre-hoc Calibration Metric: Formulates a strategy-dependent Brier Score and ECE variant that allows certifying decoding strategies using small development sets.
  • Plug-and-Play GDRM Hybridization: Demonstrates that reasoning exploration and final answer precision do not have to compete, delivering immediate gains across thinking models without parameter fine-tuning.

Limitations & Future Work

  • Scope Constrained to Closed-ended VQA: The theoretical derivation assumes unique ground-truth posterior targets; extending formal guarantees to open-ended descriptive tasks like detailed image captioning remains open.
  • Language and Domain Diversity: Benchmark evaluations are currently English-centric, leaving medical or multilingual multimodal tasks for future validation.
  • Search-Based Optimality Not Covered: Theoretical bounds apply to parameterized single-path sampling strategies (\(\alpha\) family) and do not formalize dominance over self-consistency or beam search ensembles.
  • vs Visual Contrastive Decoding (VCD): VCD mitigates hallucinations by subtracting perturbed image distributions; greedy decoding operates orthogonally at the decision boundary, enhancing correctness whether contrastive reweighting is present or not.
  • vs Dynamic Truncation (Min-p / \(\eta\)-Sampling): Dynamic truncation shrinks candidate sets based on entropy; this paper proves that in VQA, even truncated non-argmax tokens remain poorly calibrated, making full greedy reduction the optimal limit.

Rating

  • Novelty: โญโญโญโญโ˜† Rigorous theoretical treatment of decoding via calibration in multimodal contexts.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluation covering 6 benchmarks, multiple architectures, seed variance, and human verification.
  • Writing Quality: โญโญโญโญโญ Lucid formulation connecting statistical decision theory to practical inference bottlenecks.
  • Value: โญโญโญโญโญ Highly practical guidance that saves inference compute while systematically improving VQA accuracy.