Seeing to Ground: Visual Attention for Hallucination-Resilient MDLLMs¶

Conference: ECCV 2026
Paper: ECCV
Area: Multimodal VLM
Keywords: multimodal hallucination, diffusion language model, parallel masked decoding, cross-attention, training-free decoding

TL;DR¶

This paper attributes hallucination in multimodal diffusion large language models (MDLLMs) to an objective mismatch in parallel masked decoding — the decoder ranks candidate tokens purely by textual likelihood and never verifies localized visual support — and proposes VISAGE, a training-free decoding-time re-ranking framework that estimates "grounding evidence" from the spatial entropy of last-layer cross-attention over image tokens, aggregates it across heads via a β-quantile consensus into a penalty multiplier, and replaces the ranking score \(c_i\) with \(c_i(1+H_i)^{-\alpha}\), gaining 8.59% relative on MMMU-val and 7.75% on HallusionBench while barely hurting general capability.

Background & Motivation¶

Multimodal diffusion large language models (MMaDA, LLaDA-V, Lumina-DiMOO, and others) replace autoregressive token-by-token generation with parallel masked decoding: a single forward pass proposes candidate tokens for many positions at once, which cuts inference latency. That architectural efficiency does not address hallucination, where the model asserts objects, relations, or attributes that the image does not contain. Prior explanations for these failures mostly land on "model capability" — insufficient multimodal alignment during pretraining, limited capacity, imbalanced data mixtures — so the dominant mitigation strategies are either training-time alignment or inference-time intervention. The inference-time branch splits into two families: external verification (Woodpecker, MARINE and similar), which calls in auxiliary object detectors to re-check content after generation at the cost of multi-pass latency; and internal calibration, which uses the base model's own representations for contrastive penalties, suppression of particular attention heads, or information-bottleneck constraints. Both families share one presupposition — they operate inside the sequential next-token prediction paradigm. Meanwhile recent training-free decoding methods on the diffusion-language-model side (CORE remasks unreliable committed tokens, DyStruct adapts generation structure and length during decoding) work purely on text and never look at the image at all.

The root of the tension sits in the decoding algorithm itself. Each step of parallel unmasking is an implicit position-wise optimization: given a token budget \(k_t\), the decoder must pick \(k_t\) of the masked positions to commit, and it picks them by confidence, i.e. top-\(k\). That confidence only says how plausible a word is under the current context; it cannot measure whether the candidate has any image support. In other words, what the decoder actually maximizes at every step is a proxy objective containing only a language term, and greedily following it drives the model into "language shortcuts" — prematurely locking in the statistically smoothest token. The damage is compounded by the parallel setting: a prematurely fixed wrong token immediately becomes conditioning context for every subsequent position, cascading through the whole trajectory. The paper backs this reading with a direct measurement: plotting the normalized peak probability mass of the visual and language components across diffusion steps shows that for a grounded commitment the visual peak exceeds the language prior before the commit step, whereas under a hallucinated commitment the visual distribution stays flat and its peak remains suppressed below the language prior — the ranking score never reflected whether visual evidence was concentrated.

If the problem lives in the ranking score, there is no need to touch model parameters. Core idea: recast hallucination as a position-wise "proxy discrepancy" \(b_i \ge 0\) (how far textual likelihood deviates from the true multimodal objective), use the spatial Shannon entropy of cross-attention over image tokens as a computable estimator of that discrepancy, aggregate it across attention heads with a β-quantile consensus so that a single spuriously sharp head cannot fake grounding, and feed a monotone multiplier into a re-ranked TopK commitment — pulling the parallel decoding objective back toward visual grounding with no training whatsoever.

Method¶

Overall Architecture¶

The input is an image \(v\) and a text prompt \(x\) (the prompt keeps a <think> slot to elicit intermediate reasoning); the output is a response sequence. The frozen MDLLM initializes all \(L\) response positions as [MASK] and unmasks them over \(T\) decoding steps, at each step selecting a subset of positions under budget \(k_t\) and committing their candidate tokens. VISAGE touches no model weights; it inserts one re-ranking layer at the decision of "which positions to commit this step". The forward pass already emits both the candidate token and confidence \(c_i\) for every masked position, and already computes cross-attention, so VISAGE simply reuses those two quantities to derive a grounding evidence signal, obtains a corrected ranking score, and commits by that corrected score. In one sentence: it replaces "top-K by confidence" with "top-K by confidence × grounding multiplier", leaving everything else untouched.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["image + prompt<br/>all response slots set to [MASK]"] --> B["frozen MDLLM forward<br/>candidate tokens + confidence c"]
    B --> C["objective-mismatch formulation<br/>hallucination = high c, large bias"]
    C --> D["spatial entropy estimation<br/>per-head Shannon entropy"]
    D --> E["β-quantile consensus<br/>the ⌈βM⌉-th smallest head entropy"]
    E --> F["monotone reweighted commitment<br/>u = c·(1+H)^−α, take TopK"]
    F -->|budget unfilled, next decoding step| B
    F --> G["output sequence"]

Key Designs¶

1. Objective-mismatch formulation: writing hallucination as a position-wise proxy discrepancy

Standard decoding is written as follows: at step \(t\) the set of masked positions is \(\mathcal C_t\), each position \(i\)'s candidate token is obtained greedily and its confidence \(c_i\) is the probability mass of that candidate; the decoder must select a commitment set \(U_t \subseteq \mathcal C_t\) of exactly size \(k_t\), and its selection rule maximizes \(\sum_{i\in U}\log c_i\). Because the objective is separable across positions, this is equivalent to taking top-\(k_t\) by \(c_i\). The paper's observation is that this expression contains no visual correspondence term at all. An intended multimodal objective should combine textual likelihood with localized visual correspondence, \(r^{\star}(i)=\log c_i+\lambda r_{\mathrm{vis}}(i)\) (with \(\lambda>0\) scaling the grounding strength); but an exact \(r_{\mathrm{vis}}\) requires an expensive oracle that checks each candidate against image regions, which would erase the efficiency advantage of parallel decoding, so standard algorithms necessarily collapse to \(\log c_i\) alone. The gap between the two is the proxy discrepancy \(b_i\) — it measures the extent to which textual probability mass overrides localized visual evidence. Once hallucination is rewritten as "the candidate positions with large \(b_i\)", the problem turns from "the whole sequence failed to generate" into "a localized optimization error at a handful of positions", and every design below only needs to penalize those positions rather than fix the model. (⚠️ In the original equations (5)(6)(7) the sign relation between \(b_i\) and \(\lambda r_{\mathrm{vis}}\) is not self-consistent in the extracted PDF text; only the conceptual meaning is kept here — a larger \(b_i\) means the position is more likely ungrounded; refer to the original paper for exact signs.)

2. Spatial entropy estimation: using the concentration of cross-attention as visual grounding evidence

Correcting \(b_i\) requires a computable estimator \(\hat b_i\). The paper does not assume attention weights equal the causal influence of visual features; instead it treats the normalized cross-attention spatial distribution as a proxy for "where the visual evidence sits". For each masked text position \(i\), take the last-layer attention \(A^{(t,h)}_{i,j}\) from head \(h\) to image token \(j\in\mathcal I_{\text{img}}\), and renormalize over image tokens first (the raw attention is also spread over the prompt and already-generated tokens, and a numerical smoothing constant \(\delta>0\) guards against division by zero):

\[\tilde A^{(t,h)}_{i,j}=\frac{A^{(t,h)}_{i,j}+\delta/N}{\sum_{j'\in\mathcal I_{\text{img}}}\big(A^{(t,h)}_{i,j'}+\delta\big)},\qquad j\in\mathcal I_{\text{img}}\]

(⚠️ Equation (8) is corrupted in the cached text; this is a reconstruction consistent with the surrounding derivation — refer to the original paper. \(N\) is the number of image tokens.) The Shannon entropy of this spatial distribution follows, \(H^{(t,h)}_i=-\sum_{j\in\mathcal I_{\text{img}}}\tilde A^{(t,h)}_{i,j}\log\tilde A^{(t,h)}_{i,j}\). Low entropy means attention concentrates on a few image patches, so the candidate token has explicit localized visual support; high entropy means attention spreads uniformly over the image, meaning the model is only guessing from a language prior. This direction matches the empirical observation in Figure 2: grounded commitments show a visual peak above the language prior before the commit step (low entropy, localized), while hallucinated commitments keep a uniform visual distribution with a suppressed peak throughout. Entropy rather than raw attention weights is chosen because it is a scale-independent scalar that needs no trained probe and no external model, and can be computed alongside a single forward pass.

3. β-quantile consensus: grounding counts only when enough attention heads independently agree

An individual attention head occasionally collapses onto an irrelevant artifact and produces an artificially sharp, low-entropy distribution: taking the minimum head entropy would let one spurious sharp head exempt a completely ungrounded token from the penalty; conversely, when most heads are diffuse and only a few carry a genuine concentration signal, averaging dilutes that signal away. The paper sorts the \(M\) head-wise entropies in ascending order \(H_{i,(1)}\le\cdots\le H_{i,(M)}\) and takes the \(\lceil\beta M\rceil\)-th smallest value as the robust grounding entropy:

\[H_i\triangleq q_\beta\big(\{H^{(t,h)}_i\}_{h=1}^{M}\big)=H_{i,(\lceil\beta M\rceil)}\]

This quantile operator acts as a localization consensus: \(H_i\) becomes small, and the token is judged grounded, only if at least \(\lceil\beta M\rceil\) heads each exhibit low entropy. \(\beta\) thus becomes an explicit consensus threshold — \(\beta=0.5\) is the median head entropy (a majority of heads must agree) — while the experiments uniformly use \(\beta=0.25\). (⚠️ Section 4.2 illustrates the consensus meaning with \(\beta=0.5\)/median, whereas the experimental setup uses \(\beta=0.25\) throughout; the two do not match, so follow the experimental setting.) The ablation confirms the compromise is necessary: minimum entropy below mean, and mean below the quantile consensus.

4. Monotone reweighted commitment: replacing confidence ranking with a bounded-error TopK

Turning the estimated entropy into a ranking score requires a monotone multiplier: define the grounding multiplier \(g_i=1/(1+H_i)\) (larger entropy, smaller multiplier), giving the final linear ranking score

\[u_i=c_i\cdot g_i^{\alpha}=c_i\,(1+H_i)^{-\alpha}\]

where \(\alpha\ge 0\) is the penalty strength, equivalent to subtracting \(\alpha\log(1+H_i)\) in log space, i.e. it sets the penalty scale relative to the log-probability. \(\alpha\) defaults to 0.5 but drops to 0.3 on MME, because MME mixes cognition tasks such as commonsense reasoning, text translation, numerical calculation, and code reasoning, where too strong a penalty would suppress the linguistic and cognitive priors the model should keep. Since the corrected objective remains separable across candidate positions, the optimal commitment set is simply top-\(k_t\) by \(u_i\): \(U_t=\operatorname{TopK}(\{u_i\}_{i\in\mathcal C_t},k_t)\), requiring no approximate solver and no extra optimization loop. The re-ranking is stable because, with \(c_i\) fixed, \(u_i\) decreases monotonically as \(H_i\) grows, so a token is committed only if it is both confident and validated by concentrated visual evidence. The paper also provides an error guarantee: if the entropy estimator \(\hat b_i\) approximates the true discrepancy \(b_i\) within absolute error \(\varepsilon_t\), the objective loss relative to the optimal grounded subset is at most \(2k_t\varepsilon_t\). That bound comes from a worst-case rank reversal — an overestimated suboptimal token (inflated by \(\varepsilon_t\)) displaces an underestimated optimal token (deflated by \(\varepsilon_t\)), costing at most \(2\varepsilon_t\) per position over \(k_t\) positions.

A Worked Example¶

Walk through the illustration in Figure 3. The prompt is "Is there a couch in the image?", and at some decoding step the model produces candidates and confidences for five masked positions: a (0.62), couch (0.25), visible (0.48), no (0.51), . (0.31).

Ranking by plain confidence puts a first at 0.62, making it the most eligible for early commitment. VISAGE instead computes each candidate's attention spatial entropy: a spreads attention over the whole image, \(H=4.55\); couch \(H=5.25\); visible \(H=3.16\); no concentrates attention on a small image region, \(H=0.18\); . \(H=1.24\). The corresponding multipliers \(g=1/(1+H)\) are 0.18, 0.16, 0.24, 0.85, 0.44; multiplying back by confidence (with \(\alpha=0.5\), i.e. a square root) and re-ranking gives no 0.47, a 0.26, visible 0.23, . 0.20, couch 0.10. As a result no jumps to the top and is committed while a is penalized for high entropy — had a landed first, it would immediately condition every subsequent position and push the sentence toward "There is a couch…", an answer with no visual basis. (Numbers are taken from Figure 3 of the original paper.) The other example in the main text works the same way: for "Is there a cup in the image?", standard decoding gives the statistically smooth "no" a confidence as high as 0.9, whereas VISAGE uses the entropy signal to push its corrected confidence down to 0.6, blocking the premature commitment, and the model finally produces "There is a cup on the deck."

Loss & Training¶

This paper has no training objective — VISAGE works entirely at inference time, updates no parameters, introduces no extra data, and modifies no architecture; it only performs a few extra scalar computations on the already-computed attention maps before each commitment. There are only three hyper-parameters: the consensus threshold \(\beta=0.25\) (uniform across all experiments), the penalty strength \(\alpha=0.5\) by default (0.3 on MME), and the entropy stride. The penalty is multiplicative and monotone and does not alter the sampler's structure, so it drops into an existing masked decoding loop unchanged.

Key Experimental Results¶

Main Results¶

The backbone is MMaDA (standard confidence-based unmasking) and the comparison is VCD on the same backbone, which the paper adapts to the masked diffusion framework (each parallel unmasking step contrasts the logit distributions from the original image against a visually distorted counterpart). POPE and MME use string matching for F1 / Score, while HallusionBench and MMMU-val use Qwen3-8B as an external LLM-as-a-judge.

Benchmark	Metric	MMaDA (base)	+ VCD	+ VISAGE (Ours)	vs. base
MMMU-val	Acc (%) ↑	27.11	28.44	29.44	+8.59%
HallusionBench	Acc (%) ↑	34.18	34.80	36.83	+7.75%
POPE	F1 ↑	75.97	75.85	76.17	+0.26%
MME	Score ↑	1383.29	1342.21	1372.05	−0.81%

To show this is not a MMaDA-specific phenomenon, VISAGE is applied as-is to two other MDLLMs with no architecture-specific tuning:

Backbone	Method	MMMU-val ↑	MME ↑
LLaDA-V	Base	46.67	1883.57
LLaDA-V	+ VISAGE	47.11	1900.64
Lumina-DiMOO	Base	29.33	1125.23
Lumina-DiMOO	+ VISAGE	30.13	1142.16

Ablation Study¶

Config	Benchmark / Metric	Score	Note
\(\alpha=0.5\)	MME	1320.77	penalty too strong; suppresses the language priors cognition tasks need
\(\alpha=0.3\) (default for MME)	MME	1372.05	best compromise
\(\alpha=0.1\)	MME	1362.68	penalty too weak; premature commitments survive
min over heads	MMMU-val	28.56	one sharp head can fake grounding
mean over heads	MMMU-val	28.78	diffuse heads dilute the genuine signal
\(q_\beta\) quantile consensus (Ours)	MMMU-val	29.44	blocks both failure modes

Since re-ranking only reuses attention already computed in the forward pass, VISAGE's overhead is just a choice of "how often to compute the entropy". The authors introduce the entropy stride: compute the grounding signal every few steps and reuse the previous value in between.

Method	Latency (s/image) ↓	Relative overhead
MMaDA (base)	8.22	1.00×
VISAGE (every step)	10.84	1.31×
VISAGE (stride 2)	9.56	1.16×
VISAGE (stride 4)	8.58	1.04×
VISAGE (stride 8)	8.23	1.00×

Key Findings¶

The gains concentrate where language priors are most easily misused. On HallusionBench the largest improvements come from the illusion (≈9%) and map (≈12.5%) subsets, both requiring precise spatial localization, while VCD shows almost no improvement on illusion — contrastive penalties suppress generic statistical priors but cannot force the concentrated spatial verification needed for deceptive layouts. Dense structured formats (figure, math, OCR) also improve consistently, since fine-grained semantic detail is exactly what language priors tend to fabricate.
The +8.59% on MMMU-val shows the benefit is not limited to yes/no questions. In multi-step reasoning, prematurely committing an ungrounded intermediate token cascades into every later step; position-wise re-ranking anchors each step, stabilizing long-form answers.
POPE improves only 0.26%, which the authors attribute to label noise: because generation length is extended to elicit intermediate reasoning, VISAGE verifies space carefully and then identifies valid objects missing from the ground-truth annotations, or refutes incorrect ground-truth assertions — correct judgments that are nevertheless counted as errors (POPE is built on MS COCO, which is known to have missing and false-positive labels). The explanation is plausible but the paper only offers qualitative supplementary examples, so readers should read it as "the measurable gain is capped by data noise" rather than "the method is useless on POPE".
The necessity of β-quantile aggregation comes from two opposing failure modes: minimum entropy cannot resist a single spurious sharp head (28.56), while mean entropy is diluted by diffuse heads (28.78); only consensus aggregation blocks both sides (29.44).
The sensitivity to \(\alpha\) is task-dependent: on MME, 0.3 clearly beats both 0.5 and 0.1, showing visual grounding is not a universal remedy — when hallucination mainly originates on the language side (commonsense, translation, calculation), an overly strong penalty damages the cognitive priors the model should retain.
On efficiency, computing the entropy at every decoding step costs 1.31× latency, but the grounding signal varies slowly across adjacent steps, so striding brings overhead back to 1.04× (stride 4) or even 1.00× (stride 8), offering a practical accuracy–latency knob.
⚠️ The implementation details state 128 diffusion steps for MMMU-val, while the latency-table setup says 256 steps; the two conflict, so the order of magnitude is informative but the exact configuration should be taken from the original paper.

Highlights & Insights¶

Reframing hallucination as an objective mismatch of the decoding algorithm rather than a capability failure is the paper's most valuable move: it turns the question from "do we need more training" into "do we need to change the ranking score", yielding a training-free, plug-and-play method that modifies no architecture. The transferable signal is clear — any generative model using parallel/non-autoregressive decoding whose ranking score comes from a single modality carries a structurally similar mismatch.
Using the spatial entropy of attention rather than attention weights or an external detector is an economical choice: entropy is a scale-independent scalar requiring no trained probe and no external model, and it reuses intermediate results from a forward pass at near-zero cost.
The β-quantile consensus turns a vague question — "how many heads must agree before it counts" — into an explicit, tunable quantile, which is more controllable than fixed combinations such as max or mean. This trick transfers to any setting that aggregates confidence across heads, e.g. factuality probing, uncertainty estimation, or hallucination detection scoring.
Monotone reweighting plus a separable objective guarantees that "adding a regularizer" costs no optimality: because the objective stays separable across candidate positions, one TopK still suffices after the penalty, with no approximate solver or extra optimization loop. This is the engineering point that lets a complex score slide into an existing decoder.

Limitations & Future Work¶

Limitations admitted by the authors: VISAGE only handles image-to-text parallel decoding and does not explicitly model the temporal token dimension of video MDLLMs; although spatial entropy formally generalizes to spatio-temporal attention maps, the method does not enforce cross-frame consistency. This shows up empirically too — the baseline still wins on the video subset of HallusionBench, since video reasoning requires aggregating grounding across multiple frames.
Limitations I see: all evidence comes from three MDLLMs (MMaDA / LLaDA-V / Lumina-DiMOO) and four benchmarks, and the absolute gains are modest overall (+2.33 points on MMMU-val, +2.65 points on HallusionBench) with no variance report, so the effect of decoding-order randomness (unknown variance) is unquantified.
The entropy is always estimated from the last-layer cross-attention, yet there is no ablation over layer choice; Figure 2 gives only qualitative curves, with no statistics on how well attention concentration correlates with factual correctness across tasks.
Baseline coverage is narrow: only the adapted VCD is compared, with no other contemporary decoding-time interventions (inter-layer contrast, multi-head suppression, and similar autoregressive methods) ported to the masked diffusion framework.
Concrete improvements: (1) fold the temporal dimension into the entropy and constrain cross-frame attention consistency to close the video gap; (2) make \(\beta\) and \(\alpha\) sample-adaptive instead of benchmark-picked at 0.3/0.5; (3) use the entropy signal for remasking in reverse — allow committed tokens whose entropy later rises to be revoked, combining with CORE's idea.

vs VCD: this is the most directly comparable baseline; it penalizes reliance on language priors by contrasting logits from the original image against a visually degraded one, and the paper adapts it to the masked diffusion framework. The difference is the form of evidence: VCD looks at "how the distribution changes when the image is corrupted", VISAGE looks at "whether attention concentrates on the image". VISAGE wins on MMMU-val/HallusionBench, and VCD actually drops on MME (1342.21 vs. the 1383.29 base), suggesting contrastive penalties easily damage cognition tasks that need language priors, whereas an entropy-weighted penalty only punishes ungrounded positions and therefore does less collateral damage.
vs autoregressive decoding interventions such as OPERA / DoLa: these use contrastive penalties and inter-layer contrast inside autoregressive decoding, acting on the logits of sequential next-token prediction. The paper does not implement either as a baseline; it only groups them in related work as "sequential mechanisms that ignore the localized optimization error specific to parallel masked decoding". Strictly speaking, this paper does not answer whether such methods transfer to masked diffusion and achieve comparable gains. The essential difference is timing: they fix distributions token by token during generation, while VISAGE re-ranks candidates before the commitment step.
vs CORE / DyStruct: also training-free decoding methods for diffusion language models. CORE remasks already-committed but unreliable tokens and revises them after the fact; DyStruct adapts generation structure and length during decoding. Both operate purely on text and never look at the image. VISAGE differs by intervening earlier — re-ranking by visual grounding at the moment of commitment, so an ungrounded proposal is never committed rather than revoked afterwards.
vs autoregressive internal calibration methods (contrastive penalties, image-guided attention head suppression, variational information bottleneck probes): these are designed for sequential decoding and mostly rely on re-scoring already-generated tokens or extra forward passes; VISAGE uses the attention maps that parallel decoding produces at every step anyway, adding no extra forward pass and no external model.

Rating¶

Novelty: ⭐⭐⭐⭐ The reframing as objective mismatch / localized optimization error has theoretical ambition, and re-ranking by attention spatial entropy is a fresh angle; however, suppressing hallucination via attention concentration already has close analogues on autoregressive VLMs.
Experimental Thoroughness: ⭐⭐⭐ Four benchmarks + three backbones + three ablations (penalty strength, head aggregation, latency), but the absolute gains are small, no variance is reported, VCD is the only baseline, and the POPE anomaly is explained only qualitatively.
Writing Quality: ⭐⭐⭐ The motivation chain is clear and the figures land well; but the sign direction in equations (5)(6)(7) is inconsistent, \(\beta\) differs between the main text and the experiments, and the MMMU-val diffusion-step count conflicts between implementation details and the latency table, all of which the reader must sort out.
Value: ⭐⭐⭐⭐ Training-free, plug-and-play, with an accuracy–latency knob — a low-cost, reproducible improvement for any team working on diffusion language model decoding, and the line of recasting hallucination as a decoding-objective problem has room to grow.