Unmasking-Time Visual Calibration for Hallucination Mitigation in Multimodal Discrete Diffusion Language Models

Conference: ECCV 2026
Paper: ECCV Official Link
Code: https://github.com/THU-BPM/UVC
Area: Multimodal VLM / Hallucination
Keywords: multimodal discrete diffusion models, hallucination mitigation, activation calibration, training-free inference, attention head selection

TL;DR

To tackle object hallucination in multimodal discrete diffusion language models (multimodal dLLMs) caused by language priors during bidirectional demasking, this paper proposes Unmasking-Time Visual Calibration (UVC), a training-free framework that extracts multi-granularity contrastive activation shift vectors offline, selects visually informative heads via per-head AUC, and injects calibration signals exclusively at still-masked positions during inference with less than 1% runtime overhead.

Background & Motivation

Large vision-language models (LVLMs) typically rely on autoregressive (AR) left-to-right decoding. Despite notable achievements, AR models suffer from severe object hallucination—generating objects, attributes, or spatial relationships that do not exist in the visual input. A primary driver of this failure is an overreliance on language priors: when visual grounding is weak, statistical word co-occurrence dominates the output, which is then amplified along the causal token chain. Recently, discrete diffusion language models (dLLMs, such as LLaDA, Lumina-DiMOO, and MMaDA) have emerged as an alternative generation paradigm that replaces autoregressive generation with iterative demasking under full bidirectional attention. While bidirectional attention avoids sequential error cascading, it creates a new structural vulnerability: the mask configuration and length influence the representation space globally from the very first step, allowing hallucination-inducing signals to pervade all unresolved positions simultaneously.

Critically, existing hallucination mitigation methods designed for autoregressive models are fundamentally incompatible with multimodal dLLMs. Contrastive decoding strategies such as VCD require a parallel forward pass on a visually degraded input at every step, doubling inference latency; more severely, because discrete diffusion models unmask tokens based on dynamic prediction confidence, clean and degraded inputs produce drastically diverging unmasking trajectories—empirical measurements reveal that 84.7% of positions are unmasked at entirely different steps, causing step-wise contrastive alignment to collapse. Furthermore, causal prefix penalty methods such as OPERA and sequential training-time preference alignment cannot be applied to simultaneous bidirectional demasking without monotonic prefixes.

To overcome these structural roadblocks, this work inspects internal representations within attention heads: factual and perceptual grounding signals can be isolated and steered via activation interventions without retraining or parallel forward passes. The core requirement is that calibration must be step-aligned and mask-aware, operating solely on unresolved masked tokens to preserve committed semantics. The core idea is to extract multi-granularity contrastive activation shift vectors offline, identify visually sensitive attention heads via per-head AUC ranking, and inject the pre-computed calibration vectors exclusively at still-masked positions during iterative demasking, mitigating hallucinations natively with zero extra forward passes.

Method

Overall Architecture

UVC (Unmasking-Time Visual Calibration) is a lightweight, training-free framework natively tailored to the iterative demasking dynamics of multimodal dLLMs. It decouples the computationally demanding contrastive analysis and head selection to an offline preparation phase, while during online inference it only performs an in-place constant vector addition on masked positions. The pipeline comprises three stages: 1. Contrastive Activation Shift Extraction (Offline): Constructing paired global-scale (full-image Gaussian degradation) and instance-scale (bounding-box localized Gaussian degradation) contrastive images to compute mask-averaged activation differences before multi-head attention (MHA) output projections, capturing representation shift vectors when visual evidence is lost. 2. Head Selection (Offline): Fitting lightweight logistic regression classifiers on each attention head's mask-averaged activations, evaluating discriminative power between clean and degraded inputs via 2-fold cross-validated AUC, and identifying the top-\(K\) heads for global and instance scales independently. 3. Mask-aware Intervention (Online): During iterative demasking at step \(s\), injecting the scaled pre-computed shift vectors into the selected attention heads exclusively at still-masked positions \(t \in \Omega_s\) before the output projection, leaving resolved tokens unperturbed.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    subgraph OfflinePrep["Offline Preparation Stage"]
        direction TB
        A["Image Samples & Contrastive Degradation<br/>Global Gaussian noise / Bounding-box instance noise"] --> B["Contrastive Activation Shift Extraction<br/>Extract mask-averaged activation diff before MHA projection"]
        B --> C["Head Selection<br/>2-fold cross-validated AUC to identify Top-K heads"]
    end
    subgraph OnlineInference["Online Demasking Inference"]
        direction TB
        D["Input Image V + Prompt X + Initial Mask M"] --> E["Step s Bidirectional Attention Forward Pass"]
        C -.->|Inject pre-computed shift vector S| F["Mask-aware Intervention<br/>Inject calibration only at unresolved t in Ω_s"]
        E --> F
        F --> G["Project to Residual Stream & Vocabulary Logits"]
        G --> H{"Unmask highest-confidence subset<br/>Update mask set Ω_{s+1}"}
        H -- Still masked --> E
        H -- Reach final step S --> I["Faithful Output Text"]
    end
    OfflinePrep --> OnlineInference

Key Designs

1. Multi-Granularity Contrastive Activation Shift Extraction: Disentangling Layout from Fine-Grained Attributes
To derive a calibration signal that steers the model away from text-only priors, one must characterize how internal activations shift when visual features are corrupted. Because full-image degradation alters spatial layout while leaving local textures partially recognizable, and localized degradation damages object-specific details without disrupting scene context, UVC constructs contrastive pairs at two complementary granularities using 500 images from COCO train. For global contrast, an image \(V_i\) is paired with a forward Gaussian-diffused image \(V_i'\). For instance contrast, Gaussian noise is injected strictly inside the ground-truth bounding box of queried object \(O_i\), generating localized degraded image \(V_i''\). Using a forward pre-hook on the output projection matrix \(W_{o,h}^{(l)}\), the mask-averaged activation vector across all unresolved positions \(\Omega_i\) is recorded for head \(h\) at layer \(l\): $$ \mathbf{a}^{(i)}{c}(l,h) = \frac{1}{|\Omega_i|}\sum} $$ Averaging differences across }\mathbf{h}^{(i)}_{c,t}(l,h),\quad c\in{\text{clean}, \text{glob}, \text{inst\(n\) samples yields global shift vectors \(\mathbf{S}_{\text{glob}}^{(l,h)}\) and instance shift vectors \(\mathbf{S}_{\text{inst}}^{(l,h)}\), defining the exact geometric directions from degraded representations back to visually grounded states.

2. AUC-Guided Head Selection: Specialization Discovery and Sparse Intervention
Across all \(L \times H\) attention heads, many specialize in syntactic parsing, punctuation, or positional tracking rather than visual perception. Perturbing all heads uniformly introduces severe noise and degrades linguistic coherence. UVC trains an independent logistic regressor \(f^{(l,h)}\) for each head on the mask-averaged vectors to classify whether an activation originates from a clean or degraded image. Evaluating discriminability via 2-fold cross-validated AUC allows ranking all heads and selecting the top-\(K\) heads into global set \(\mathcal{S}_{\text{glob}}\) and instance set \(\mathcal{S}_{\text{inst}}\). Analysis demonstrates that these two sets are functionally segregated: global-scale heads reside primarily in middle layers (layers 14 to 20, consolidating spatial layout), whereas instance-scale heads concentrate in deeper layers (layers 21 to 26, refining fine-grained object semantics). Restricting intervention to at most \(2K\) heads prevents corrupting non-visual linguistic structures.

3. Mask-Aware Online Intervention: Concentrating Calibration on Unresolved Tokens with Zero Extra Forward Passes
During iterative demasking, the set of masked tokens \(\Omega_s\) shrinks monotonically. Altering representations across the entire sequence would overwrite already committed tokens, causing vocabulary instability. UVC introduces a position-aware indicator function \(\mathbb{I}_{\text{glob},h,t}^{(l,s)}\) and \(\mathbb{I}_{\text{inst},h,t}^{(l,s)}\), which evaluates to 1 if and only if head \((l,h)\) is selected and token position \(t\) remains masked (\(t \in \Omega_s\)): $$ \mathbf{H}t^{(l+1)} = \mathbf{H}_t^{(l)} + \sum}^{H}\Bigl(\mathbf{o{h,t}^{(l)} + \mathbb{I}},h,t}^{(l,s)}\,\lambda_{\text{glob}}\,\mathbf{S{\text{glob}}^{(l,h)} + \mathbb{I}},h,t}^{(l,s)}\,\lambda_{\text{inst}}\,\mathbf{S{\text{inst}}^{(l,h)}\Bigr)\mathbf{W} $$ Because shift vectors }^{(l)\(\mathbf{S}\) are static pre-computed constants, injection requires only an in-place vector addition per selected head, bypassing any redundant forward pass and incurring under 1% latency overhead. As generation progresses, calibration naturally focuses on the remaining uncertain positions where language prior drift is most severe.

A Worked Example

Consider an inquiry asking whether a potted plant exists in an image: - Vanilla model: In step 1 of demasking, driven by language priors in a fully masked sequence, the model prematurely predicts the end punctuation . (conf=0.790) and negative token No (conf=0.562). Bound by bidirectional attention to satisfy the committed No, the model subsequently hallucinates a contradictory description: No, there is a potted plant on the bathroom shelf in the image. - UVC model: In step 1, calibration vectors injected into masked positions steer attention heads toward visual grounding. The model first unmasks salient object tokens plant (conf=0.771) and a (conf=0.596). Guided by these grounded anchor tokens, subsequent demasking steps correctly produce: Yes, there is a potted plant in the middle of the image.

Key Experimental Results

Main Results

On the POPE object hallucination benchmark across MS COCO, GQA, and A-OKVQA under Random, Popular, and Adversarial negative sampling splits (27,000 total question-answer pairs), UVC was evaluated against Vanilla, UVC-Global, UVC-Instance, and UVC-Both across MMaDA and Lumina-DiMOO.

Model	Dataset & Split	Vanilla Acc	Vanilla F1	UVC-Global F1	UVC-Instance F1	UVC-Both Acc (\(\Delta A\))	UVC-Both F1 (\(\Delta F\))
MMaDA	COCO Random	66.57	74.81	83.24	81.22	79.83 (+13.26)	82.16 (+7.35)
MMaDA	COCO Popular	61.73	72.10	81.06	79.85	76.40 (+14.67)	80.21 (+8.11)
MMaDA	COCO Adversarial	58.00	70.24	76.54	75.40	71.57 (+13.57)	75.79 (+5.55)
MMaDA	GQA Random	64.53	73.75	83.64	80.73	75.17 (+10.64)	82.81 (+9.06)
MMaDA	GQA Popular	61.53	72.04	79.14	79.51	70.30 (+8.77)	80.77 (+8.73)
MMaDA	GQA Adversarial	58.13	70.29	75.80	75.36	68.50 (+10.37)	76.21 (+5.92)
MMaDA	A-OKVQA Random	63.23	73.06	82.44	79.63	77.33 (+14.10)	81.03 (+7.97)
MMaDA	A-OKVQA Popular	60.03	71.29	78.88	78.79	74.27 (+14.24)	79.94 (+8.65)
MMaDA	A-OKVQA Adversarial	56.13	69.34	73.12	74.04	69.80 (+13.67)	74.54 (+5.20)
Lumina-DiMOO	COCO Random	73.70	64.95	70.72	68.88	78.40 (+4.70)	73.74 (+8.79)
Lumina-DiMOO	COCO Popular	73.47	64.75	70.55	68.67	78.00 (+4.53)	73.39 (+8.64)
Lumina-DiMOO	COCO Adversarial	72.37	63.81	69.27	67.45	75.50 (+3.13)	71.48 (+7.67)
Lumina-DiMOO	GQA Random	75.20	68.45	71.82	72.01	79.00 (+3.80)	75.79 (+7.34)
Lumina-DiMOO	GQA Popular	71.20	65.13	69.99	68.27	73.73 (+2.53)	71.41 (+6.28)
Lumina-DiMOO	GQA Adversarial	70.27	64.26	69.41	67.59	73.10 (+2.83)	71.17 (+6.91)
Lumina-DiMOO	A-OKVQA Random	78.20	73.15	77.96	76.02	81.90 (+3.70)	79.73 (+6.58)
Lumina-DiMOO	A-OKVQA Popular	76.87	71.97	75.25	74.80	80.03 (+3.16)	77.97 (+6.00)
Lumina-DiMOO	A-OKVQA Adversarial	73.73	69.41	74.30	72.86	75.80 (+2.07)	74.47 (+5.06)

Ablation Study

On open-ended image captioning (500 COCO 2014 val images, max length 128 tokens over 64 demasking steps), UVC was evaluated using sentence-level \(C_I\) and instance-level \(C_S\) CHAIR metrics. Computational latency across sequence lengths was also profiled.

Model	Variant	\(C_S\) (Instance-level) \(\downarrow\)	\(C_I\) (Sentence-level) \(\downarrow\)	Avg Generated Length	Relative Hallucination Reduction
Lumina-DiMOO	Vanilla	37.40	18.14	126.1	Baseline with heavy language priors
Lumina-DiMOO	UVC-Both	21.40	12.13	125.4	\(C_S\) reduced by 42.8%, \(C_I\) by 33.1%, length preserved
MMaDA	Vanilla	35.20	15.21	62.1	Baseline
MMaDA	UVC-Both	15.40	8.78	63.1	\(C_S\) reduced by 56.3%, \(C_I\) by 42.3%, no verbosity drop

Inference latency benchmarking across generation length configurations \((N_m, S)\):

Model	Configuration \((N_m, S)\)	Vanilla Latency (ms)	UVC-Both Latency (ms)	Slowdown Factor
Lumina-DiMOO	(16, 8)	6552	6560	1.00× (negligible)
Lumina-DiMOO	(64, 32)	26610	26696	1.00× (negligible)
Lumina-DiMOO	(128, 64)	55118	55445	1.01× (< 1% overhead)
MMaDA	(16, 8)	2094	2116	1.01× (< 1% overhead)
MMaDA	(64, 32)	4247	4295	1.01× (< 1% overhead)
MMaDA	(128, 64)	9118	9206	1.01× (< 1% overhead)

Key Findings

• Granularity synergy across perception tasks: On MME perception scores, UVC-Both achieves total scores of 475.00 on Lumina-DiMOO (+50.00) and 546.67 on MMaDA (+53.34). UVC-Global excels on spatial layout errors (Position score improved by 22.2% on Lumina), while UVC-Instance excels on entity presence and count (Count score improved by 46.5% on MMaDA). Combining both delivers the highest overall performance.
• Demasking trajectory divergence explains AR failure on dLLMs: Tracking unmasking dynamics across 500 images demonstrates that the Jaccard similarity of unmasked token sets between clean and blurred inputs decays sharply to near zero within early steps, with 84.7% of positions unmasked at completely mismatched steps. This empirical finding explains why step-wise AR contrastive decoding (e.g., VCD) fails on discrete diffusion models.
• Robust hyper-parameter behavior and superiority over random heads: Grid search across intervention strength \(\lambda\) and head budget \(K\) demonstrates broad stability, with \(K \in [10, 20]\) yielding peak gains. AUC-selected heads consistently outperform randomly selected heads across all \(K\), confirming that visual steering relies on functionally specialized attention circuits.

Highlights & Insights

• Zero-overhead offline-online decoupling: Shifting contrastive pair generation and head selection offline enables online calibration via simple in-place tensor additions, curbing hallucinations by over 40% with under 1% latency increase.
• Unveiling demasking divergence in discrete diffusion: Providing the first rigorous empirical quantification of token unmasking divergence (84.7% positional mismatch) between clean and degraded inputs, establishing why AR contrastive decoding cannot be naively transferred to dLLMs.
• Hierarchical visual perception circuits: Discovering that middle layers (layers 14–20) encode coarse spatial topology while deep layers (layers 21–26) refine fine-grained object attributes, offering valuable architectural insights for multimodal diffusion interpretability.

Limitations & Future Work

• Reliance on Gaussian degradation: Shift extraction currently relies on forward Gaussian diffusion noise; investigating semantically informed counterfactual perturbations or targeted segment erasures could capture richer visual nuances.
• Requirement for internal activation access: Because UVC intervenes directly before attention output projections, it is applicable only to open-weight models (e.g., LLaDA, Lumina-DiMOO, MMaDA) and cannot be deployed on black-box API models.
• Static vs. dynamic shift vectors: Pre-computed shift vectors are averaged across a static training set, lacking dynamic adaptive weighting based on complex multi-turn conversational contexts.

Related Work & Insights

• vs Visual Contrastive Decoding (VCD): VCD executes parallel degraded forward passes at each step to contrast logit distributions, doubling latency and failing under dLLMs due to 84.7% unmasking step misalignment. UVC injects pre-computed shift vectors natively into still-masked positions during a single forward pass, retaining \(1.00\times\text{–}1.01\times\) inference speed.
• vs OPERA: OPERA detects and penalizes over-trust patterns using unidirectional causal attention prefixes, which do not exist under full bidirectional attention and iterative demasking. UVC directly edits head representations in feature space without relying on causal prefixes.
• vs Training-Time Alignment (e.g., RLHF-V): Supervised or preference fine-tuning requires substantial computational resources and curated data, risking out-of-domain degradation. UVC is completely training-free, utilizing 500 offline COCO samples to generalize zero-shot across GQA and A-OKVQA.

Rating

• Novelty: ⭐⭐⭐⭐⭐ First principled framework addressing object hallucination natively within multimodal discrete diffusion language models via mask-aware activation calibration.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Exhaustive evaluation across 9 POPE configurations, 4 MME tasks, CHAIR captioning, and unmasking divergence diagnostics.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear problem formulation, well-structured methodology, and compelling visual/empirical evidence.
• Value: ⭐⭐⭐⭐⭐ Provides an efficient, training-free blueprint for advancing reliable generation in emerging non-autoregressive multimodal architectures.