yard y architecture register decoding for efficient hallucination mitigation in
content_hash: 1f144ba1e55c375f ---# YARD: Y-Architecture Register Decoding for Efficient Hallucination Mitigation in Large Vision-Language Models
Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: Hallucination Mitigation, Contrastive Decoding, Y-Architecture, Register Tokens, Training-Free Inference
TL;DR¶
To tackle the sub-optimal degraded branch and doubled inference latency in contrastive decoding for large vision-language models, this paper proposes YARD—a training-free Y-Architecture Register Decoding framework that splits decoding at the middle layer and substitutes local visual patches with training-free register tokens capturing coarse global image semantics, significantly reducing latency while slashing hallucinations.
Background & Motivation¶
Large vision-language models (LVLMs) connect visual encoders with large language models, exhibiting remarkable multimodal understanding and text generation across tasks like image captioning and visual question answering. Nevertheless, these models frequently generate syntactically fluent responses that contradict actual visual evidence—a failure mode termed hallucination, including creating non-existent objects, confusing visual attributes, or hallucinating unsupported spatial relationships. Among training-free solutions, contrastive decoding (CD) has emerged as an effective logit-level intervention that isolates hallucination components by contrasting predictions from a standard multimodal input branch against a deliberately degraded visual branch, subtracting biased priors favored by weak visual grounding.
However, existing degraded branch designs suffer from severe practical dilemmas. On the one hand, text-only degradation completely removes or masks visual tokens inside the LLM; while simple, this severs the multimodal image context entirely, causing the degraded branch to hallucinate based on generic, image-agnostic language priors that poorly align with the specific input image. On the other hand, pixel-level degradation perturbs input images via Gaussian noise, masking, or local distortion. Because visual attributes, boundaries, textures, and object semantics are tightly entangled in pixel space, pixel perturbations cannot selectively disrupt hallucination-inducing local evidence, frequently contaminating the degraded branch with authentic visual evidence while doubling inference latency due to two full forward passes.
Cross-modal information flow analysis reveals that in deeper decoder layers, visual evidence is predominantly internalized into text representations, rendering late visual token modifications ineffective; conversely, early-layer cross-modal attention is diffuse. Local visual grounding predominantly crystallizes in the middle layers (around one-third of decoder depth). Core idea: share shallow decoder computations and branch internally at the middle layers, replacing fine-grained image patch tokens in the degraded branch with training-free register tokens that capture coarse global semantics without local grounding, forming a high-purity contrastive distribution that suppresses hallucinations at negligible computational cost.
Method¶
Overall Architecture¶
The end-to-end inference pipeline of YARD consists of visual encoding with register construction, shared prefix forward propagation, middle-layer Y-architecture branching, and register-conditioned contrastive decoding. An input image \(I\) is processed by the visual encoder into patch tokens while a training-free sink-shift mechanism redirects anomalous attention sink activations into appended register slots, yielding coarse global register tokens. The combined token sequence enters the LLM decoder, sharing full causal self-attention across shallow layers (layers \(1\) to \(K\)). At the middle branching layer \(K\), the computation forks: the clean branch retains all visual patches and register tokens, while the degraded branch discards local patch tokens and proceeds solely with register and text tokens. Both branches independently process deeper decoder layers to generate vocabulary logits, which are contrasted via thresholded subtraction to produce the next token.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Image I and Query X"] --> B["Training-Free Register Sink-Shift<br/>Extract Patches Z and Global Registers R"]
B --> C["Shared Shallow Prefix Decoding<br/>Compute H(K) across layers 1 to K"]
C --> D["Middle-Layer Y-Branching<br/>Clean keeps Z+R+T, Degraded keeps R+T"]
D --> E["Register-Conditioned Contrastive Decoding<br/>Logit subtraction suppresses ungrounded hallucination"]
E --> F["Hallucination-Mitigated Output Response"]
Key Designs¶
1. Training-Free Register Sink-Shift: Unsupervised Global Visual Semantics Extraction
Standard degraded branches either fall back to image-agnostic text priors or employ all-token average pooling, where the former loses image context and the latter linearly mixes local features. Drawing on the finding that Vision Transformers naturally produce high-norm outlier activations (attention sinks) in spatially low-informative regions that aggregate coarse image-level information, YARD constructs register tokens on the fly without retraining. During an offline one-time discovery phase, the top-10 global neuron indices in mlp.fc1 exhibiting peak anomalous activations are identified using only 18 calibration images through the vision encoder and stored in a lightweight cache. At test time, \(M\) empty register slots (default \(M=1\)) are appended to the visual sequence. Forward hooks redirect the maximum anomalous activation from patch tokens into the register slots while preserving regular patch activations:
$\(a_{r,k} \leftarrow \max_{1 \le j \le N} a_{j,k}, \quad a_{j,k} \leftarrow 0, \quad \forall j \in \{1,\dots,N\}, \; k \in \mathcal{K}_{\text{reg}}\)$
Both patch tokens \(Z\) and register tokens \(R\) are projected into the LLM embedding space. Because registers absorb global image-level semantics without fixed spatial patch coordinates, they provide an image-aware yet locally under-grounded visual reference.
2. Shallow Prefix Sharing & Middle-Layer Y-Branching: Aligning Grounding Dynamics with Low Latency
Addressing the severe latency penalty of two independent forward passes in input-level contrastive decoding, YARD introduces an internal Y-architecture decoder split. Grounding analysis shows that fine-grained text-to-vision alignment does not mature in shallow layers but emerges distinctly at the middle layers (around one-third decoder depth). Branching too early lacks cross-modal interaction, whereas branching too late leaves little room to suppress visual grounding because text representations have already absorbed visual cues. For an LLM decoder of depth \(D\), both branches share computation through layer \(K\) (e.g., layer 10 for LLaVA-1.5-7B): $\(H^{(K)} = \mathcal{F}_{1:K}([Z; R; T])\)$ At layer \(K\), computation forks: the clean branch maintains \(H^{(K)}_c = [Z^{(K)}; R^{(K)}; T^{(K)}]\), whereas the degraded branch strips all \(N\) patch tokens to retain only \(H^{(K)}_d = [R^{(K)}; T^{(K)}]\) for subsequent layers \(K+1\) through \(D\). Because \(|R| + |T| \ll |Z| + |R| + |T|\), the degraded branch eliminates duplicate shallow prefix execution and operates over a substantially shorter sequence in deeper layers, slashing computational overhead.
3. Register-Conditioned Contrastive Decoding: Targeted Logit Subtraction under Dual Degradation
Middle-layer branching combined with local token removal creates a dual degradation effect: the degraded branch inherits multimodal context before local grounding fully consolidates, while being deprived of fine-grained patch tokens in deeper layers. Consequently, the degraded logits \(\ell_d^i\) strongly emphasize tokens that appear plausible under coarse image semantics but lack genuine local visual grounding. At decoding step \(i\), the contrastive logit is computed as: $\(\ell_{cd}^i = (1 + \alpha) \ell_c^i - \alpha \ell_d^i\)$ where \(\alpha\) sets the contrastive subtraction strength (default \(\alpha=1.0\)). To prevent unlikely tokens from being falsely elevated, YARD enforces a clean-branch plausibility constraint \(\mathcal{V}_i = \{y \in \mathcal{V} \mid p_c(y) \ge \tau \max_{y'} p_c(y')\}\), restricting probability renormalization and token selection strictly to candidate tokens supported by the clean branch.
Loss & Training¶
YARD is a completely training-free inference-time decoding framework requiring zero parameter updates, fine-tuning, or external reward models. Hyperparameter settings adhere to uniform heuristics across backbones: the branching layer \(K\) is set near one-third of the decoder depth (\(K=10\) for 32-layer LLaVA-1.5-7B, \(K=13\) for 40-layer 13B models, and \(K=12\) for 36-layer Qwen3-VL-8B); the register token count is fixed at \(M=1\); and the contrastive coefficient is set to \(\alpha=1.0\). All base vision encoders and autoregressive LLMs remain completely frozen with original pre-trained weights.
Key Experimental Results¶
Main Results¶
Across both generative benchmarks (AMBER, Object HalBench, MME-Hallucination) and discriminative evaluations (POPE, AMBER-Discrimination), YARD establishes state-of-the-art hallucination mitigation against leading training-free baselines.
| Benchmark / Dataset | Metric | LLaVA-1.5-7B Baseline | Strongest Baseline (Method) | YARD (Ours) | Relative Improvement |
|---|---|---|---|---|---|
| Object HalBench | CHs ↓ / CHi ↓ | 27.0 / 10.5 | 20.3 / 7.3 (FuzzyCD) | 19.7 / 6.5 | Slashes baseline by 7.3 / 4.0 |
| AMBER (Generative) | CH ↓ / Hal ↓ | 8.3 / 32.0 | 4.6 (OPERA) / 18.8 (EVAS) | 4.6 / 21.6 | Matches best CH, achieves highest Cov. 48.9↑ |
| MME-Hallucination | Total ↑ | 553.00 | 614.99 (TAME) | 613.33 | Achieves highest Color score (163.33), strong overall |
| POPE-MSCOCO | ALL Acc. ↑ / F1 ↑ | 83.93 / 82.24 | 84.86 / 83.55 (AVISC) | 86.14 / 85.85 | Acc. gains +2.21, F1 gains +3.61 |
| AMBER (Discrim.) | Accuracy ↑ / F1 ↑ | 67.00 / 71.10 | 77.93 / 83.81 (EVAS) | 78.70 / 84.10 | Acc. jumps +11.70, F1 jumps +13.00 |
| InstructBLIP POPE | ALL Acc. ↑ / F1 ↑ | 80.37 / 80.50 | 83.49 / 82.68 (VCD) | 89.06 / 88.45 | Acc. jumps +8.69, F1 jumps +7.95 |
Ablation Study¶
On LLaVA-1.5-7B, degraded branch constructions and split layers \(K\) were rigorously ablated under both standard contrastive decoding and an \(\alpha\)-free direct subtraction stress test (\(\ell_{w/o\,\alpha} = \ell_c - \ell_d\)):
| Degraded Branch Variant | \(w/\ \alpha\) CHs ↓ / CHi ↓ | \(w/o\ \alpha\) Stress Test CHs ↓ / CHi ↓ | POPE Acc. ↑ / F1 ↑ | Observation / Note |
|---|---|---|---|---|
| Pixel-level (Input Noise) | 23.0 / 8.3 | 28.7 (+5.7) / 12.1 (+3.8) | - / - | Severe hallucination surge without \(\alpha\); retains too much true visual signal |
| Text-only (Pure Text Prior) | 21.7 / 7.0 | 24.0 (+2.3) / 7.9 (+0.9) | 85.1 / 84.2 | Drifts toward image-agnostic prior; fails on specific image contexts |
| Local (attn.) (Top-Attended Patches) | 22.0 / 10.8 | 24.0 (+2.0) / 13.0 (+2.2) | - / - | Preserving local patch evidence heavily corrupts correct visual predictions |
| Average (mean) (All-Patch Pooling) | 23.7 / 7.7 | 25.4 (+1.7) / 8.3 (+0.6) | 84.9 / 84.3 | Linear mixture of patches provides diffuse, imprecise contrastive signal |
| YARD (K=1, Early Branching) | 22.9 / 7.3 | - / - | 84.5 / 83.7 | Insufficient early multimodal interaction |
| YARD (K=25, Late Branching) | 25.0 / 7.9 | - / - | 83.2 / 81.6 | Local visual cues already internalized into text states |
| YARD (K=10, Register Default) | 19.7 / 6.5 | 21.0 (+1.3) / 7.0 (+0.5) | 86.1 / 85.9 | Only +0.5 CHi drift without \(\alpha\); highest purity and best discriminative metrics |
Key Findings¶
- Unprotected Stress Test Confirms Purity: Under direct logit subtraction without the protective clean anchor \(\alpha\), pixel-level degradation causes CHs to spike by \(+5.7\) due to visual evidence contamination, whereas YARD exhibits minimal degradation (\(+1.3\) on CHs, \(+0.5\) on CHi). This demonstrates that register-conditioned degradation isolates hallucination-prone tokens with surgical precision.
- Middle Layers Represent Grounding Transition: Branching layer sensitivity confirms that \(K=10\) (approximately \(1/3\) depth) consistently yields optimal performance, outperforming shallow (\(K=1\)) and late (\(K=25\)) splits. The one-third depth heuristic holds robustly on 13B models (\(K=13\)) and Qwen3-VL-8B (\(K=12\)).
- Near-Vanilla Inference Latency: For generating 128 tokens on an RTX PRO 6000 GPU, vanilla decoding takes 2.6 seconds, whereas YARD requires only 3.2 seconds (adding merely 0.6 seconds of overhead). YARD is 1.34× to 7.59× faster than competing methods like VCD (5.1s), ICD (4.3s), and OPERA (24.3s). The one-time register discovery takes only 0.925s on average and is computed once per backbone.
- Preservation of General Multimodal Perception: General multimodal evaluation confirms zero degradation of foundational capabilities: GQA marginally improves from 61.9 to 62.1, and MME total score rises from 1594 to 1741, demonstrating that hallucination suppression does not handicap general perception.
Highlights & Insights¶
- Repurposing Attention Sink Artifacts as Degraded Reference: By redirecting ViT high-norm attention sink activations into appended register tokens without training, YARD brilliantly repurposes an architectural artifact into an ideal global, non-local contrastive condition.
- Y-Architecture Internal Splitting Solves Efficiency Bottleneck: By identifying that visual grounding crystallizes in middle decoder layers, YARD moves contrastive branch construction inside the decoder, sharing shallow layers and truncating sequence length to eliminate the costly second forward pass.
- Broad Model-Agnostic Transferability: YARD operates as a universal plug-and-play decoding strategy, consistently boosting performance across heterogeneous LVLM families including Qwen3-VL, Qwen2-VL, LLaVA-NeXT, InstructBLIP, and Mini-Gemini without architectural tuning.
Limitations & Future Work¶
- Reliance on ViT High-Norm Outliers: The training-free register construction relies on attention sink phenomena characteristic of Vision Transformers; adapting YARD to non-Transformer vision backbones (e.g., pure ConvNets) will require alternative global feature formulation.
- Fixed Branching Layer Heuristic: The current implementation fixes the branching layer at roughly one-third decoder depth across all inputs, without dynamic adaptation based on input prompt complexity or image visual density.
- Future Directions: Potential avenues include extending register-conditioned Y-decoding to video-language models and exploring dynamic token-level branching thresholds conditioned on generation uncertainty.
Related Work & Insights¶
- vs VCD / ICD (Input-Level Contrastive Decoding): VCD injects Gaussian noise into images while ICD perturbs textual instructions. Both approaches either corrupt authentic visual semantics or suffer from double forward-pass latency. YARD constructs feature-level degradation inside the LLM decoder, achieving shared prefix computation and superior signal purity.
- vs DoLa / LayerCD (Layer-Wise Contrastive Decoding): DoLa and LayerCD contrast logits between different transformer layers. While avoiding input corruption, they do not explicitly model the cross-modal grounding failure process. YARD intervenes precisely at the critical window of visual evidence extraction.
- vs OPERA (Decoding Regularization): OPERA detects attention over-trust and applies beam search retrospection penalty, incurring massive runtime latency (24.3s). YARD achieves superior hallucination mitigation while maintaining near-vanilla generation speed (3.2s).
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Ingenious training-free redirection of ViT attention sinks to construct an image-aware, locally under-grounded degraded condition]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluated across 6 LVLM families, comprehensive generative and discriminative benchmarks, and rigorous \(\alpha\)-free stress testing]
- Writing Quality: ⭐⭐⭐⭐⭐ [Exceptionally clear motivation, rigorous empirical grounding flow analysis, and lucid methodology]
- Value: ⭐⭐⭐⭐⭐ [Training-free, highly accurate, and near-zero latency overhead; exceptional deployment value for production LVLMs]