Skip to content

SHICD: Self-Hallucination-Inducing Contrastive Decoding for Efficient LLM Hallucination Mitigation

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: Hallucination Mitigation, Contrastive Decoding, Inducing Layer Selection, FFN Distributional Shift, Adaptive Gating

TL;DR

SHICD proposes a training-free, single-pass contrastive decoding framework that mitigates hallucinations in LLMs and vision-language models without requiring auxiliary amateur models or external retrievers. By dynamically identifying the semantic emergence layer, extracting FFN-induced distributional shifts with semantic-preserving reweighting, and adaptively triggering contrastive decoding via Manhattan distance gating, it achieves state-of-the-art factual accuracy with negligible inference overhead (only 1.15ร— latency and 1.10ร— GPU memory).

Background & Motivation

Large Language Models and Vision-Language Models have demonstrated remarkable capabilities across text generation and multimodal reasoning, yet their pervasive tendency to generate factually incorrect outputs with high confidence severely undermines their trustworthiness in high-stakes applications such as medical diagnosis, legal counseling, and autonomous vehicle control. Existing training-free mitigation techniques are predominantly split into retrieval-augmented generation (RAG) and logit-based contrastive decoding (CD). While retrieval approaches enhance factual grounding, they inevitably incur significant query latencies and heavily depend on external knowledge base quality. Conversely, classical contrastive decoding frameworks (such as VCD and ICD) perturb the input with noise or misleading instructions to produce an "amateur" response, effectively doubling the forward inference passes and memory footprints.

To circumvent multi-model and multi-pass bottlenecks, recent efforts have focused on mining internal layer representations to directly reshape next-token distributions. Pioneering methods like DoLa contrast the final output with intermediate layer distributions, while SLED and MoLE track logit evolution across all network layers. However, scanning or tracking the entire layer stack introduces substantial computational overhead. More critically, applying static, indiscriminate contrastive penalties across every decoding step disrupts normal reasoning continuity and damages logical dependencies in multi-step Chain-of-Thought (CoT) tasks, precipitating the well-known "hallucination snowballing" phenomenon where early false corrections compound into catastrophic errors.

Inspired by cognitive developmental studies indicating that early interventions are pivotal when foundational concepts first emerge, the authors observe that LLMs transition abruptly from syntactic processing to semantic competence at specific intermediate layers. Beyond this boundary, excessive factual parameter injection by Feed-Forward Networks (FFNs) frequently induces "overthinking" and factual drift. Core idea: dynamically identify the semantic emergence layer during a single forward pass, synthesize a self-induced hallucination direction from FFN distributional shifts calibrated by semantic-preserving reweighting, and activate contrastive decoding via an adaptive distance gate only when distributional divergence indicates factual uncertainty.

Method

Overall Architecture

SHICD operates entirely during the single forward inference pass of an unmodified language model, without updating parameters or querying auxiliary networks. The pipeline proceeds through three tightly synchronized stages at each autoregressive step: (1) Dynamic Inducing Layer Selection, which probes early-exit cumulative probabilities via Logit Lens to identify the critical layer where preliminary semantic confidence emerges; (2) Self-Hallucination Inducing, which extracts FFN parameter-injected probability shifts, filters out beneficial factual evolution via Semantic-Preserving Reweighting (SPR), and constructs a targeted hallucination logit vector; and (3) Adaptive Contrastive Decoding (ACD), which computes the Manhattan distance between the hallucination-induced distribution and the final output distribution to trigger contrastive repulsion solely when necessary.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Sequence & Prefix Context<br/>Single-Pass Autoregressive Forward"] --> B["Dynamic Inducing Layer Selection<br/>Early-exit Logit Lens cumulative probability threshold"]
    B --> C["Self-Hallucination Inducing & SPR<br/>FFN shift extraction & directional consistency reweighting"]
    C --> D["Adaptive Contrastive Decoding Gating<br/>Manhattan divergence thresholding for dynamic activation"]
    D --> E["Calibrated Next-Token Prediction<br/>Truthful, hallucination-suppressed generation"]

Key Designs

1. Dynamic Inducing Layer Selection: Pinpointing Early Semantic Emergence

Prior internal-contrastive methods either rely on static, manually selected layers or exhaustively evaluate all layers, introducing noise from uninformative syntactic representations in shallow blocks. SHICD recognizes that lower Transformer layers primarily process lexical and syntactic patterns, whereas higher layers consolidate complex factual knowledge. To intervene precisely when initial semantics form without interfering with downstream factual consolidation, SHICD applies the final LM projection head \(\phi\) (Logit Lens) directly to the intermediate hidden states. For candidate generation step \(t\), it calculates the cumulative probability over the final layer's top-\(k\) vocabulary set \(\mathcal{V}_{N-1}^{\text{top-}k}\): $\(C_l(x_t \mid x_{<t}) = \sum_{v \in \mathcal{V}_{N-1}^{\text{top-}k}} P(x_t = v \mid x_{<t})\)$ The inducing layer \(l_{\text{ind}}\) is dynamically designated as the earliest layer where \(C_l\) surpasses a predefined confidence threshold \(\tau\): \(l_{\text{ind}} = \min\{l \mid C_l > \tau\}\), defaulting to layer \(N-2\) if the threshold is not reached. This token-specific selection isolates the exact transition point where preliminary semantics have solidified, preventing inappropriate contrastive penalties against early grammatical formations.

2. Self-Hallucination Inducing & Semantic-Preserving Reweighting: Mining FFN Shifts Without Degradation

External perturbation methods require costly duplicate model evaluations. Drawing from mechanistic interpretability insights that excessive parametric factual injection within Feed-Forward Networks (FFNs) is the primary driver of model "overthinking" and hallucinations, SHICD quantifies knowledge injection via the Manhattan distance between the probability distributions before and after the FFN at each layer \(l \ge l_{\text{ind}}\): $\(d_l = \sum_{x_t \in \mathcal{V}_{N-1}^{\text{top-}k}} |p_l^{\text{final}}(x_t \mid x_{<t}) - p_l^{\text{mid}}(x_t \mid x_{<t})|\)$ These distances are normalized into layer-wise inducing weights \(\alpha_l = d_l / \sum_{j \ge l_{\text{ind}}} d_j\). However, directly amplifying large FFN shifts risks destroying valid semantic progress. To safeguard truthful features, SHICD introduces Semantic-Preserving Reweighting (SPR). It evaluates the alignment between the local layer evolution direction \(P_l^{\text{local}} = \text{softmax}(\text{logits}_l^{\text{final}} - \text{logits}_{l-1}^{\text{final}})\) and the global trajectory toward the final representation \(\text{logits}_{N-1}^{\text{final}} - \text{logits}_l^{\text{final}}\) using cosine similarity \(c_l = \text{CosSim}(\cdot)\). A high similarity indicates that layer \(l\) contributes positively to the ground-truth prediction, and its hallucination weight is discounted accordingly: \(w_l = \alpha_l \cdot (1 - \phi(c_l))\). The calibrated shift vector \(\Delta D = \sum_{l=l_{\text{ind}}}^{N-2} w_l (\text{logits}_{l+1}^{\text{final}} - \text{logits}_l^{\text{final}})\) is added to the inducing layer logits to yield the self-induced hallucination distribution \(\text{logits}_{\text{hal}} = \text{logits}_{\text{ind}}^{\text{final}} + \Delta D\).

3. Adaptive Contrastive Decoding: Gated Intervention Preserving Reasoning Coherence

Static contrastive decoding indiscriminately penalizes all tokens, leading to catastrophic failure on multi-step reasoning benchmarks where rigid constraints distort the natural logical flow and induce early error snowballing. SHICD incorporates an Adaptive Contrastive Decoding (ACD) gate that activates only when genuine factual divergence is detected. At step \(t\), the Manhattan distance \(\delta_t\) between the hallucination distribution \(P_{\text{hal}}\) and the final distribution \(P_{\text{final}}\) is evaluated over the top-\(k\) vocabulary: $\(\delta_t = \sum_{x_t \in \mathcal{V}_{N-1}^{\text{top-}k}} |P_{\text{hal}}(x_t \mid x_{<t}) - P_{\text{final}}(x_t \mid x_{<t})|\)$ A binary gate \(\gamma_t = \mathbb{I}(\delta_t > \eta)\) activates contrastive penalization when divergence exceeds tolerance threshold \(\eta\): $\(\text{logits}_{\text{acd}} = \text{logits}_{N-1}^{\text{final}} - \gamma_t \cdot \lambda \cdot (\text{logits}_{\text{hal}} - \text{logits}_{N-1}^{\text{final}})\)$ When divergence is low (\(\delta_t \le \eta\), \(\gamma_t = 0\)), the model seamlessly defaults to standard greedy decoding. This adaptive intervention leaves predictable functional and logical tokens completely undisturbed, while fiercely repressing factual hallucination spikes.

Benchmark & Evaluation Protocol

The empirical evaluation spans diverse model families and tasks. Factual benchmarks include multiple-choice tasks (TruthfulQA MC1, MC2, MC3; FACTOR-Wiki long-paragraph verification), open-ended generation (TruthfulQA evaluated with GPT-4o-mini for %Truth, %Info, and %Truthร—Info; TriviaQA, PopQA, NQ-Open), and multi-step reasoning tasks (StrategyQA, GSM8K). Tested base models include LLaMA-2 (7B/13B, Base and Chat), LLaMA-3-8B-IT, Gemma-7B, DeepSeek-V2-16B, and Qwen3-8B. Baselines comprise standard greedy decoding, layer-contrasting DoLa, and full-layer evolution decoding SLED. All experiments are executed with 16-bit precision on NVIDIA RTX 4090 GPUs.

Key Experimental Results

Main Results

Across all eight evaluated LLM backbones, SHICD demonstrates consistent and substantial factual gains while maintaining top-tier reasoning capabilities:

Backbone Model Method TruthfulQA (MC Avg. โ†‘) FACTOR (Wiki โ†‘) TruthfulQA (Open %T*I โ†‘) StrategyQA (CoT โ†‘) GSM8K (Acc โ†‘)
LLaMA-2-7B Greedy 41.46 58.38 21.38 60.96 13.98
+DoLa 42.05 62.66 28.95 60.61 14.52
+SLED 42.91 67.20 21.76 62.49 14.78
+SHICD (Ours) 43.69 66.89 32.29 61.91 14.99
LLaMA-2-7B-chat Greedy 41.73 56.71 41.93 63.67 21.25
+DoLa 41.90 56.51 52.81 64.32 21.00
+SLED 44.57 64.81 42.62 64.67 21.42
+SHICD (Ours) 45.43 65.00 56.28 66.02 21.91
LLaMA-2-13B-chat Greedy 44.10 61.96 46.03 69.87 36.66
+DoLa 42.42 58.13 55.39 67.90 35.03
+SLED 44.58 67.50 51.30 69.96 37.01
+SHICD (Ours) 45.16 66.93 63.71 71.01 37.65
LLaMA-3-8B-IT Greedy 47.82 59.49 28.76 67.54 38.73
+DoLa 44.89 61.06 38.50 66.38 37.21
+SLED 49.34 67.17 28.12 67.51 39.15
+SHICD (Ours) 49.75 70.29 42.84 69.92 40.26
Gemma-7B Greedy 33.99 60.42 28.08 74.80 35.53
+DoLa 31.49 36.07 41.81 73.54 36.10
+SLED 35.80 65.56 29.41 74.93 38.83
+SHICD (Ours) 36.91 68.21 46.61 73.54 39.00
DeepSeek-V2-16B Greedy 42.74 72.11 42.35 74.16 57.31
+DoLa 41.54 65.42 44.69 71.00 56.71
+SLED 44.08 76.61 39.80 75.10 57.79
+SHICD (Ours) 44.85 78.81 52.96 75.10 58.03
Qwen3-8B Greedy 40.98 62.11 34.17 67.08 37.74
+DoLa 41.55 63.66 49.57 67.84 31.00
+SLED 42.88 63.83 36.32 70.05 38.20
+SHICD (Ours) 43.58 64.40 52.30 74.55 38.49

In terms of decoding efficiency and GPU memory consumption (measured on LLaMA-2-7B-chat on a single RTX 4090):

Method 20 Tokens (ms) โ†“ 50 Tokens (ms) โ†“ 100 Tokens (ms) โ†“ Per-sample Total (ms) โ†“ Peak GPU Memory (MB) โ†“
Greedy 477.9 (ร—1.00) 1214.5 (ร—1.00) 2443.0 (ร—1.00) 4032.7 (ร—1.00) 13471 (ร—1.00)
DoLa 608.1 (ร—1.27) 1574.2 (ร—1.30) 3115.8 (ร—1.28) 5104.8 (ร—1.27) 15560 (ร—1.15)
SLED 670.0 (ร—1.40) 1785.4 (ร—1.47) 3551.4 (ร—1.45) 6038.9 (ร—1.50) 15942 (ร—1.18)
SHICD (Ours) 554.7 (ร—1.16) 1421.6 (ร—1.17) 2799.0 (ร—1.15) 4624.8 (ร—1.15) 14818 (ร—1.10)

Ablation Study

Ablation experiments on LLaMA-2-7B-chat on the TruthfulQA (MC) benchmark isolate the contributions of Semantic-Preserving Reweighting (SPR) and Adaptive Contrastive Decoding (ACD):

Configuration MC1 (Single-choice โ†‘) MC2 (Multi-choice โ†‘) MC3 (Combined โ†‘) Avg. Score (โ†‘) Impact Analysis
Full SHICD 39.01 64.35 32.93 45.43 Peak overall factual accuracy
w/o ACD (Static token intervention) 38.15 63.96 32.07 44.73 Drops 0.70 points due to disruption of confident tokens
w/o SPR (Raw FFN shifts only) 37.03 61.98 31.89 43.63 Drops 1.80 points, indicating suppression of valid factual evolution
Greedy Baseline 35.62 57.46 32.10 41.73 3.70 points below full model
SLED Baseline 36.96 63.84 32.91 44.57 0.86 points lower with 1.3ร— latency penalty

Key Findings

  • SPR Prevents False Penalization of Factual Evolution: Disabling Semantic-Preserving Reweighting causes the largest performance drop (-1.80 points on TruthfulQA Avg.). Large FFN distributional shifts are not exclusively erroneous; they often represent genuine knowledge retrieval. Calibrating shifts via directional cosine consistency ensures that beneficial factual signals are preserved.
  • ACD Protects Step-by-Step Reasoning Coherence: Baselines like DoLa suffer significant regressions on mathematical reasoning benchmarks (dropping from Greedy's 21.25% to 21.00% on GSM8K with LLaMA-2-7B-chat, and down to 18.88% on LLaMA-2-13B), because static contrastive constraints disrupt intermediate derivation tokens. In contrast, SHICD's ACD gate only activates when distribution divergence exceeds \(\eta\), maintaining flawless continuity in reasoning steps.
  • Minimal Per-Token Latency and Memory Footprint: Across 20-, 50-, and 100-token generation profiles, SHICD introduces only a 1.15ร— latency overhead and a 1.10ร— peak GPU memory footprint over standard greedy decoding, dramatically outperforming SLED (1.50ร— latency, 1.18ร— memory) and DoLa (1.27ร— latency, 1.15ร— memory).
  • Hyperparameter Robustness: Parameter sweeps on FACTOR (Wiki) establish that performance gains peak at +7.75% when the activation threshold is \(\eta \approx 0.4\) and the confidence threshold is \(\tau \in [0.4, 0.6]\). Performance remains reliably high across candidate set sizes \(k \in [50, 75]\) and scaling strength \(\lambda \approx 1.25\).

Highlights & Insights

  • Developmental Cognitive Analogy for Inducing Layer Selection: Rather than brute-force layer searches, SHICD mirrors early cognitive development by targeting the initial emergence of semantic representation. Capturing the phase transition via early-exit cumulative probability ensures that negative signals are grounded before parametric overthinking sets in.
  • Converting FFN Overthinking into Useful Negative Feedback: The method ingeniously repurposes the internal FFN knowledge-injection mechanism. Instead of relying on external negative prompts or perturbed models, the FFN's own propensity to overthink is harvested as an internal negative sample.
  • Surgical, On-Demand Contrastive Intervention: The adaptive distance gate restricts contrastive penalties to tokens with high distributional uncertainty, proving that contrastive decoding should be applied selectively rather than uniformly across the generation sequence.

Limitations & Future Work

  • Author-Admitted Limitations: Evaluations were conducted on open-weight models up to 16B parameters (LLaMA-2, LLaMA-3, Gemma, DeepSeek-V2, Qwen3). Validation on 70B+ frontier models remains to be explored. Furthermore, because SHICD requires access to intermediate hidden states, it cannot be deployed on black-box, API-only proprietary models.
  • Observed Constraints: In multimodal vision-language architectures, dense visual tokens may alter the layer-wise semantic emergence trajectory, requiring calibrated \(\tau\) thresholds depending on visual resolution and context length.
  • Future Directions: Extending the dynamic inducing layer concept to interleaved multimodal reasoning, agentic tool-use trajectories, and long-context document synthesis where parametric hallucination risks are particularly acute.
  • vs Input-Perturbed Contrastive Decoding (VCD / ICD): VCD injects Gaussian image noise and ICD constructs negative textual prompts, each requiring two or more complete model executions. SHICD derives negative signals purely from internal hidden states within a single forward pass, slashing latency overhead from 2.0ร—+ to just 1.15ร—.
  • vs Layer-Contrasting Decoding (DoLa): DoLa contrasts the final layer with fixed or dynamically searched lower layers for every single token without semantic weighting. SHICD introduces dynamic emergence layer selection, directional reweighting, and adaptive gating, preventing the reasoning degradations seen in DoLa.
  • vs Full-Layer Logit Evolution Decoding (SLED): SLED calculates evolution vectors across all layers, incurring substantial computational and memory overhead. SHICD focuses strictly on layers above the semantic inducing point and applies top-\(k\) filtering, beating SLED's factuality scores while reducing peak memory overhead from 1.18ร— to 1.10ร—.

Rating

  • Novelty: โญโญโญโญโญ Self-inducing hallucination via internal FFN shifts combined with directional consistency reweighting is a highly original and effective paradigm.
  • Experimental Thoroughness: โญโญโญโญโญ Comprehensive evaluations across eight diverse LLM backbones spanning multiple-choice, open-ended generation, and complex CoT reasoning, backed by extensive ablations and efficiency profiles.
  • Writing Quality: โญโญโญโญโญ Crisp mathematical formulation, intuitive biological analogies, and transparent experimental reporting.
  • Value: โญโญโญโญโญ As a lightweight, training-free, single-pass decoding framework with negligible latency and memory footprints, it offers immense practical utility for production deployments.