Skip to content

DARE to Mitigate Hallucination: Dual-path Auto-Regressive-aware Editing

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/KU-VGI/DARE
Area: Multimodal VLM
Keywords: Large Vision-Language Models, Object Hallucination, Representation Editing, Autoregressive Dynamics, Null-space Projection

TL;DR

DARE demonstrates that conventional teacher-forcing (TF) representation editing fails to flip relative logit margins during actual autoregressive generation and compromises visual attention, resolving these issues via a training-free weight editing framework that unifies textual contrasts, decoding state transitions, and counterfactual visual differences without incurring any inference overhead.

Background & Motivation

Large Vision-Language Models (LVLMs) such as LLaVA, MiniGPT-4, and mPLUG-Owl2 have demonstrated remarkable capabilities in multimodal perception, conversational reasoning, and visual description. However, object hallucination—the generation of descriptions inconsistent with or unsupported by visual inputs—remains a persistent vulnerability that impedes reliable deployment in mission-critical applications. To mitigate hallucinations without the steep computational costs of end-to-end retraining or instruction fine-tuning, training-free representation editing approaches like Nullu have gained traction. Typically, these techniques contrast hidden representations of hallucinated versus truthful responses under teacher-forcing (TF) conditions, extract a hallucination subspace (HalluSpace) using singular value decomposition (SVD), and project model weights onto its null space to eliminate hallucination-inducing directions.

Despite their theoretical appeal, existing representation editing methods often deliver limited practical gains during actual generation due to two fundamental architectural discrepancies. First is the generation dynamics mismatch. Under teacher-forcing evaluation, correct tokens frequently remain ranked in the top-5 candidate positions with high probability (\(93.48\% \pm 1.47\%\)), revealing that the model's latent representation still retains grounded visual evidence. However, during real autoregressive (AR) greedy decoding, TF-based editing produces minimal shifts in the logit margin between truthful and hallucinated tokens (the difference clusters tightly around zero), meaning the hallucinated token continues to dominate and gets generated. Second is the entangled multimodal interference. Because prior editing vectors are computed solely from text-only contrasts, applying them across deep multimodal attention layers indiscriminately perturbs features supporting cross-modal fusion, inducing an average \(7.53\%\) drop in visual-token attention across layers and subsequently degrading grounded perception.

Overcoming these limitations requires grounding hallucination suppression directly within autoregressive state transitions while safeguarding visual-token communication channels. Core idea: DARE (Dual-path Auto-Regressive-aware Editing) constructs an Augmented HalluSpace integrating sequential AR decoding transitions with teacher-forced textual contrasts, paired with a counterfactual visual-difference subspace from object-manipulated image pairs, executing decoupled null-space projections onto FFN down-projection and self-attention output projection layers respectively.

Method

Overall Architecture

DARE operates entirely offline at the parameter level without introducing any runtime latency or auxiliary inference branches. The framework orchestrates three distinct representation extraction pipelines: 1. Teacher-Forced Textual Contrasts (TF HalluSpace): Captures global linguistic differences between hallucinated and factual descriptions under identical prompts; 2. Autoregressive Decoding Awareness (AR HalluSpace): Traces token-by-token generation trajectories on hallucinated outputs to capture sequential state transition increments \(\Delta h\) immediately preceding hallucination occurrences; 3. Counterfactual Visual-Difference Contrasts (Visual-Difference HalluSpace): Leverages object-erased paired images under teacher-forcing to isolate representation shifts driven purely by visual evidence.

In the parameter editing stage, DARE orthogonalizes the AR vectors against the TF subspace via residual projection, constructs an orthonormal Augmented HalluSpace \(V_\omega^{Aug}\) via QR decomposition, and performs null-space projection on the FFN down-projection weights (\(W_{\omega, down}\)). Simultaneously, the visual-difference subspace \(V_{\omega, k_{vis}}^{vis}\) is projected out from the self-attention output projection weights (\(W_{\omega, o}\)), creating a coordinated, dual-path editing scheme.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Data Pairs<br/>LURE text pairs + BEAF counterfactual image pairs"] --> B["AR-aware HalluSpace<br/>Captures state transitions in temporal window prior to hallucination"]
    A --> C["TF-based HalluSpace<br/>Extracts pooled hidden representation differences from text pairs"]
    A --> D["Visual-Difference HalluSpace<br/>Isolates representation shifts induced by object addition/removal"]
    B --> E["Augmented HalluSpace Construction<br/>AR vector residualization against TF + QR orthonormalization"]
    C --> E
    E --> F["FFN Down-Projection Weight Editing<br/>Null-space projection on W_down to eliminate generation/textual hallucination"]
    D --> G["Attention Output Projection Weight Editing<br/>Null-space projection on W_o to prevent visual attention collapse"]
    F --> H["Edited LVLM<br/>Standard autoregressive decoding with zero inference overhead"]
    G --> H

Key Designs

1. AR-aware HalluSpace Construction: Capturing Local Decoding Transitions Static TF representations only reveal global distributions conditioned on full ground-truth sequences, failing to reflect error propagation and token competition during autoregressive decoding. DARE leverages the LURE dataset and CHAIR evaluation protocol on greedy decoding sequences \(y = (y_1, \dots, y_T)\) to pinpoint token position \(t\) where an object hallucination first emerges. At each target layer \(\omega\), the sequential representation increments between consecutive decoding steps are computed: $\(\Delta h_{\omega, j} = h_{\omega, j} - h_{\omega, j-1}\)$ For every hallucinated token event, a temporal window of size \(w\) preceding the event is sampled: \(\{\Delta h_{\omega, t}, \Delta h_{\omega, t-1}, \dots, \Delta h_{\omega, t-w+1}\}\). Stacking these state transition vectors across samples yields the matrix \(G_\omega \in \mathbb{R}^{N_{AR} \times D}\). Performing Singular Value Decomposition (SVD): $\(G_\omega = U_\omega \Sigma_\omega (V_\omega^{AR})^\top\)$ The top-\(k_{AR}\) right singular vectors define the AR-aware subspace \(V_\omega^{AR} \in \mathbb{R}^{D \times k_{AR}}\), effectively capturing the transient dynamic drift driving the model toward hallucinated token selection.

2. Augmented HalluSpace Editing: Orthogonal Residualization and FFN Down-Projection Because static TF directions \(V_\omega^{TF} \in \mathbb{R}^{D \times k_{TF}}\) and AR directions \(V_\omega^{AR}\) share semantic components, directly concatenating them introduces redundancy and over-constrains parameter capacity. DARE projects each AR direction onto the orthogonal complement of the TF subspace: $\(\tilde{v}_j^{AR} = \left( I - V_\omega^{TF} (V_\omega^{TF})^\top \right) v_j^{AR}\)$ The concatenated matrix \([V_\omega^{TF}, \tilde{V}_\omega^{AR}]\) is then orthonormalized via QR decomposition: $\([V_\omega^{TF}, \tilde{V}_\omega^{AR}] = Q_\omega^{Aug} R_\omega^{Aug}\)$ The orthonormal basis \(V_\omega^{Aug} := Q_\omega^{Aug}\) spans the Augmented HalluSpace. Since feed-forward network (FFN) down-projection layers act as associative memory banks projecting representations back into the residual stream, DARE applies null-space projection operator \(P_\omega = I - V_\omega^{Aug} (V_\omega^{Aug})^\top\) to edit the down-projection weights: $\(W_{\omega, down}^{edit} = P_\omega W_{\omega, down}^{orig}\)$ This surgical intervention suppresses linguistic and dynamic hallucination pathways without disturbing foundational language modeling representations.

3. Visual-Difference Editing: Counteracting Multimodal Attention Degradation To address the collateral reduction in visual attention caused by pure textual editing, DARE incorporates counterfactual image pairs from the BEAF benchmark, where target objects are naturally edited out from identical MSCOCO scenes. Crucially, hidden representations are harvested under teacher-forcing with identical prompts, ensuring language contexts remain static so representation differences isolate purely visual discrepancies. The pooled hidden state difference matrix \(E_{vis}^\omega = X_H^\omega - X_T^\omega\) undergoes SVD: $\(E_{vis}^\omega = U_\omega \Sigma_\omega (V_{\omega, k_{vis}}^{vis})^\top\)$ The leading \(k_{vis}\) singular vectors form the visual-difference subspace \(V_{\omega, k_{vis}}^{vis}\). DARE assigns this subspace exclusively to edit the self-attention output projection weights: $\(W_{\omega, o}^{edit} = \left( I - V_{\omega, k_{vis}}^{vis} (V_{\omega, k_{vis}}^{vis})^\top \right) W_{\omega, o}^{orig}\)$ Because the attention output projection directly modulates how visual and cross-modal contextual representations merge into the residual stream, filtering out hallucinated visual artifacts at this junction shields grounded visual attention channels from being inadvertently silenced.

Loss & Training

DARE requires no iterative gradient-based optimization or auxiliary training loss. The execution pipeline involves three sequential steps: 1. Extracting pooled hidden states offline via standard forward passes over paired datasets (LURE and BEAF); 2. Computing compact truncated SVD and QR factorizations on small covariance matrices; 3. Updating the target layer weight matrices in-place via closed-form matrix multiplications. The resulting model preserves identical architecture and inference latency as the unedited base checkpoint.

Key Experimental Results

Main Results

DARE is evaluated against standard decoding strategies, inference-time decoding interventions (VCD, OPERA, ONLY, CMI-VLD), and representation editing (Nullu) on the standard captioning hallucination benchmark CHAIR (MSCOCO) and object existence benchmark POPE.

Table 1: CHAIR evaluation results across three LVLM backbones (max token length 64, original paper Table 1)

Category Method LLaVA-1.5 \(CHAIR_S \downarrow\) LLaVA-1.5 \(CHAIR_I \downarrow\) LLaVA-1.5 \(BLEU \uparrow\) MiniGPT-4 \(CHAIR_S \downarrow\) mPLUG-Owl2 \(CHAIR_S \downarrow\)
Decoding Baseline Greedy \(24.32 \pm 2.92\) \(8.91 \pm 0.46\) \(15.16 \pm 0.30\) \(35.91 \pm 3.15\) \(24.96 \pm 1.43\)
Decoding Baseline Beam Search \(22.61 \pm 1.64\) \(7.01 \pm 0.95\) \(16.17 \pm 0.13\) \(27.33 \pm 2.08\) \(21.80 \pm 1.31\)
Decoding Intervention VCD (CVPR24) \(21.15 \pm 1.25\) \(7.19 \pm 0.26\) \(14.91 \pm 0.46\) \(30.60 \pm 1.95\) \(20.56 \pm 2.40\)
Decoding Intervention OPERA (CVPR24) \(17.61 \pm 1.16\) \(6.09 \pm 1.11\) \(15.21 \pm 0.49\) \(29.64 \pm 0.18\) \(20.19 \pm 0.15\)
Latent Intervention SID (ICLR25) \(17.95 \pm 1.93\) \(6.64 \pm 0.11\) \(15.04 \pm 0.24\) \(27.16 \pm 2.06\) \(19.66 \pm 1.43\)
Weight Editing Nullu (CVPR25) \(20.53 \pm 1.40\) \(7.03 \pm 0.86\) \(15.37 \pm 0.18\) \(23.93 \pm 0.31\) \(18.80 \pm 0.72\)
Dynamic Decoding CMI-VLD (NeurIPS25) \(16.42 \pm 0.75\) \(6.59 \pm 0.81\) \(15.40 \pm 0.86\) \(23.80 \pm 1.73\) \(17.76 \pm 0.84\)
Ours DARE (Ours) \(14.93 \pm 1.68\) \(6.07 \pm 0.82\) \(15.61 \pm 0.59\) \(22.64 \pm 1.49\) \(16.81 \pm 0.43\)

On the POPE random sampling benchmark (LLaVA-1.5, Table 2 in original paper), DARE attains an Accuracy of \(89.74 \pm 0.54\%\), Precision of \(91.40 \pm 0.71\%\), and F1 Score of \(90.04 \pm 0.37\%\), significantly surpassing both Greedy decoding (\(79.53\% / 82.64\% / 82.47\%\)) and the state-of-the-art editing baseline Nullu (\(88.73\% / 88.70\% / 87.56\%\)).

Ablation Study

To dissect the individual contributions and mutual interactions of TF textual editing, AR-aware decoding transitions, and visual counterfactual contrasts, comprehensive component ablations were conducted on LLaVA-1.5.

Table 2: Ablation study of DARE components on LLaVA-1.5 (original paper Table 3)

Configuration \(Editing_{TF}\) \(Editing_{AR}\) \(VisualDiff\) \(CHAIR_S \downarrow\) \(CHAIR_I \downarrow\) Description
Base - - - \(24.32 \pm 2.92\) \(8.91 \pm 0.46\) Unmodified baseline model
(A) TF Only ✓ - - \(20.53 \pm 1.40\) \(7.03 \pm 0.86\) Conventional static TF editing
(B) AR Only - ✓ - \(17.61 \pm 1.64\) \(6.94 \pm 1.06\) Strongest individual component (-6.71 CS)
(C) Visual Only - - ✓ \(17.81 \pm 1.28\) \(7.20 \pm 0.73\) Demonstrates strong visual counterfactual efficacy
(D) TF + AR ✓ ✓ - \(15.83 \pm 1.48\) \(6.47 \pm 1.03\) Residualized combination provides marked gain
(E) AR + Visual - ✓ ✓ \(15.57 \pm 1.70\) \(6.41 \pm 0.83\) Best-performing pairwise combination
(F) TF + Visual ✓ - ✓ \(17.17 \pm 1.22\) \(6.62 \pm 0.50\) Visual grounding repairs pure text degradation
DARE (Full Model) ✓ ✓ ✓ \(14.93 \pm 1.68\) \(6.07 \pm 0.82\) Optimal synergy across all three pathways

Hyperparameter explorations (original paper Table 4) show that: - Target Layers: Editing higher layers (layers 16–31) yields the lowest hallucination (\(CHAIR_S = 15.11\)) compared to early layers (layers 0–15: \(17.70\)), confirming hallucinations solidify near the output language decoding stage. - Subspace Rank: A compact rank of \(k=2\) achieves optimal performance (\(CHAIR_S = 15.11\)), whereas increasing \(k\) to 8 degrades the score to \(17.09\) due to over-erasure of general linguistic capabilities. - Temporal Window: A window size of \(w=3\) preceding the hallucinated token yields the best trade-off (\(CHAIR_S = 15.11\)), isolating pertinent local state deviations.

Key Findings

  • AR dynamics constitute the primary driver of hallucination reduction: Isolated AR editing achieves a \(6.71\) drop in \(CHAIR_S\), nearly double the reduction achieved by standalone TF editing (\(3.79\)), confirming that modeling generation-time state transitions is far more effective than analyzing static prompt-completion representations.
  • Visual counterfactual editing prevents attention collapse: While Nullu causes an average \(7.53\%\) decrease in visual-token attention across layers 16–31, DARE restricts this drop to just \(2.32\%\), preserving the model's visual grounding capability.
  • Robust scalability across modern backbones: When deployed on recent models including Qwen3-VL-4B/8B and Gemma-3-4B/12B (original paper Table 5), DARE consistently outperforms base models and Nullu (e.g., cutting Gemma-3-12B \(CHAIR_S\) from \(33.40\) to \(30.91\)), demonstrating wide architectural generalizability.

Highlights & Insights

  • Diagnosing the logit margin paralysis: The paper goes beyond checking whether representations move apart in feature space; it directly probes the empirical CDF of logit margins under greedy decoding, revealing why TF editing reshapes hidden vectors without actually altering output tokens.
  • Orthogonal residualization prior to QR decomposition: Rather than naively merging non-orthogonal subspaces, projecting AR vectors onto the orthogonal complement of the TF space isolates novel, non-redundant generation cues while preserving parameter expressiveness.
  • Component-targeted functional routing: Assigning language and decoding dynamics (Augmented HalluSpace) to FFN down-projection layers while routing visual contrast vectors to self-attention output projections aligns mathematically with the distinct functional roles of MLP memory recall and attention feature routing.

Limitations & Future Work

  • Reliance on supervised hallucination localization datasets: Constructing the AR-aware subspace requires identifying exact hallucination onset tokens via CHAIR on LURE captions, making it dependent on predefined object vocabularies rather than free-form conceptual errors.
  • Static rank and temporal window across heterogeneous concepts: The framework fixes hyper-parameters (\(w=3, k=2\)) globally; however, complex compositional concepts or fine-grained visual reasoning might benefit from dynamic, token-adaptive rank allocations.
  • Potential extension to video and embodied trajectories: The formulation of autoregressive transition increments (\(\Delta h\)) is naturally suited for multi-frame video understanding and sequential action planning in embodied agents.
  • vs Nullu (CVPR 2025): Nullu pioneered teacher-forced null-space projection for LVLM hallucination mitigation. DARE builds directly upon its training-free efficiency but overcomes its two core flaws—inability to overturn AR logit decisions and loss of visual attention—by incorporating AR trajectory dynamics and visual counterfactuals.
  • vs Decoding Interventions (OPERA, VCD, CMI-VLD): Decoding-based methods introduce substantial runtime overhead by performing contrastive passes or attention retrospection during generation. DARE achieves superior hallucination suppression while maintaining native greedy decoding speeds (\(5\times\) faster than OPERA).

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Identifies logit margin insensitivity in TF editing and proposes a unified dual-path AR-aware weight editing framework]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluated across 7 backbones spanning LLaVA, MiniGPT-4, mPLUG-Owl2, Qwen3-VL, and Gemma-3 on CHAIR, POPE, and MME]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Insightful diagnostic experiments, clearly motivated design choices, and rigorous mathematical formulations]
  • Value: ⭐⭐⭐⭐⭐ [Provides an efficient, plug-and-play, training-free paradigm for enhancing reliability and factual grounding in production LVLMs]