Skip to content

ROT: Rotating Hidden States towards Contextual Vectors for Hallucination Mitigation in LVLMs

Conference: EMNLP 2026
Paper: ACL Anthology
Area: Multimodal VLM
Keywords: Large Vision-Language Models, Object Hallucination, Hidden State Rotation, Training-Free Intervention, Geometric Alignment

TL;DR

Addressing the empirical revelation that hallucinated tokens in Large Vision-Language Models (LVLMs) geometrically deviate from the multimodal contextual subspace rather than simply over-relying on linguistic priors, ROT introduces a training-free framework that dynamically detects intermediate semantic derailment, performs norm-preserving rotation back toward the grounded context plane, and stabilizes deep feature trajectories via representational smoothing.

Background & Motivation

Large Vision-Language Models (LVLMs) have demonstrated extraordinary proficiency in synthesizing complex visual perception with open-ended linguistic reasoning. Despite these breakthroughs, the persistent issue of object hallucination—wherein models generate descriptions of objects, attributes, or spatial relationships absent from the visual input—continues to pose severe reliability risks in safety-critical deployments. To circumvent the prohibitive computational expenses of post-hoc supervised fine-tuning or reinforcement learning from human feedback (RLHF), training-free inference-time interventions have emerged as an attractive frontier. Existing methodologies predominantly manipulate attention weights, discard visual tokens via epistemic uncertainty, or perform logits contrasting across distorted inputs (such as VCD, OPERA, and MemVR). Nevertheless, these approaches merely adjust attention weights or passively substitute key-value tokens, which fails to directly govern the deep semantic vectors propagating into the final predictive layers.

Conventional wisdom has long presumed that object hallucination is triggered by an overwhelming reliance on strong unimodal language priors over visual signals. Under this assumption, an internal representation undergoing hallucination would be expected to align tightly with the textual context while exhibiting weak correlation with the visual evidence. However, rigorous layer-wise empirical analysis on LLaVA-1.5-7B decodes a counter-intuitive phenomenon: when hallucinated tokens are generated, their hidden state vectors \(h_k\) after self-attention and residual addition exhibit a simultaneous and steep collapse in cosine similarity with respect to both the textual context vector \(C_T\) and the visual context vector \(C_V\). Rather than over-fitting to text, the hidden states suffer from a catastrophic contextual deviation, derailing completely from the grounded multimodal factual plane spanned by the legitimate visual and textual anchors.

This geometric diagnosis underscores that passive attention modification cannot rectify a representation that has already drifted astray in the high-dimensional latent space; what is required is an active geometric intervention that steers the derailed hidden states back toward the factual context plane without compromising token activation magnitude. Recognizing that the semantic deviation peaks predominantly within intermediate Transformer layers, intervention must be targeted both spatially and temporally. Core idea: dynamically detect contextual deviation in the sensitive middle layers, apply a norm-preserving spherical rotation to steer the derailed hidden state back toward the grounded multimodal context plane, and enforce representational smoothing across subsequent deep layers to ensure a coherent and stable trajectory into the language modeling head.

Method

Overall Architecture

The ROT (Rotating Hidden States towards Contextual Vectors) framework executes during standard autoregressive decoding. It specifically intervenes on the hidden state vector \(h_k\) immediately following self-attention and residual addition, before layer normalization and the feed-forward network (FFN). The overall architecture comprises three coordinated stages: first, normalized attention weights are partitioned to aggregate isolated textual and visual context anchor vectors; next, in the sensitive middle layers, a contextual deviation indicator triggers a norm-preserving rotation whenever the semantic divergence breaches a risk threshold; finally, subsequent deeper layers apply representational smoothing through temporal interpolation to preserve trajectory stability.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Token Forward Pass & Self-Attention"] --> B["Multimodal Context Projection Construction<br/>Extract Text Anchor CT and Visual Anchor CV"]
    B --> C["Dynamic Deviation Detection & Critical Layer Selection<br/>Compute Deviation Indicator E vs Trigger Threshold τ"]
    C -->|Deviation E ≥ τ| D["Norm-Preserving Rotation in Intermediate Layers<br/>Slerp Driven Hidden State Rotation toward Ctarget"]
    C -->|Normal E < τ| E["Preserve Original Hidden State"]
    D --> F["Representational Smoothing in Subsequent Layers<br/>Weighted Layer-wise Interpolation for Trajectory Stability"]
    E --> F
    F --> G["LayerNorm and FFN for Vocabulary Prediction"]

Key Designs

1. Multimodal Context Projection Construction: Partitioning Attention Weights into Modality-Grounded Anchors

To ascertain whether a hidden state vector has derailed, the model requires an explicit, layer-specific coordinate system representing verified factual context. Rather than treating past token representations indiscriminately, ROT extracts the raw self-attention weights assigned from the current query to all preceding tokens at layer \(k\). These tokens are split into two mutually exclusive sets: textual token indices \(\mathcal{T}\) and visual token indices \(\mathcal{V}\). The attention weights within each subset are independently normalized via Softmax, yielding \(\tilde{A}_{t,j}\) and \(\tilde{A}_{v,i}\). By computing the attention-weighted sum of the preceding layer's output states \(X^{k-1}\), ROT constructs pure textual and visual context anchors: $\(C_T = \sum_{j \in \mathcal{T}} \tilde{A}_{t,j} X_j^{k-1}, \quad C_V = \sum_{i \in \mathcal{V}} \tilde{A}_{v,i} X_i^{k-1}\)$ These two vectors define the grounded multimodal semantic plane \(\mathcal{P} = \text{span}\{C_T, C_V\}\), providing an immutable factual subspace within the high-dimensional hidden space.

2. Dynamic Deviation Detection & Critical Layer Selection: Layer-Hierarchical Gating via Risk Metric

Modifying hidden states indiscriminately across all layers compromises low-level feature extraction in early layers and linguistic probability shaping in final layers. Empirical profiling demonstrates that contextual deviation is concentrated within an intermediate transition zone—specifically layers 6 through 26 in models like LLaVA-1.5-7B—where multimodal fusion peaks. To quantify risk, ROT formulates a contextual deviation indicator \(E\): $\(E = (1 - S_T) + (1 - S_V) = 2 - S_T - S_V\)$ where \(S_T = \cos(h_k, C_T)\) and \(S_V = \cos(h_k, C_V)\). Active geometric correction is confined exclusively to this active execution zone, and further restricted on a per-token basis: intervention is triggered if and only if \(E \ge \tau\). This dual-level spatial and token-adaptive gating ensures that factual tokens pass through entirely undisturbed, while tokens exhibiting severe semantic derailment are precisely captured.

3. Norm-Preserving Rotation in Intermediate Layers: Lossless Directional Calibration via Slerp

When a hidden state is flagged for intervention, naïve projection or additive feature injection disrupts the feature activation magnitude, distorting the input distribution to LayerNorm and inducing syntactic collapse. ROT solves this through norm-preserving spherical linear interpolation. Within plane \(\mathcal{P}\), an ideal grounded target direction \(C_{\text{target}}\) is formulated: $\(C_{\text{target}} = \text{Norm}((1 - \alpha) \cdot C_T + \alpha \cdot C_V)\)$ where \(\alpha \in [0, 1]\) balances visual versus textual context proximity. The current hidden state \(h_k\) is then rotated toward \(C_{\text{target}}\) on the hypersphere, while strictly maintaining the original \(L_2\) norm: $\(h_{\text{new}} = \|h_k\| \cdot \text{Norm}(\text{Slerp}(h_k, C_{\text{target}}; \beta))\)$ The hyperparameter \(\beta \in [0, 1]\) modulates the rotation intensity. By enforcing \(\|h_{\text{new}}\| = \|h_k\|\), the intervention redirects the semantic orientation directly back to the factual plane without damping the token's intrinsic representational strength.

4. Representational Smoothing in Subsequent Layers: Inter-Layer Inertial Filtering against Deep Fluctuation

While rotation effectively resolves peak divergence in intermediate layers, enforcing aggressive directional shifts close to the final layers perturbs vocabulary distribution calibration. To guarantee that corrected semantics smoothly propagate to the output head, ROT switches from rotation to representational smoothing across subsequent layers (from layer 27 to \(L\)). At each later layer, the hidden state is updated via a temporal first-order autoregressive filter: $\(h_k = \gamma h_{k-1} + (1 - \gamma) h_k\)$ where \(\gamma \in [0, 1]\) is the smoothing factor. Acting as a low-pass filter, this operation mitigates high-frequency secondary turbulence caused by earlier rotations, ensuring smooth and coherent contextual flow into the language modeling head.

Benchmark & Evaluation Protocol

The empirical evaluation validates ROT across four diverse architectures: LLaVA-1.5-7B, LLaVA-1.5-13B, Qwen2-VL-7B, and the large-scale Qwen2.5-VL-32B. Testing encompasses three authoritative hallucination benchmarks: 1. CHAIR: Assesses open-ended image captioning using sentence-level hallucination rate (\(C_S\)) and instance-level rate (\(C_I\)), alongside Recall and Length metrics; 2. POPE: Evaluates binary object existence probing across Adversarial, Popular, and Random subsets, tracking Accuracy and F1-score; 3. MME: Evaluates fine-grained multimodal perception across Existence, Position, and Color subtasks (each subtask scored up to 200 points based on combined basic and pairwise accuracy). All experiments were conducted on a single computing node equipped with an NVIDIA A800 (80GB) GPU under strictly identical prompt templates.

Key Experimental Results

Main Results

The table below presents the quantitative comparison of ROT against greedy decoding and leading training-free baselines on LLaVA-1.5-7B across CHAIR, POPE, and MME.

Method CHAIR \(C_S\) ↓ CHAIR \(C_I\) ↓ CHAIR Recall ↑ CHAIR Length ↑ POPE Acc ↑ POPE F1 ↑ MME Exist. ↑ MME Pos. ↑ MME Color ↑ MME Total ↑
Greedy 54.4 15.7 75.2 82.7 85.5 85.9 175.67 114.00 151.00 440.67
VCD (Leng et al., 2024) 51.1 14.7 75.2 82.3 85.0 85.3 184.67 128.67 153.00 466.34
OPERA (Huang et al., 2024) 47.0 14.6 74.5 77.1 85.2 84.2 180.67 111.67 123.33 415.67
RITUAL (Woo et al., 2024) 45.2 13.2 74.3 80.3 84.3 85.2 187.50 125.00 164.17 476.67
Vissink (Kang et al., 2025) 52.4 14.5 75.1 83.4 86.5 86.0 190.00 138.33 155.00 483.33
SID (Huo et al., 2024) 44.2 12.8 73.0 80.4 85.8 85.6 183.90 127.80 155.90 467.60
TAME (Tang et al., 2025) 41.3 12.4 74.4 79.9 85.7 85.4 193.00 139.00 164.67 496.67
LVLMs-Saliency (Zhang et al., 2026) 35.6 8.2 75.4 79.5 87.5 87.0 190.00 133.33 175.00 498.33
ROT (Ours) 33.0 6.7 75.6 85.1 86.6 86.3 195.00 143.33 168.33 506.66

The table below demonstrates the generalization performance across diverse model families and scales on CHAIR and the POPE benchmark subsets.

Base Model Method CHAIR \(C_S\) ↓ CHAIR \(C_I\) ↓ POPE Adv Acc ↑ POPE Adv F1 ↑ POPE Pop Acc ↑ POPE Pop F1 ↑ POPE Rand Acc ↑ POPE Rand F1 ↑
LLaVA-1.5-7B Greedy 54.4 15.7 80.4 81.7 86.2 86.4 89.9 89.6
VCD 51.1 14.7 81.2 82.1 85.7 85.9 88.0 88.1
MemVR 50.4 14.3 79.8 81.8 86.3 85.9 89.4 89.6
ROT (Ours) 33.0 6.7 82.3 82.6 87.2 86.6 90.2 89.8
LLaVA-1.5-13B Greedy 41.4 12.7 83.8 84.4 86.1 86.8 87.5 88.4
VCD 44.7 13.1 81.9 81.8 85.7 85.1 87.4 86.6
MemVR 42.2 11.5 81.2 82.4 87.1 85.3 88.2 88.5
ROT (Ours) 30.8 4.3 85.0 84.5 87.7 86.9 89.5 88.6
Qwen2-VL-7B Greedy 24.2 8.0 85.9 84.8 87.6 86.4 89.3 88.6
ROT (Ours) 21.4 4.9 87.3 86.2 89.2 88.5 90.5 89.9
Qwen2.5-VL-32B Greedy 43.6 9.5 85.8 84.6 86.9 85.6 88.5 87.1
ROT (Ours) 30.9 8.7 86.5 85.5 87.4 86.4 89.3 88.1

Ablation Study

To evaluate the necessity of the Norm-Preserving Rotation and Representational Smoothing mechanisms, component ablation experiments were conducted on LLaVA-1.5-7B and Qwen2-VL-7B (CHAIR benchmark):

Configuration LLaVA-1.5-7B \(C_S\) ↓ LLaVA-1.5-7B \(C_I\) ↓ Qwen2-VL-7B \(C_S\) ↓ Qwen2-VL-7B \(C_I\) ↓ Description
Greedy Baseline 54.5 15.7 24.2 8.0 Uncalibrated baseline decoding
w/o Rotation (Smoothing only) 40.5 12.4 23.4 6.6 Disabling rotation causes \(C_I\) to surge, proving passive smoothing cannot steer derailed states
w/o Smoothing (Rotation only) 38.2 8.3 21.9 5.2 Omitting smoothing triggers secondary ungrounded deep fluctuations, raising \(C_I\) by 1.6 points
Full ROT (Complete) 33.0 6.7 21.4 4.9 Optimal synergy delivering decisive hallucination suppression

Key Findings

  • Norm-Preserving Rotation provides the primary corrective power: Ablating rotation on LLaVA-1.5-7B increases instance-level hallucination \(C_I\) from 6.7 to 12.4, demonstrating that direct geometric redirection in the hidden space is essential to halt semantic derailment.
  • Representational Smoothing prevents secondary deep-layer divergence: Disabling smoothing degrades \(C_I\) from 6.7 to 8.3, confirming that abruptly rotated vectors require inertial damping across subsequent layers to stabilize logits convergence.
  • Universal scaling benefits: Across all four tested architectures, ROT achieves an average relative reduction of 26.4% in \(C_S\) and 42.7% in \(C_I\). Even on the massive 32B model Qwen2.5-VL-32B, ROT drops \(C_S\) from 43.6 to 30.9, illustrating that internal geometric derailment is an architecture-agnostic vulnerability that ROT reliably resolves.

Highlights & Insights

  • Counter-Intuitive Geometric Discovery: The paper refutes the prevailing assumption that hallucinations stem merely from language-prior dominance, showing empirically that hallucinated tokens concurrently lose alignment with both textual and visual context anchors in latent space.
  • Norm-Preserving Spherical Interpolation: By employing Slerp combined with explicit norm conservation \(\|h_{\text{new}}\| = \|h_k\|\), ROT re-anchors the semantic orientation toward the factual plane without disrupting activation magnitudes prior to LayerNorm.
  • Layer-Disentangled Dual Intervention: By mapping semantic divergence depth-wise, ROT strategically confines active rotation to intermediate fusion layers (layers 6–26) and transitions to representational smoothing in deep layers, maintaining both corrective efficacy and syntactic stability.

Limitations & Future Work

  • Context Plane Bound to Local Attention: The anchors \(C_T\) and \(C_V\) are derived exclusively from local self-attention weights over the provided image and prompt, omitting external knowledge bases or structured scene graphs that could provide global factual constraints.
  • Heuristic Active Zone and Trigger Threshold: The identification of layers 6–26 and hyperparameter \(\tau\) currently relies on empirical grid searches on representative models. Highly divergent architectures (e.g., Mixture-of-Experts or ultra-deep models) may require automated or differentiable threshold discovery.
  • vs VCD (Leng et al., 2024): VCD requires contrastive decoding between original and visually distorted inputs, effectively doubling forward-pass computational cost and latency. In contrast, ROT operates in-situ on hidden states in a single forward pass with negligible computational overhead.
  • vs OPERA (Huang et al., 2024): OPERA detects over-trust patterns in self-attention matrices and triggers rollback penalties during beam search. ROT shows that attention adjustments indirectly affect deep predictive states, whereas direct rotation on \(h_k\) achieves superior grounding without search rollbacks.
  • vs MemVR (Zou et al., 2024): MemVR forcibly overwrites intermediate hidden states using historical visual key-value representations; ROT instead executes continuous manifold rotation, preserving the token's original representational integrity while realigning its trajectory.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ [Pioneers a hidden-state geometric intervention paradigm, overturning the language-prior orthodoxy via rigorous empirical evidence]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluated extensively from 7B to 32B models across CHAIR, POPE subsets, and MME, with thorough ablations and qualitative dynamics]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical formulations, highly intuitive geometric illustrations, and tightly integrated experimental analysis]
  • Value: ⭐⭐⭐⭐⭐ [An elegant, highly effective, training-free plug-and-play solution that significantly enhances LVLM generation truthfulness at inference time]