ORBIT: Overcoming Hallucination Risks via Bi-manifold Interaction and Traction

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Area: Multimodal VLM
Keywords: Large Vision-Language Models / Hallucination Mitigation / Representation Geometry / Bi-manifold Hypothesis / Semantic Turbulence Monitoring

TL;DR

Addressing the "anchoring trap" where conventional attention-based detectors miss hallucinations despite accurate visual focus, ORBIT formulates a bi-manifold representation geometry hypothesis, dynamically triggering an orthogonal residual intervention via the Geo-Semantic Turbulence Index (GSTI) to calibrate hidden states without fine-tuning.

Background & Motivation

Large Vision-Language Models (LVLMs) demonstrate exceptional multimodal perception and multi-step reasoning by bridging large language model backbones with visual encoders. However, their pervasive propensity toward "hallucination"—generating textual claims inconsistent with visual facts—presents a severe barrier to deployment in safety-critical domains such as medical diagnostics and autonomous driving. To mitigate hallucinations, early approaches relied on supervised fine-tuning or preference alignment (e.g., RLHF / DPO). Nevertheless, retraining incurs prohibitive computational costs and frequently degrades linguistic fluency, causing catastrophic forgetting of general commonsense priors.

Consequently, research has pivoted toward lightweight, tuning-free inference-time interventions. Most existing techniques rely on internal attention diagnostics, operating under the assumption that hallucinations stem from visual attention dispersion or neglect. However, deeper analysis reveals the "anchoring trap": even when cross-attention attends accurately to visual targets (e.g., correctly focusing on a solitary apple), strong parametric language priors in deeper transformer layers can overpower visual evidence, generating contradictory outputs (e.g., claiming "two apples"). Thus, attention maps alone cannot detect representational decoupling.

To overcome this dilemma, this paper investigates the geometry of LVLM internal representations across transformer layers. Empirical findings show that hallucinations coincide with an abrupt geometric drift: hidden representations detach from a static visual subspace anchor and become co-opted by a dynamic linguistic subspace governed by textual co-occurrence. Core idea: Impose dual geometric constraints by constructing static visual and dynamic linguistic subspaces, monitor representational decoupling in real time via the Geo-Semantic Turbulence Index (GSTI), and execute closed-loop calibration through orthogonal residual extraction, visual recentering, and norm-preserving injection upon detecting turbulence anomalies.

Method

Overall Architecture

ORBIT formalizes the evolution of LVLM hidden states as discrete dynamical trajectories within Euclidean space \(\mathcal{H} = \mathbb{R}^d\). During inference, hidden states are projected onto two geometric structures: a prefill-cached static visual subspace \(\mathcal{S}_{vis}^{(l)}\) and an autoregressively extracted dynamic language subspace \(\mathcal{S}_{lang}^{(t,l)}\). During generation, the system computes the Geo-Semantic Turbulence Index (GSTI) at each step. When the index exceeds a statistical \(3\sigma\) threshold, indicating representational derailment, closed-loop intervention extracts orthogonal residuals, recenters onto visual anchors, and injects calibrated features while preserving euclidean norm before continuing forward propagation.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Inputs & Hidden Trajectory<br/>Image Patches and Autoregressive Text Sequence"] --> B["Dual Subspace Spectral Projection<br/>Cached Static Visual Space + Dynamic Language Space"]
    B --> C["Geo-Semantic Turbulence Monitoring<br/>Measure Visual Deviation & Semantic Volatility (GSTI)"]
    C -->|GSTI Exceeds 3σ Threshold| D["Closed-Loop Geometric Intervention<br/>Orthogonal Residual Extraction → Visual Recentering → Norm-Preserving Injection"]
    C -->|Normal Generation| E["Maintain Original States<br/>Standard Autoregressive Forward Pass"]
    D --> F["Calibrated Hidden State<br/>Restored Visual Anchoring for Downstream Layers"]
    E --> F

Key Designs

1. Dual Subspace Spectral Decomposition: Disentangling Static Vision from Dynamic Text Treating multimodal hidden states as unified mixtures obscures the underlying competition between sensory evidence and language priors. ORBIT decouples hidden state geometry into static visual and dynamic linguistic subspaces. During prefilling, layer-specific visual patch activation matrices \(\mathbf{V}^{(l)} \in \mathbb{R}^{N_v \times d}\) undergo randomized SVD: $\((\mathbf{V}^{(l)} - \boldsymbol{\mu}_{vis}^{(l)})^\top \approx \mathbf{U}^{(l)} \mathbf{\Sigma}^{(l)} (\mathbf{Q}^{(l)})^\top\)$ The top \(k\) principal components capturing \(\eta = 0.95\) cumulative variance form the orthogonal projection operator \(\mathbf{P}_{vis}^{(l)} = \mathbf{U}_{1:k}^{(l)} (\mathbf{U}_{1:k}^{(l)})^\top\), computed once and cached with zero decoding latency. During autoregressive decoding, a sliding window of size \(K=5\) captures recent pure text tokens \(\mathbf{H}_{ctx}^{(t,l)} \in \mathbb{R}^{K \times d}\). Thin SVD yields the dynamic language projector \(\mathbf{P}_{lang}^{(t,l)} = \mathbf{U}_{lang, 1:k_{lang}}^{(t,l)} (\mathbf{U}_{lang, 1:k_{lang}}^{(t,l)})^\top\) to model instantaneous syntactic momentum.

2. Geo-Semantic Turbulence Index: Adaptive Gating via Deviation and Volatility Faithful tokens remain aligned with the visual subspace, abstract words exhibit stable high deviation, and hallucinated tokens manifest abrupt detachment combined with trajectory volatility. The Geo-Semantic Turbulence Index (GSTI, \(\Omega_t^{(l)}\)) fuses two geometric statistics: Visual deviation \(\delta_t^{(l)}\) quantifies visual subspace energy leakage: $\(\delta_t^{(l)} = \frac{\|(\mathbf{I} - \mathbf{P}_{vis}^{(l)}) \mathbf{h}_t^{(l)}\|_2}{\|\mathbf{h}_t^{(l)}\|_2} \in [0, 1]\)$ Semantic volatility \(\kappa_t^{(l)}\) measures trajectory instability across layers via velocity vectors \(\mathbf{v}_t^{(l)} = \mathbf{h}_t^{(l)} - \mathbf{h}_t^{(l-1)}\): $\(\kappa_t^{(l)} = 1 - \frac{\langle \mathbf{v}_t^{(l)}, \mathbf{v}_t^{(l-1)} \rangle}{\|\mathbf{v}_t^{(l)}\|_2 \|\mathbf{v}_t^{(l-1)}\|_2}\)$ These are integrated via a scaled sigmoid gating function: $\(\Omega_t^{(l)} = \sigma\left(1.702 \cdot \frac{\delta_t^{(l)} - \mu_\delta}{\sigma_\delta}\right) \cdot \kappa_t^{(l)}\)$ Intervention triggers exclusively when \(\Omega_t^{(l)} > \mu_\Omega^{(l)} + 3\sigma_\Omega^{(l)}\), preventing false positives on abstract language tokens.

3. Closed-Loop Geometric Intervention: Residual Extraction, Recentering, and Norm-Preserving Injection Naively adding raw visual embeddings disrupts LayerNorm scaling and attention distributions. ORBIT executes a three-stage calibration: - Orthogonal Residual Extraction: Project hidden states onto the orthogonal complement of the language subspace: \(\mathbf{r}_{ortho}^{(t,l)} = (\mathbf{I} - \mathbf{P}_{lang}^{(t,l)}) \mathbf{h}_t^{(l)}\), isolating suppressed visual residuals from linguistic momentum. - Visual Recentering: Using \(\mathbf{r}_{ortho}^{(t,l)}\) as a query, compute normalized attention scores over cached visual patch tokens with nucleus sampling (\(\rho=0.9\)) to filter noise, projecting the aggregate centroid back onto the visual subspace: $\(\tilde{\mathbf{c}}_{vis}^{(t,l)} = \mathbf{P}_{vis}^{(l)} \left( \sum_{i \in \mathcal{S}_{vis}^{(t,l)}} \beta_i \mathbf{v}_i^{(l)} \right)\)$ - Norm-Preserving Injection: Blend the calibrated centroid while strictly preserving original Euclidean magnitude: $\(\hat{\mathbf{h}}_t^{(l)} = \|\mathbf{h}_t^{(l)}\|_2 \cdot \frac{(1 - \lambda_t) \mathbf{h}_t^{(l)} + \lambda_t \tilde{\mathbf{c}}_{vis}^{(t,l)}}{\|(1 - \lambda_t) \mathbf{h}_t^{(l)} + \lambda_t \tilde{\mathbf{c}}_{vis}^{(t,l)}\|_2}\)$ where \(\lambda_t = (1 - H(\boldsymbol{\beta}) / \log |\mathcal{S}_{vis}^{(t,l)}|) \cdot \tanh(\Omega_t^{(l)})\), modulating intervention intensity dynamically according to visual entropy and turbulence severity.

Key Experimental Results

Main Results

Evaluations across hallucination benchmarks (POPE, CHAIR) on LLaVA-v1.5-7B and Qwen2.5-VL-7B comparing against inference-time intervention baselines:

Model Backbone	Method	POPE F1 ↑	POPE Acc ↑	CHAIR CS ↓	CHAIR CI ↓	Output Length
LLaVA-v1.5-7B	Baseline	85.3	84.1	51.2	15.3	102.0
LLaVA-v1.5-7B	DoLa	80.4	83.0	56.8	15.1	97.4
LLaVA-v1.5-7B	VCD	85.2	85.1	50.8	14.8	102.1
LLaVA-v1.5-7B	OPERA	84.1	85.3	47.2	14.5	95.5
LLaVA-v1.5-7B	HALC	84.0	83.9	50.4	12.3	97.0
LLaVA-v1.5-7B	SID	85.7	85.9	44.0	12.1	99.5
LLaVA-v1.5-7B	VAR	86.1	86.4	44.2	14.6	103.2
LLaVA-v1.5-7B	AGLA	84.7	85.4	43.2	14.0	98.6
LLaVA-v1.5-7B	ORBIT (Ours)	87.5	88.0	38.1	8.5	100.3
Qwen2.5-VL-7B	Baseline	88.4	89.0	26.5	7.2	98.4
Qwen2.5-VL-7B	DoLa	86.8	87.5	29.1	8.5	95.2
Qwen2.5-VL-7B	VCD	88.5	89.0	26.8	7.4	100.1
Qwen2.5-VL-7B	OPERA	88.5	88.9	26.2	7.3	94.8
Qwen2.5-VL-7B	SID	88.9	89.3	26.1	7.8	98.8
Qwen2.5-VL-7B	VAR	89.1	89.5	25.5	7.5	101.5
Qwen2.5-VL-7B	ORBIT (Ours)	90.3	90.8	18.4	5.3	98.5

General VQA and fine-grained visual perception benchmark results:

Backbone	Method	POPE ↑	CHAIR CS ↓	CHAIR CI ↓	MME ↑	MMBench ↑	CV-Bench ↑	CLEVR ↑
LLaVA-v1.5-7B	Baseline	85.3	51.2	15.3	1495.5	64.2	56.6	43.6
LLaVA-v1.5-7B	VAR	86.1	44.2	14.6	1512.2	65.1	56.8	44.6
LLaVA-v1.5-7B	ORBIT (Ours)	87.5	38.1	8.5	1525.8	66.2	58.0	46.4
LLaVA-v1.5-13B	Baseline	85.8	24.6	8.2	1501.2	67.7	58.2	68.0
LLaVA-v1.5-13B	ORBIT (Ours)	88.2	16.4	6.1	1542.5	69.1	60.2	70.1
Qwen2.5-VL-7B	Baseline	88.4	26.5	7.2	1973.3	82.1	73.5	74.3
Qwen2.5-VL-7B	ORBIT (Ours)	90.3	18.4	5.3	1988.2	83.8	74.6	75.5
Qwen2.5-VL-32B	Baseline	90.5	18.5	4.5	2046.5	83.5	77.0	78.2
Qwen2.5-VL-32B	ORBIT (Ours)	91.2	15.2	4.1	2080.4	84.5	78.1	79.4

Ablation Study

Component ablations on LLaVA-v1.5-7B evaluating GSTI gating and intervention steps:

Configuration	Deviation \(\delta\)	Volatility \(\kappa\)	Residual Ext.	Recentering	Norm Pres.	POPE F1 ↑	CHAIR CS ↓	CHAIR CI ↓	Note
Baseline	–	–	–	–	–	85.3	51.2	15.3	Unmodified model
No GSTI Gating	–	–	✓	✓	✓	86.9	41.3	10.1	Blind intervention every step
w/o Deviation \(\delta\)	–	✓	✓	✓	✓	87.0	40.7	9.8	Lacks geometric distance signal
w/o Volatility \(\kappa\)	✓	–	✓	✓	✓	87.2	39.8	9.4	Ignores trajectory velocity drift
w/o Residual Ext.	✓	✓	–	✓	✓	86.0	45.8	12.4	Injected vision polluted by text priors
w/o Recentering	✓	✓	✓	–	✓	86.3	44.2	11.6	High-frequency noise degrades anchor
w/o Norm Pres.	✓	✓	✓	✓	–	86.6	42.7	10.8	Alters layer norm energy distribution
Full ORBIT	✓	✓	✓	✓	✓	87.5	38.1	8.5	Optimal synergy across all metrics

Latency comparisons show ORBIT incurs only \(1.8\times\) baseline runtime (\(7.73\text{s}\) vs. \(4.21\text{s}\) per sample), outperforming greedy rollback approaches like OPERA (\(29.2\text{s}\), \(6.9\times\)) and HALC (\(26.6\text{s}\), \(6.3\times\)).

Key Findings

• Residual Extraction is Crucial: Omitting orthogonal residual extraction degrades CHAIR CS most severely (from 38.1 to 45.8), proving that visual re-injection fails unless active linguistic momentum is explicitly subtracted.
• Evading the Refusal Trap: While baselines like OPERA reduce hallucinations partly by truncating output length (95.5 tokens), ORBIT maintains standard output length (100.3 / 98.5 tokens), correcting semantic direction rather than resorting to brief evasive responses.
• Robustness on Frontier Architectures: On Qwen2.5-VL-7B where attention-based methods like AGLA experience performance regression (POPE drops from 88.4 to 87.7), ORBIT elevates POPE to 90.3 and reduces CHAIR CS by 30.5% (26.5 \(\to\) 18.4).

Highlights & Insights

• Resolving the Anchoring Trap: Demonstrates that high cross-attention is necessary but insufficient for grounding; internal representational trajectory direction governs whether hallucinations manifest.
• Efficient Static-Dynamic Decoupling: Pre-computing visual SVD projections during prefilling ensures inference-time intervention requires only lightweight inner products and projections.
• Constructive Geometry over Logit Penalties: Rather than penalizing token probabilities, ORBIT guides hidden states along verified visual manifolds, yielding simultaneous performance gains on fine-grained spatial and counting tasks.

Limitations & Future Work

• Linear Subspace Assumption: While layer-specific SVD approximates local geometry, highly nonlinear cross-modal manifolds may incur minor geometric information loss under linear projections.
• Visual Encoder Quality Dependency: Calibration relies on initial visual patch features; extreme sensor degradation or low resolution may limit recovery headroom.
• Future Directions: Exploring Riemannian manifold projections and extending bi-manifold trajectory monitoring to Video-LLM temporal reasoning.

Related Work & Insights

• vs. Attention-Based Methods (OPERA, VAR): Attention interventions fail when models fall into the anchoring trap. ORBIT acts directly on hidden states to steer representations away from language prior dominance.
• vs. Logit Contrastive Decoding (VCD, DoLa): Contrastive methods perturb output logits without geometric interpretability; ORBIT operates internally with norm preservation, maintaining generation stability and fluency.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Groundbreaking bi-manifold representation geometry framework formalizing the anchoring trap and trajectory decoupling.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across 4 model scales, 6 diverse benchmarks, detailed latency profiles, and comprehensive ablations.
• Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical formulation, clear conceptual motivation, and polished empirical exposition.
• Value: ⭐⭐⭐⭐⭐ Practical tuning-free intervention that substantially improves LVLM reliability with minimal computational overhead.