Diffusion-Based Immersive Visual Reasoning¶
Conference: ECCV 2026
Paper: ECCV Official
Area: Multimodal VLM / LLM Reasoning
Keywords: Immersive Visual Reasoning, Diffusion Model, Vision-Language Model, Active Perception, Viewport Selection
TL;DR¶
To overcome partial observability and rigid open-loop perception in 360° scenes, DIVR reframes immersive visual reasoning as a denoising diffusion process where a Latent Hypothesis Refiner and a Reasoning-Guided Observer iteratively couple belief updating with active viewport selection, significantly outperforming existing VLMs.
Background & Motivation¶
Immersive 360° visual content plays a vital role across emerging frontiers including virtual reality (VR), augmented reality (AR), and embodied robotics. In contrast to conventional narrow field-of-view (FoV) images where full scene evidence is simultaneously available, 360° environments inherently impose the challenge of partial observability. Critical visual clues required to answer complex questions are typically scattered across different spatial viewpoints, frequently residing completely outside an agent's immediate perspective. This requires an agent to behave not merely as a passive observer, but as an active explorer that dynamically determines where to look next based on evolving reasoning requirements.
However, existing Vision-Language Models (VLMs) remain constrained by an inflexible "look first, think later" open-loop paradigm. To process spherical panoramas, prior approaches typically adopt one of two strategies: either compressing the entire equirectangular image into global representations via spherical convolutions/attention, which introduces severe geometric distortions and information overload that drown fine-grained local details; or pre-sampling the scene into a static collection of viewports (such as six cubemap faces or static saliency-based crops) and processing them in parallel. Both strategies fundamentally decouple perception from reasoning. The agent is forced to reason over a pre-determined, static visual budget. If critical evidence—such as a small non-salient doorway or an off-center ceiling fixture—is omitted during the initial static sampling, the reasoning chain breaks irreparably.
Overcoming this limitation necessitates reshaping immersive reasoning into an active, closed-loop system where internal reasoning uncertainty actively directs perceptual exploration, and newly gathered evidence continuously refines the reasoning hypothesis. Core idea: reframe immersive visual reasoning as an iterative probabilistic belief updating process modeled via a denoising diffusion process, where a Latent Hypothesis Refiner (LHR) and a Reasoning-Guided Observer (RGO) operate in a tight feedback loop to dynamically gather evidence and resolve reasoning uncertainty.
Method¶
Overall Architecture¶
DIVR models multi-step visual reasoning as a conditional reverse diffusion process in a continuous semantic latent space. The agent's evolving comprehension and reasoning plan are encapsulated in a latent reasoning hypothesis vector \(z_t \in \mathbb{R}^D\). The reasoning trajectory initiates at timestep \(t = T\) with maximal uncertainty represented by standard Gaussian noise \(z_T \sim \mathcal{N}(0, \mathbf{I})\), and progressively denoises over \(T\) discrete steps into a coherent, evidence-grounded hypothesis vector \(z_0\).
Within each reverse diffusion cycle from step \(t\) to \(t-1\), the system executes a tightly coupled perception-reasoning loop: first, the Reasoning-Guided Observer (RGO) uses the current uncertain hypothesis \(z_t\) and the query \(q\) to actively select the most informative candidate viewport via a learned policy network, extracting visual tokens and updating an accumulated observation history; subsequently, the Latent Hypothesis Refiner (LHR) performs one step of conditional Transformer denoising on \(z_t\), using the accumulated multi-modal evidence as conditioning context, yielding the refined hypothesis \(z_{t-1}\). Once \(T\) refinement cycles conclude, the fully denoised hypothesis \(z_0\) is prepended as prefix tokens to a frozen/fine-tuned VLM decoder to generate the final textual answer.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Equirectangular Panorama + Question"] --> B["Initial Gaussian Noise Hypothesis zT"]
B --> C["Reasoning-Guided Observer<br/>Gumbel-Softmax viewport selection & GRU accumulation"]
C --> D["Latent Hypothesis Refiner<br/>Transformer cross-attention conditional denoising"]
D -->|Update hypothesis to zt-1| C
D -->|Converge to z0 after T steps| E["VLM Decoder<br/>Prefix-guided final answer generation"]
Key Designs¶
1. Latent Hypothesis Refiner (LHR): Denoising-driven hypothesis evolution Classical symbolic execution networks easily collapse under partial observability when early visual steps fail, while discrete autoregressive text reasoning lacks continuous gradient signals for active perceptual steering. The LHR represents the reasoning state as a continuous vector \(z_t\) and frames uncertainty elimination as a reverse diffusion process. Consisting of \(L=6\) stacked Transformer blocks, LHR projects the hypothesis \(z_t\) and sinusoidal timestep embeddings into Queries, while projecting the conditioning vector \(c_t = [q; x_t]\) (formed by concatenating question embeddings \(q\) and accumulated visual representations \(x_t\)) into Keys and Values. Multi-head cross-attention allows the latent hypothesis to directly attend to visual evidence addressing current reasoning gaps, predicting the noise component \(\boldsymbol{\epsilon}\) and reconstructing the refined hypothesis state \(z_{t-1}\). The smooth evolution in continuous latent space provides substantial resilience against incomplete intermediate evidence.
2. Reasoning-Guided Observer (RGO): Task-driven differentiable active perception policy Standard panoramic equi-sampling or generic human saliency predictions capture only high-contrast visual regions, overlooking low-saliency features that provide crucial structural transitions (e.g., an unobtrusive corridor connecting two spaces). The RGO turns viewport selection into a goal-directed search policy. Across \(K=36\) uniformly distributed overlapping candidate viewports covering multiple FoV scales, a 3-layer MLP policy network \(\pi_\phi\) processes concatenated embeddings \([z_t; q; v_k]\) to compute a scalar utility score \(u_k = \text{MLP}_\phi([z_t; q; v_k])\). To ensure end-to-end differentiability during backpropagation, RGO leverages Gumbel-Softmax reparameterization to sample selection probabilities: $\(p_k = \frac{\exp((u_k + g_k)/\tau)}{\sum_{j=1}^{K} \exp((u_j + g_j)/\tau)}\)$ where \(g_k \sim \text{Gumbel}(0, 1)\) and \(\tau\) is a temperature hyperparameter. The sampled viewport feature \(\hat{v}_t\) is fed into a Gated Recurrent Unit (GRU) to maintain temporal observation history \(O_{\ge t}\), mapped via an Adapter into visual tokens \(x_t\). As \(z_t\) denoises, RGO adaptively transitions from broad exploratory gazing to fine-grained, target-focused inspection.
3. Teacher Knowledge Distillation & Closed-Loop Joint Optimization To endow the continuous latent vector \(z_0\) with clear semantic ground truth while bypassing expensive dense manual annotation of step-by-step reasoning chains, the authors adopt an offline knowledge distillation strategy. A frontier VLM (GPT-5) functions as a passive supervisor, receiving the 360° panorama, question, ground-truth answer, and salient viewports selected via eye-tracking and inhibition-of-return masks. GPT-5 generates a natural-language rationale outlining explicit deduction steps, whose language embedding serves as the target \(z_0\). The entire pipeline is trained end-to-end using a composite objective: $\(\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{diff}} + \lambda \mathcal{L}_{\text{cls}}\)$ where \(\mathcal{L}_{\text{diff}} = \mathbb{E}_{t, \boldsymbol{\epsilon}}[\|\boldsymbol{\epsilon} - s_\theta(z_t, t, c_t)\|^2]\) trains LHR to faithfully denoise hypotheses, and \(\mathcal{L}_{\text{cls}} = -\log p(y | z_0, c_0)\) guides the VLM head to predict correct answers. This joint optimization forces policy network \(\pi_\phi\) to discover viewpoints that maximally diminish reasoning uncertainty.
A Worked Example¶
Consider a multi-hop spatial query: "Is the kitchen connected to a room with a chandelier?" 1. Initial State (\(t=T\)): Latent vector \(z_T\) is pure Gaussian noise corresponding to broad intent ("Locate kitchen, verify connection to room with chandelier"). RGO selects Viewport 1 centering on high-saliency cabinets and countertops (Kitchen confirmed). 2. Intermediate Denoising (\(t \to t-1\)): LHR cross-attends to Viewport 1, updating the hypothesis to focus on finding passageways leading out of the kitchen. RGO bypasses kitchen appliances and actively inspects an adjacent, low-saliency doorway (Viewport 2). 3. Convergence (\(t \to 0\)): Viewport 2 confirms a connected room transition. The hypothesis denoises to \(z_0\) ("Scan adjoining room's ceiling for light fixtures"). Guided by this specific instruction, RGO navigates directly to Viewport 3 capturing an elevated chandelier, allowing the VLM decoder to confidently output "Yes".
Key Experimental Results¶
Main Results¶
DIVR was evaluated across three panoramic benchmarks: IQVA (immersive video keyframe VQA), 360+X (panoptic scene understanding converted to structured VQA), and Zillow Indoor (ZInD, indoor room layout spatial reasoning split into single-viewport Easy and multi-viewport Hard subsets). Baselines include spherical-specialized SPHTR, high-resolution SLiME-8B, and frontier GPT-5 across three input modalities.
| Model | Input Strategy | IQVA Acc (%) | 360+X Acc (%) | ZInD Easy (%) | ZInD Hard (%) |
|---|---|---|---|---|---|
| SPHTR | Equirectangular | 45.96 | 48.26 | 57.73 | 23.55 |
| SLiME-8B | Equirectangular | 41.78 | 48.62 | 61.48 | 28.46 |
| SLiME-8B | Cubemap | 46.50 | 54.18 | 63.90 | 33.35 |
| SLiME-8B | Saliency | 53.43 | 54.41 | 65.85 | 34.14 |
| GPT-5 (Zero-shot) | Equirectangular | 47.78 | 50.36 | 61.04 | 33.30 |
| GPT-5 (Zero-shot) | Cubemap | 53.42 | 56.47 | 65.82 | 35.49 |
| GPT-5 (Zero-shot) | Saliency | 55.84 | 58.62 | 68.01 | 35.64 |
| DIVR + GLM-4.1V-9B | Active Loop (DIVR) | 56.61 | 63.77 | 71.33 | 41.36 |
| DIVR + Qwen2.5-VL-7B | Active Loop (DIVR) | 56.75 | 64.72 | 70.45 | 41.89 |
Ablation Study¶
Ablations dissecting the individual and combined impact of LHR and RGO were conducted using Qwen2.5-VL-7B as the base model:
| Reasoning Config | Perception Policy | IQVA Acc (%) | 360+X Acc (%) | ZInD Easy (%) | ZInD Hard (%) |
|---|---|---|---|---|---|
| w/o LHR (Direct VLM) | Static Saliency | 52.69 | 56.72 | 67.30 | 36.98 |
| w/o LHR (Direct VLM) | Heuristic CLIP | 53.22 | 57.90 | 67.57 | 39.23 |
| w/o LHR (Direct VLM) | Active Policy (RGO) | 54.24 | 58.51 | 68.34 | 39.44 |
| LHR (Diffusion Denoising) | Static Saliency | 53.46 | 57.71 | 66.39 | 37.58 |
| LHR (Diffusion Denoising) | Heuristic CLIP | 55.14 | 58.59 | 67.40 | 38.68 |
| LHR (Diffusion Denoising) | Active Policy (RGO) | 56.75 | 64.72 | 70.45 | 41.89 |
Key Findings¶
- Crucial advantage on multi-step reasoning: On the ZInD Hard split requiring complex cross-viewpoint evidence gathering, DIVR achieves an absolute accuracy gain of over 6.2% compared to the strongest saliency-guided baseline (41.89% vs 35.64%), highlighting the limitations of static crops.
- Closed-loop paradigm dwarfs input format engineering: Switching input formats from equirectangular to cubemap or saliency yields modest improvements (2%~5%), whereas incorporating the DIVR closed-loop active perception framework produces a substantial performance leap (e.g., from 55.92% to 64.72% on 360+X).
- Strong perception-reasoning synergy: Disabling either LHR or RGO triggers noticeable degradation. When paired, DIVR's selected viewports mirror human gaze far more closely than static saliency on IQVA, boosting Normalized Scanpath Saliency (NSS) from 0.36 to 0.49 and lowering KL divergence from 2.68 to 2.34.
- Controllable test-time compute trade-off: On ZInD Hard, using just \(T=5\) diffusion steps delivers 38.46% accuracy at 3.57 QPS (already surpassing all baselines). Extending to \(T=20\) reaches peak accuracy of 41.89% at 1.08 QPS, providing flexible efficiency-accuracy trade-offs.
Highlights & Insights¶
- Formulating reasoning uncertainty as diffusion denoising: Bridging generative diffusion mechanics with latent reasoning provides a mathematically sound, differentiable approach to belief updating that naturally handles ambiguity and missing information.
- End-to-end differentiable perception via Gumbel-Softmax: Relaxing discrete viewport selection allows downstream task gradients to shape active gaze exploration, teaching the observer to actively bypass visual clutter and prioritize decision-critical evidence.
- Extensible active perception paradigm: The core methodology readily extends beyond 360° images to gigapixel pathology/satellite imagery inspection, long-form video temporal skipping, and embodied robotic visual search.
Limitations & Future Work¶
- Linear latency scaling with diffusion steps: Executing sequential reverse diffusion steps requires repeated feature aggregation and cross-attention passes, resulting in lower queries per second compared to single-pass models (1.08 QPS at \(T=20\)).
- Reliance on distilled teacher rationales: Ground-truth target hypotheses \(z_0\) depend on offline GPT-5 rationales; subtle hallucinations in teacher explanations could inject structured noise into the latent space.
- Future directions: Incorporating consistency models or few-step flow matching to accelerate denoising to 2-4 steps, and exploring online reinforcement learning (such as GRPO) with pure outcome-based rewards to remove distillation dependence.
Related Work & Insights¶
- vs SPHTR / Panoramic CNNs: Traditional methods attempt to mitigate distortion using specialized spherical convolutions on static projections; DIVR bypasses distortion entirely by adaptively steering standard perspective crops to critical regions.
- vs Embodied VQA: Embodied AI approaches entangle visual reasoning with complex physical navigation dynamics and sparse rewards; DIVR isolates the core reasoning problem by formalizing lightweight active perception (where to look, not how to move).
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ [Pioneering integration of diffusion models with active viewport selection for immersive visual reasoning]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across 3 benchmarks, multiple input formats, detailed ablation of reasoning/perception coupling, and human fixation alignment]
- Writing Quality: ⭐⭐⭐⭐⭐ [Rigorous formalization, lucid architectural diagrams, and compelling qualitative step-by-step case studies]
- Value: ⭐⭐⭐⭐⭐ [Offers a robust, principled blueprint for multi-step reasoning under partial observability in multimodal AI]