Skip to content

How and What to Imagine? Visual Thinking in Unified Multimodal Models for Cross-View Spatial Reasoning

Conference: EMNLP 2026
Paper: ACL Anthology
Code: https://github.com/MyLittleChange/VDrop
Area: Multimodal VLM
Keywords: Cross-View Spatial Reasoning, Unified Multimodal Models, Visual Thinking, View Dropout, Learnability-Informativeness Tradeoff

TL;DR

Addressing the pervasive failure where unified multimodal models generate intermediate "thinking-images" only to bypass them when answering, this paper introduces View Dropout (VDrop) to make visual thinking causally load-bearing, and establishes a Learnability–Informativeness tradeoff framework demonstrating that panoramic visual thinking is the optimal intermediate representation for out-of-domain cross-view spatial reasoning.

Background & Motivation

Cross-view spatial reasoning requires an agent to deduce the three-dimensional layout of a scene, the relative metric distances among objects, and their viewer-centric orientations across partially overlapping visual fields. It serves as a bedrock capability for diverse multimodal applications, spanning from mobile embodied agents navigating indoor rooms to video foundation models integrating temporally disparate observations. Despite remarkable progress in single-image scene perception, even state-of-the-art vision-language models (VLMs) perform marginally above chance on rigorous multi-view benchmarks. This persistent deficiency originates from a fundamental representational mismatch: cross-view spatial reasoning is inherently visual and geometric, whereas standard VLMs are confined to reasoning exclusively via symbolic linguistic tokens, verbalizing fine-grained geometric configurations into coarse textual tokens that inevitably discard the precise spatial structures required by the task.

To bridge this representational divide, emerging paradigms explore "thinking with images," empowering unified multimodal models (UMMs) to natively synthesize intermediate visual representations—such as depth maps, camera trajectories, or reconstructed layouts—prior to predicting textual answers. However, recent critical analyses reveal a pervasive under-use phenomenon: under standard supervised fine-tuning (SFT), the visual content of these intermediate images is predominantly ignored by the subsequent answer-generation pathway. The model takes a shortcut directly from the original input views, rendering the generated thinking-image a merely decorative side-product of training rather than a causally functional reasoning step. This raises the critical foundational question: how can we train multimodal models so that visual thinking becomes genuinely indispensable and causally load-bearing during inference?

Once the intermediate visual representation is forced to carry true perceptual weight, an equally vital question emerges: which visual thinking representation is most effective for cross-view reasoning? Prior research has introduced diverse candidate representations—such as bird's-eye top-down floorplans, panoramic view unifications, and cross-view point-matching overlays—without disentangling the intrinsic value of the representation from the model's practical capability to generate it. Core idea: View Dropout (VDrop) masks a contiguous region of one input view from the answer span during training to block input shortcuts and force reasoning to route through the thinking-image, paired with a Learnability–Informativeness (L–I) tradeoff framework showing that panoramic representations uniquely combine high spatial informativeness with feasible generative learnability.

Method

Overall Architecture

The framework operates on unified multimodal models (UMMs, such as the diffusion-coupled BAGEL or the flow-matching-based SenseNova-U1) capable of native interleaved image-text generation. Given two partially overlapping input views \(V_1, V_2\) and a spatial query \(q\), the model autoregressively generates an end-to-end visual-thinking trace sequence: \((V_1, V_2, q) \to I_{vt} \to a\), where \(I_{vt}\) is the synthesized intermediate thinking-image and \(a\) is the final textual answer.

To prevent the answer generation pathway from taking shortcuts directly through \(V_1\) and \(V_2\), the training regime incorporates View Dropout (VDrop), which dynamically manipulates the decoder attention mask to obscure visual evidence from the answer tokens while leaving the thinking-image generation pathway fully unconstrained. Guided by the Learnability–Informativeness framework, the model is trained on procedurally generated indoor environments (Infinigen Indoors) using a structured curriculum that smoothly transitions the model from unmasked representation learning to fully masked causal routing.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Overlapping Dual Views V1, V2 + Question q"] --> B["View Dropout Attention Mask<br/>Mask contiguous patch region of V1/V2 from answer span"]
    B --> C["Visual Thinking Strategy Synthesis<br/>Generate intermediate thinking-image I_vt with full view attention"]
    C --> D["Progressive Annealing Curriculum<br/>Warmup with zero mask, then linearly anneal to full masking"]
    D --> E["Output: Grounded Answer a Routed via Thinking-Image"]

Key Designs

1. View Dropout Attention Mask: Eliminating Shortcut Connections to Enforce Causal Dependency In conventional transformer architectures with causal or cross-attention, answer query tokens \(Q_a\) attend simultaneously to the input views \(V_1, V_2\), the query \(q\), and the generated thinking-image \(I_{vt}\). Because all spatial cues needed to solve the question are already embedded across the two input images, the optimizer converges to directly reading \(V_1\) and \(V_2\), bypassing \(I_{vt}\) entirely. To shatter this shortcut, VDrop samples a single primary view \(v \in \{V_1, V_2\}\) uniformly at each training iteration, and defines a contiguous axis-aligned rectangular region of patch tokens \(D_v\) covering a fraction \(\rho\) (default \(\rho = 0.5\)) of view \(v\). The answer-span attention mask \(M\) is explicitly modified to sever attention to these patches: $\(M_{qk} \leftarrow -\infty, \quad \forall q \in Q_a, \; k \in D_v\)$ Crucially, the attention mask for thinking-image queries \(Q_{vt}\) remains unperturbed (\(M_{qk} = 0\)), allowing \(I_{vt}\) to observe both complete input views. Because \(D_v\) is contiguous rather than scattered, the answer tokens cannot interpolate the missing content from adjacent patches, compelling the answer pathway to retrieve the hidden scene structure exclusively through \(I_{vt}\). In dual-encoder architectures like BAGEL, the mask is applied jointly across both semantic ViT tokens and generative VAE tokens to prevent inter-channel information leakage.

2. Progressive Annealing Curriculum: Preventing Early Generative Collapse via Phased Masking Enforcing VDrop from the very beginning of training (step 0) causes severe gradient divergence and learning collapse: the model is forced to route evidence through \(I_{vt}\) before it has learned what \(I_{vt}\) should represent or how to synthesize coherent visual layouts. To ensure stable convergence, the masking probability \(p_{\text{mask}}(s)\) is scheduled over training step \(s\) via a tripartite curriculum: $\(p_{\text{mask}}(s) = \begin{cases} 0, & s < s_w \\ \frac{s - s_w}{s_a}, & s_w \le s < s_w + s_a \\ 1, & s \ge s_w + s_a \end{cases}\)$ During the warmup interval (\(s_w = 500\) steps), \(p_{\text{mask}} = 0\), allowing the model to establish stable generation of \(I_{vt}\) alongside unconstrained answer decoding. In the annealing phase (\(s_a = 1500\) steps), the dropout probability increases linearly, gradually shifting the answer query's reliance away from the raw inputs and toward the thinking-image. Beyond \(s_w + s_a\), the masking is fully active (\(p_{\text{mask}} = 1\)), cementing \(I_{vt}\) as an essential perceptual intermediary.

3. Learnability–Informativeness Tradeoff Framework: Grounding the Choice of Visual Intermediate Having established that VDrop makes visual thinking causally necessary, the paper formulates the selection among candidate visual thinking representations as an imitation-learning bound comprising two orthogonal axes: - Informativeness \(I(T)\): The theoretical upper bound on downstream reasoning difficulty reduction provided by an optimal representation \(T\), evaluated by supplying ground-truth visual renders to frozen frontier VLMs (Qwen3-VL-32B/235B). Panoramic renders yield massive performance uplifts (+10.83% on Qwen3-VL-235B) across distance and directional reasoning, whereas point-matching overlays provide negligible or negative gains (-0.83%) as they remain locked in the original camera perspectives. - Learnability \(L(T)\): The fidelity and reliability with which a UMM can synthesize instances of \(T\) from data. Top-down bird's-eye views offer substantial informativeness (+2.92%), but require the model to invent an unseen external allocentric perspective, causing severe 2D projection artifacts and misplacements that leave net end-to-end gains negative (-0.69%). Point matching requires millimeter-level marker precision on tiny objects, which current tokenizers fail to maintain. Consequently, panoramic visual thinking emerges as the unique intermediate that simultaneously satisfies high informativeness and high generative learnability.

Loss & Training

The network is optimized end-to-end with LoRA (rank 32, alpha 64) on 4×H100 GPUs for 7,000 steps using Adam optimizer with cosine learning rate decay initialized at \(1 \times 10^{-5}\). The overall objective is an equally weighted combination of autoregressive text language modeling cross-entropy \(\mathcal{L}_{\text{CE}}\) and image generation loss \(\mathcal{L}_{\text{gen}}\): $\(\mathcal{L} = \mathcal{L}_{\text{CE}} + \mathcal{L}_{\text{gen}}\)$ The training corpus comprises 7,921 procedurally generated question-answer pairs spanning 1,584 photorealistic synthetic indoor scenes from Infinigen Indoors. The dataset deliberately overweights challenging relation-level tasks (3,000 Relative Distance and 3,000 Relative Direction samples) while allocating fewer samples to Anchor Identification (730) and Counting (1,191), avoiding any real-world training leakage or dependency on external 3D geometric tools.

Key Experimental Results

Main Results

Evaluation spans the in-domain (ID) synthetic benchmark COSMIC across four subtasks (Anchor, Counting, Relative Distance, Relative Direction) alongside five challenging real-world out-of-domain (OOD) benchmarks: MMSI-Bench (1,000 expert-crafted multi-image questions), MindCube-Tiny (1,050 samples), OmniSpatial (Complex Logic and Perspective Taking splits), STARE-Perspective, and BLINK-MultiView:

Model / Strategy Data Scale VDrop COSMIC (ID) Avg. MMSI MindCube OmniSpatial (CL/PT Avg.) STARE-Persp. BLINK-MultiView OOD Avg.
Qwen3-VL-4B - ✗ 42.5 27.4 29.0 34.4 29.2 48.9 33.8
Qwen3-VL-8B - ✗ 46.7 28.0 34.4 35.5 31.6 55.6 37.0
BAGEL (Vanilla) 0 ✗ 26.4 26.9 31.7 35.0 28.0 45.1 33.3
BAGEL-Zebra-CoT 182K ✗ 21.6 23.2 21.7 36.0 28.4 24.8 26.8
ThinkMorph 24K ✗ 38.8 26.5 39.2 39.2 28.8 52.6 37.2
BAGEL + No-Think 8K ✗ 80.6 27.4 41.1 37.1 24.8 45.1 35.1
BAGEL + Text CoT 8K ✗ 50.4 24.6 25.3 32.6 26.0 54.9 32.7
BAGEL + Point Matching 8K ✗ 85.4 27.8 35.2 36.4 24.0 52.6 35.2
BAGEL + Point Matching 8K ✓ 85.3 28.3 34.1 38.5 34.4 45.1 36.1 (+0.9)
BAGEL + Top-down 8K ✗ 84.3 28.8 35.2 35.6 28.0 58.7 37.3
BAGEL + Top-down 8K ✓ 83.1 32.0 36.5 38.4 26.0 57.1 38.0 (+0.7)
BAGEL + Panoramic 8K ✗ 85.2 24.9 36.9 38.4 32.4 55.6 37.6
BAGEL + Panoramic (Ours) 8K ✓ 84.0 26.0 34.1 42.1 35.6 62.4 40.0 (+2.4)

Ablation Study

Systematic ablations on mask geometry, masking scope, drop ratio \(\rho\), curriculum warmup, and causal intervention probes confirm the structural validity of VDrop:

Configuration Category Mask Strategy Masked Scope Drop Ratio \(\rho\) Curriculum Schedule COSMIC (ID) OOD Avg. Finding / Note
SFT Reference None (No VDrop) n/a 0% - 85.2 37.6 Standard SFT; thinking-image remains largely bypassed
Default Model Contiguous Region One View 50% Warmup + Anneal 84.0 40.0 Forces causal dependence on \(I_{vt}\); +2.4 OOD gain
Mask Ratio Sweep Contiguous Region One View 30% Warmup + Anneal 85.2 32.6 Insufficient masking leaves shortcut pathway intact (-7.4)
Mask Ratio Sweep Contiguous Region One View 80% Warmup + Anneal 85.9 36.5 Excessive masking destroys necessary visual context (-3.5)
Mask Scope Sweep Contiguous Region Two Views 50% Warmup + Anneal 83.8 36.8 Deprives anchor coordinate reference (-3.2)
Patch Topology Random Patches One View 50% Warmup + Anneal 83.1 35.6 Local patch interpolation leaks information (-4.4)
Curriculum Ablation Contiguous Region One View 50% No Warmup (Step 0) 81.6 33.7 Early generative instability causes learning collapse (-6.3)

Key Findings

  • Universal Effectiveness Across Representations and Architectures: VDrop consistently boosts OOD cross-view reasoning for all visual thinking types (Panoramic: 37.6% \(\to\) 40.0%, Top-down: 37.3% \(\to\) 38.0%, Point Matching: 35.2% \(\to\) 36.1%). When transferred to SenseNova-U1 (an encoder-free 8B model based on pixel flow-matching), standard SFT actually causes an OOD performance drop (34.2% vs. 35.1% untrained), whereas adding VDrop restores reasoning dependence and achieves 36.7% (+2.5% gain).
  • Small-Scale Visual Thinking Outperforms Massive Text-CoT and Scaled Baselines: Trained on only 8K synthetic traces, BAGEL + Panoramic + VDrop (40.0%) outperforms ThinkMorph (37.2%, 24K traces) and BAGEL-Zebra-CoT (26.8%, 182K traces). Meanwhile, Text CoT achieves only 32.7% (lagging behind direct answering at 35.1%), proving that linguistic reasoning is prone to ambiguity and surface-form inconsistencies when verbalizing continuous 3D scenes.
  • Generate-then-Blind Verification Confirms True Causal Reliance: In an inference probe where the generated thinking-image is blinded to the answer tokens, the standard SFT model exhibits virtually no drop in accuracy (demonstrating that its thinking-image was purely decorative). In contrast, the VDrop-trained model suffers severe accuracy drops under blinding (+12.8% drop on BLINK; +10.1% drop on MMSI Measurement questions), proving that the thinking-image has become a true load-bearing perceptual bridge.
  • Attention Probes Reveal Shift in Early-to-Mid Layers: Tracking answer-token visual attention across transformer layers demonstrates that VDrop raises the thinking-image's attention share from 63.6% to 65.2% overall, with a prominent +3.9 percentage point surge across the first 14 layers, indicating that early spatial grounding depends directly on the synthesized visual intermediate.

Highlights & Insights

  • Turning Decorative Outputs into Functional Intermediates via Structured Attention Dropping: Rather than relying on reinforcement learning or auxiliary discriminators, VDrop utilizes a simple, zero-parameter training-time attention modification to eliminate shortcut learning without altering inference architecture.
  • Formulation of the Learnability–Informativeness Dual-Axis Tradeoff: The paper dismantles the naive assumption that more abstract or global views (like top-down floorplans) are inherently better: if the generative backbone cannot reliably hallucinate an allocentric viewpoint, high theoretical informativeness is neutralized by low learnability.
  • Pure Synthetic Pretraining Enables Strong Real-World OOD Generalization: By training purely on procedurally rendered geometric primitives from Infinigen Indoors, the model acquires robust, domain-invariant spatial mental modeling capabilities that generalize effectively to complex real-world benchmarks like MMSI and BLINK.

Limitations & Future Work

  • Scope Confined to Rigid Cross-View Spatial Geometry: The empirical validation centers on static indoor spatial relationships; extending VDrop and visual thinking to dynamic, non-rigid object transformations or abstract diagrammatic problem-solving remains unexplored.
  • No Direct Optimization of Thinking-Image Generative Fidelity: VDrop forces the model to attend to the thinking-image, but does not inherently guarantee high geometric precision in challenging out-of-distribution visual environments; hallucinations in the intermediate image can still propagate errors to the answer.
  • Future Directions: Integrating test-time self-consistency with adaptive visual imagination, coupling reinforcement learning with answer-correctness verification rewards for intermediate image quality, and formulating hybrid visual-symbolic chains of thought.
  • vs. ThinkMorph / BAGEL-Zebra-CoT: While previous UMM works pioneered interleaved multimodal generation, they relied on unconstrained SFT where the answer pathway largely ignores intermediate images; VDrop eliminates this shortcut, achieving superior OOD accuracy with only 8K samples compared to their 24K–182K datasets.
  • vs. SpatialVLM / STARE (Language-Centric Spatial Modeling): Traditional spatial VLMs force spatial reasoning through text tokens, which lose precise geometric coordinates; visual thinking maintains reasoning in the native continuous pixel domain.
  • vs. Tool-Augmented 3D Systems (Think with 3D / SplatTalk): Unlike pipelines requiring external depth estimators or Gaussian splatting optimizers, this approach maintains an elegant, fully native end-to-end multimodal architecture without non-differentiable external scaffolding.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Formulates the under-use phenomenon in multimodal visual reasoning, presents the simple and effective VDrop mechanism, and introduces the L–I theoretical framework.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Thorough evaluation across two distinct UMM architectures, one ID and five OOD benchmarks, extensive mask parameter ablations, and rigorous causal blinding and attention probes.
  • Writing Quality: ⭐⭐⭐⭐⭐ Lucid, narrative-driven exposition with clear conceptual definitions, disciplined mathematical notation, and compelling empirical analysis.
  • Value: ⭐⭐⭐⭐⭐ Provides critical foundational insights and practical training methodologies for native visual chain-of-thought in next-generation unified foundation models.