Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models

Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/m-bigverdi/IPT
Area: VLM Reasoning
Keywords: spatial reasoning, vision-language models, imaginative perception tokens, unified multimodal models, viewpoint transformation

TL;DR

This paper introduces Imaginative Perception Tokens (IPT), intermediate visual representations generated by a unified multimodal model that predict unobserved spatial configurations (novel views, side views, or top-down BEV maps) to dramatically enhance 3D spatial reasoning in multimodal language models.

Background & Motivation

Spatial reasoning remains a notorious bottleneck for contemporary vision-language models (VLMs). While modern VLMs demonstrate remarkable proficiency in recognizing objects and describing semantic attributes, they consistently fail when confronted with 3D spatial transformations, perspective shifts, severe occlusions, or multi-view synthesis (as benchmarked by ViewSpatial-Bench, 3DSRBench, and MindCube). When humans tackle complex spatial questions—such as predicting what becomes visible after walking around a corner and turning left, or determining the total count of duplicate furniture scattered across multiple rooms—they naturally rely on mental simulation and visual imagination. Humans mentally reconstruct unseen viewpoints and integrate piecemeal visual cues into a coherent, birds-eye spatial map.

Prior attempts to alleviate this limitation typically rely on visible cues. Textual chain-of-thought (Text CoT) forces geometric constraints, coordinate shifts, and occlusion patterns into sequential natural language tokens; this severe modality mismatch introduces spatial hallucinations and cognitive friction. Conversely, recent visual scratchpads (such as Visual Sketchpad) or structured perceptual tokens (depth maps or bounding boxes) merely refine, trace, or extract structures that are already directly visible in the input images. They cannot infer missing spatial structures that are implied by the context but hidden from direct observation.

To address the fundamental challenge where critical spatial evidence is completely absent from the input observation, this work embraces human-like spatial imagination by externalizing unobserved visual configurations before predicting answers. Core idea: supervise unified multimodal models to generate Imaginative Perception Tokens (IPT)—intermediate perceptual representations of unobserved spatial states such as target viewpoints, path side-views, or top-down BEV maps—serving as a grounded cognitive substrate for downstream spatial reasoning.

Method

Overall Architecture

The framework is built upon BAGEL, a unified autoregressive multimodal backbone capable of natively interleaving visual understanding and image generation within a single sequence. Given an input context \(C\) comprising observed images \(I_{\text{obs}} = \{I_1, \dots, I_k\}\) and a spatial query \(Q\), the architecture executes a two-stage generative process. First, the model generates intermediate imaginative perception tokens \(\hat{I}_{\text{imag}}\) representing the unobserved scene structure implied by the task. Second, the generated visual intermediate is decoded, re-encoded, and appended back into the unified context sequence, allowing the model to attend to its own imagination to autoregressively predict the final answer \(A\).

The overall pipeline and component flow are illustrated below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Input Observed Images and Query<br/>I_obs + Q"] --> T1["Three Spatial Imagination Tasks<br/>PET Novel-View / PT Sideview / MVC BEV Map"]
    T1 --> T2["Unified Multimodal Backbone<br/>BAGEL MoT Architecture + Semantic & Latent Tokens"]
    T2 --> T3["Flow-Matching Generation of IPT<br/>Rectified Flow Denoising on Latent Tokens"]
    T3 --> T4["Perception Re-Injection and Reasoning<br/>Decode-Encode Loop + Autoregressive VQA Answer"]
    T4 --> Out["Output Final Spatial Answer<br/>Option Choice A"]

Key Designs

1. Three Spatial Imagination Tasks: Probing Unobserved Spatial Configurations

Standard spatial benchmarks rely primarily on discriminative QA where models exploit superficial text priors. This work formulates three core spatial tasks where success strictly necessitates constructive visual imagination, each paired with ground-truth physically rendered intermediates. Perspective Taking (PET) presents a single first-person indoor image with a ground marker 'X' and queries relative direction or distance changes after moving to 'X' and rotating 90°; its imagination target is a novel-view ground-truth rendering from that target pose. Path Tracing (PT) provides a top-down floorplan path with waypoints 1, 2 and midpoint M, asking which object is visible on a queried side at M; its imagination target is an egocentric sideview ground-truth rendering at M. Multiview Counting (MVC) provides several partial views with heavy occlusion and cross-view duplicate instances; its imagination target is an overhead bird's-eye view (BEV) map that explicitly projects every instance onto a unified coordinate plane. Across AI2-THOR, Habitat, and ScanNet++, over 86k training examples and human-verified evaluation suites are constructed.

2. Unified Multimodal Backbone: Lossless Interleaving of Understanding and Generation

Externalizing visual thoughts and feeding them back into sequential reasoning requires an integrated model rather than decoupled tool-calling pipelines. The approach builds on BAGEL-7B, utilizing a Mixture-of-Transformer-Experts (MoT) architecture. Across all layers, shared self-attention allows seamless information routing between an Understanding Transformer Expert and a Generation Transformer Expert. Image representations are dual-pathed: visual understanding tokens (\(U\)) are extracted via a SigLIP2 ViT encoder to capture semantic abstractions, while visual generation tokens (\(G\)) are latent patches encoded by a FLUX VAE for high-fidelity continuous synthesis. Text, understanding tokens, and latent tokens inhabit a single shared context window, preventing the semantic truncation and latency bottlenecks inherent in external diffusion wrappers.

3. Flow-Matching Generation of IPT: Geometrically Constrained Latent Denoising

Imagination cannot be an unconstrained text-to-image hallucination; it must remain strictly faithful to observed scene geometry and physical layouts. In the latent space, BAGEL adopts a Rectified Flow velocity prediction formulation. Conditioned on context \(C\), the model is trained with a flow-matching loss to transform Gaussian noise into the target ground-truth latent \(G_{\text{gt}}\) representing the unobserved viewpoint:

\[L_{\text{fm}} = \mathbb{E}_{t, G_0, C} \left[ \| v_t(G_t | C) - (G_{\text{gt}} - G_0) \|^2 \right]\]

where \(G_0 \sim \mathcal{N}(0, I)\) and \(G_{\text{gt}}\) is rendered from the 3D environment. This objective forces the generation expert to encode consistent 3D geometric transformations directly into the model's internal representations.

4. Perception Re-Injection and Reasoning: Closed-Loop Downstream Multi-Task Optimization

During inference under imagination mode, the model integrates the predicted velocity field to produce the synthesized latent \(\hat{G}_{\text{imag}}\), decodes it into pixel space \(\hat{I}_{\text{imag}}\), and re-encodes it into both semantic ViT tokens and generative VAE latents, forming an expanded context: \(C^\uparrow = [C, \text{ViT}(\hat{I}_{\text{imag}}), \text{VAE}(\hat{I}_{\text{imag}})]\). The model then computes the autoregressive language modeling loss over the answer sequence:

\[L_{\text{lm}} = - \sum_{i=1}^{|A|} \log P(a_i | C, U_{\text{gt}}, G_{\text{gt}}, a_{<i})\]

The joint multi-task objective is \(\mathcal{L}_{\text{total}} = \omega_{\text{fm}} L_{\text{fm}} + \omega_{\text{lm}} L_{\text{lm}}\). Crucially, this joint optimization allows the model to benefit from imagination supervision even in answer-only inference mode, where intermediate image generation is bypassed while the internal spatial representations remain structurally enhanced.

A Worked Example

Consider a Multiview Counting (MVC) query in a cluttered office with 4 partial egocentric photos: "What is the total number of office chairs in this scene?" The office chairs are partially occluded behind desks and visible across multiple overlapping frames. Upon receiving the 4 input images and query, the unified model triggers the IPT generation phase: the flow-matching generation expert performs continuous denoising in latent space to synthesize a high-resolution 1024×1024 overhead BEV map of the complete office floor. In this synthetic BEV map, duplicate chair instances across different camera angles are unified into unique spatial coordinates, revealing 8 distinct chairs without perspective occlusion. The model re-encodes this BEV map into the sequence, and the language head directly reads the spatial occupancy from the generated map, immediately outputting the correct choice "8" without the double-counting errors typical of text-only reasoning.

Key Experimental Results

Main Results

The main benchmark evaluated in-domain AI2-THOR test splits as well as out-of-domain transfer benchmarks (Habitat for PET, and Matterport3D real-world panoramas for PT), comparing against leading commercial closed-source models, open-source VLMs, unified models, and fine-tuned BAGEL variants.

Model	Type	PET (AI2-THOR)	PT (AI2-THOR)	MVC (AI2-THOR)	PET (Habitat OOD)	PT (Real OOD)
GPT-5	Proprietary VLM	79.8%	60.2%	53.5%	69.3%	80.9%
GPT-5.2	Proprietary VLM	45.5%	32.9%	44.2%	54.0%	63.0%
Gemini 3 Flash	Proprietary VLM	55.0%	42.3%	56.9%	51.3%	83.2%
Qwen3-VL-8B	Open-weight VLM	52.0%	35.9%	43.8%	46.7%	64.1%
Janus-Pro-7B	Unified Model	51.8%	33.5%	33.1%	44.7%	35.3%
Chameleon 7B	Early-fusion Model	34.3%	16.3%	5.4%	47.3%	24.5%
Bagel (base)	Unified Baseline	40.3%	29.9%	35.4%	62.7%	42.7%
Bagel (label-only)	Fine-tuned (Answer)	97.5%	65.7%	63.9%	82.0%	54.7%
+ Text CoT	Fine-tuned (Text CoT)	83.1%	49.7%	62.3%	70.3%	52.2%
+ IPT (Ours)	Fine-tuned (IPT)	96.8%	49.0%	67.3%	87.0%	57.5%
+ Mixed Training	Fine-tuned (IPT + Label)	97.8%	66.7%	62.3%	87.7%	58.6%

Ablation Study

1. Latent Resolution Ablation

Ablation evaluating how imagination latent resolution (from Latent-4 at 64×64 to Latent-64 at 1024×1024) impacts downstream spatial accuracy under thought generation mode.

Latent Size	Pixel Resolution	PET (AI2-THOR)	PET (Habitat OOD)	MVC (AI2-THOR)	Note
Latent-4	64 × 64	87.4%	73.3%	53.5%	Severely blurry, loses fine object boundaries
Latent-16	256 × 256	95.3%	81.0%	56.2%	Coarse geometry resolved, notable accuracy gain
Latent-32	512 × 512	95.0%	87.0%	58.9%	Optimal balance between detail and generalizability
Latent-64	1024 × 1024	96.8%	83.3%	63.1%	Highest in-domain fidelity and count accuracy

2. Thought Modality and Inference Mode Ablation

Comparing thought modalities (Text CoT vs. IPT) and inference modes (explicit thought generation, direct answer-only, and oracle ground truth) on AI2-THOR benchmarks.

Training Signal	Inference Mode	PET Accuracy	PT Accuracy (EgoDir)	MVC Accuracy	Core Insight
Text CoT	w/ text generation	83.1%	53.1%	61.5%	Text serialization degrades geometric precision
Text CoT	answer-only	78.3%	55.8%	62.3%	Lacks structured spatial visual scaffolding
IPT (Ours)	w/ image generation	96.8%	50.4%	67.3%	MVC benefits directly from explicit BEV map de-duplication
IPT (Ours)	answer-only	96.8%	61.1%	62.3%	Strong internalized 3D representations without generation artifacts
IPT (Oracle)	w/ GT image	96.7%	86.7%	67.3%	Massive +36.3% jump on PT highlights generative headroom

Key Findings

• Detrimental modality mismatch in Text CoT: Textual chain-of-thought substantially degrades performance compared to simple label-only supervision (e.g., dropping from 97.5% to 83.1% on PET, and from 65.7% to 49.7% on PT). Serializing 3D camera geometry and topological spaces into text strings squanders model capacity and injects hallucinations.
• Internalized spatial representations via answer-only inference: Models trained with IPT supervision achieve top-tier performance even when evaluated in answer-only mode without executing image denoising at test time (reaching 61.1% on PT EgoDir and 96.8% on PET). Supervising latent generative velocity fields enforces 3D geometric awareness directly within shared attention layers.
• Perception quality is the primary bottleneck for complex paths: When supplied with ground-truth sideview images on Path Tracing, accuracy surges from 50.4% to 86.7% (+36.3%). This proves that downstream reasoning logic is robust, and synthesis fidelity of intermediate novel views governs task headroom.

Highlights & Insights

• Repurposing visual generation from media synthesis to spatial computation: Instead of treating image generation as a consumer-facing text-to-image end product, IPT leverages flow-matching generation as a functional cognitive intermediate for solving unobserved 3D queries.
• Implicit spatial computation dividend: The framework demonstrates that visual thought supervision acts as a powerful geometric regularizer. Deployments can bypass the heavy computational overhead of diffusion sampling at test time while preserving significant spatial gains.
• Broad cross-task spatial transfer: Pre-training with IPT data delivers consistent 7–12% accuracy improvements across out-of-category spatial benchmarks including ScanNet (real-world counting), MindCube (spatial mental modeling), and All-Angles-Bench (cross-view matching).

Limitations & Future Work

• Visual generation artifacts on multi-step trajectories: On long-horizon path tracing, autonomous image generation suffers from slight geometric distortion and object drift, making generated images occasionally less accurate than internalized answer-only inference.
• Heavy dependence on synthetic 3D simulator rendering: Generating paired ground-truth images for unobserved viewpoints requires simulator access (AI2-THOR/Habitat), creating challenges for scaling across uncurated in-the-wild web data.
• Future directions: Integrating test-time compute (TTC) search loops to verify and filter candidate imagined views, and adopting 3D Gaussian Splatting (3DGS) representations as intermediate physical substrates.

Related Work & Insights

• vs Visual Sketchpad / MVoT: Visual Sketchpad operates on input canvases via visual marks and 2D bounding boxes over already visible elements; IPT explicitly synthesizes unobserved perspectives and bird's-eye aggregations that are completely absent from the input.
• vs Textual Chain-of-Thought: Text CoT struggles to serialize spatial relationships into language tokens; IPT honors the visual nature of spatial reasoning by maintaining geometric computations within the visual domain.
• vs External Tool Pipelines: Unlike decoupled pipelines that pipe prompts to external Stable Diffusion models, IPT natively operates inside a unified MoT autoregressive backbone with full gradient propagation and unified cross-attention.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Pioneering the use of native visual generation tokens as functional intermediate representations for unobserved 3D spatial reasoning]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Rigorous evaluations across three core spatial tasks, out-of-domain environments, resolution scaling, and modality ablations]
• Writing Quality: ⭐⭐⭐⭐⭐ [Exemplary clarity in framing the modality mismatch problem and articulating the constructive imagination paradigm]
• Value: ⭐⭐⭐⭐⭐ [Offers a foundational blueprint for equipping next-generation multimodal foundation models with genuine spatial intelligence]

Related Papers

• [AAAI 2026] Graph-of-Mark: Promote Spatial Reasoning in Multimodal Language Models with Graph-Based Visual Prompting
• [ECCV 2026] Spatial Amsan: A Benchmark for Perception-Grounded Spatial Reasoning and Action Evaluation in Egocentric Manipulation
• [ICLR 2026] InternSpatial: A Comprehensive Dataset for Spatial Reasoning in Vision-Language Models
• [CVPR 2026] Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokens
• [ECCV 2026] Towards Spatial Trace with Reasoning in Vision-Language Models for Robotics