MindBlock: Probing Spatial Assembly and Structure in Unified Multimodal Models¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://github.com/yyyybq/MindBlock
Area: 3D Vision / Multimodal VLM
Keywords: Unified Multimodal Models, Spatial Intelligence, 3D Gaussian Splatting, Spatial Assembly, Geometric Equivariance
TL;DR¶
Addressing the limitation of spatial reasoning in unified multimodal models to passive 2D VQA, this paper introduces MindBlock, a benchmark focusing on pixel-space active construction, alongside 3DGS-Eval, a volumetric consistency evaluation protocol using 3D Gaussian Splatting to uncover the 2D-3D consistency gap, spatial drift, and the perception-execution gap.
Background & Motivation¶
Spatial intelligence in the physical world demands far more than passively observing static scenes; its hallmark lies in purposeful construction, physical manipulation, and state deduction. Human children begin systematically building complex block structures from visual diagrams between the ages of four and six, mentally rotating three-dimensional perspectives to evaluate spatial relationships. In contrast, existing spatial reasoning benchmarks for Vision-Language Models (VLMs) and Multimodal Large Language Models (MLLMs) remain largely restricted to passive Visual Question Answering (VQA) or multiple-choice formats. High accuracy on these multiple-choice tasks often stems from statistical shortcuts or superficial linguistic correlations, failing to reflect a coherent internal 3D world model or translate into physical assembly manipulation.
The emergence of Unified Multimodal Models (UMMs) with parameter-shared backbones capable of processing and generating interleaved image-text sequences enables testing spatial reasoning directly via generative execution in pixel space. However, existing evaluation metrics predominantly rely on 2D perceptual similarities such as CLIP-S, PSNR, and SSIM. These 2D heuristics mask critical 3D geometric contradictions: a model may generate multi-view images that look locally plausible and photorealistic in isolation, yet completely contradict each other upon 3D spatial integration, displaying severe geometric flaws such as floating blocks, topological shifts, and inverted occlusions.
To move beyond passive selection and penetrate the deceptive facade of 2D heuristics, this work shifts the paradigm of spatial evaluation from passive answer selection to active visual construction. Core idea: establish MindBlock, an active construction benchmark spanning atomic operations, long-horizon assembly-deconstruction loops, and multi-view geometric transformations, accompanied by 3DGS-Eval, a volumetric consistency validation protocol based on 3D Gaussian Splatting that serves as an internal 3D geometric polygraph.
Method¶
Overall Architecture¶
MindBlock establishes a holistic evaluation and diagnostic framework integrating physically-grounded data synthesis, dual-axis spatial probing, and 3D volumetric consistency verification. The architecture connects a physics simulation layer, a dual-axis spatial task suite, and a 3DGS-Eval geometric validation layer. In data synthesis, a physics engine guarantees equilibrium and causal support hierarchies for each assembly step across full 360-degree viewpoints. In task probing, models are evaluated across "Spatial Assembly" (atomic additions/deletions and long-horizon loops) and "Spatial Structure" (local component rotations and global viewpoint shifts). In evaluation, multi-view generations from the model are fed into independent 3D Gaussian Splatting optimizers to determine whether the implicit scene converges into a geometrically coherent 3D field.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input: Goal configurations / partial observations & instructions"] --> B["Physics-Grounded Data Synthesis<br/>Genesis simulation + 360-degree multi-view rendering + CoT traces"]
B --> C["Dual-Axis Spatial Probing Tasks<br/>Spatial Assembly: single-step / long-horizon loop + Spatial Structure: local / global equivariance"]
C --> D["Unified Multimodal Model Generation<br/>Autoregressive / diffusion interleaved generation of states and reasoning"]
D --> E["3DGS-Eval Volumetric Consistency<br/>Independent 3DGS optimization on disjoint view subsets & cross-validation"]
E --> F["Output: Spatial Diagnostic Metrics<br/>2D perceptual similarity + 3D geometric discrepancy & execution gap"]
Key Designs¶
1. Physics-Grounded Causal Assembly Synthesis: Ensuring Mechanical Stability and Multi-View Ground Truth
Multimodal training data often lacks physical plausibility, leading to ungrounded or floating objects that undermine rigorous causal reasoning. MindBlock builds a simulation pipeline anchored in the Genesis physics engine to construct a stable 3D brick structure library. The data encompasses both semantic structures (furniture, vehicles, animals, buildings that exploit real-world priors) and random structures (neutral configurations requiring pure geometric and topological reasoning), with sequences ranging from simple 2-block builds to complex long-horizon trajectories exceeding 40 blocks. Each incremental step strictly verifies static structural stability to enforce physical dependencies. At each step, 8 egocentric viewpoints covering 360 degrees are rendered alongside template-synthesized Chain-of-Thought (CoT) traces specifying exact \((x, y, z)\) spatial coordinates, block attributes, and hierarchical support relationships (e.g., "Place block A on top of block B").
2. Dual-Axis Spatial Probing Framework: Formalizing Spatial Intelligence as Pixel-Space State Transitions
Traditional benchmarks evaluate spatial cognition through static recognition, failing to assess dynamic transitions and frame-of-reference transformations. MindBlock decouples spatial intelligence into two complementary axes comprising six specific tasks: - Spatial Assembly: Spans atomic operations and long-horizon loops. "Targeted Deletion" requires removing a single block while inpainting occluded backgrounds under unchanged camera perspectives, testing object-centric persistence. "Goal-Directed Completion" requires inferring and rendering the single next critical step toward a target structure. "Forward Assembly" and "Reverse Deconstruction" evaluate long-horizon reasoning by demanding interleaved text-image generation across multiple steps, where deconstruction rigorously tests causal reasoning by forbidding the removal of load-bearing blocks. - Spatial Structure: Tests local and global equivariance. Local manipulation requires rotating a specific sub-component around its local origin while preserving the rest of the assembly. Global consistency requires generating novel views under global camera rotations (e.g., 90 or 180 degrees), testing whether the latent space maintains invariant 3D topology and accurate projective geometry across perspective shifts.
3. 3DGS-Eval Volumetric Consistency Protocol: A Geometric Polygraph Beyond 2D Pixel Metrics
Standard 2D perceptual metrics (e.g., CLIP-S, SSIM) are easily tricked by independently realistic images that lack global 3D coherence. To expose these inconsistencies, 3DGS-Eval employs 3D Gaussian Splatting (3D-GS) to reconstruct volumetric scene fields directly from model-generated multi-view outputs. The 8 generated viewpoints are partitioned into two disjoint subsets, each used to independently optimize a 3DGS representation parameterized by position, opacity, anisotropic covariance, and spherical harmonics color. If the model-generated views represent a unified 3D physical entity, the two reconstructions will smoothly converge to identical volumetric geometry under adaptive density control. If the multi-view outputs contain topological contradictions, the optimization fails to form a stable structure, exhibiting floating artifacts or geometric collapse. Both optimized representations are then re-rendered from shared evaluation poses to compute PSNR and SSIM, mean absolute depth error (Depth Err.), and Chamfer Distance (CD) across sampled Gaussian points, mathematically benchmarking cross-view 3D consistency.
Loss & Training¶
To overcome the limitation that most off-the-shelf UMMs cannot natively generate long-context interleaved text-image sequences, the authors perform continual multi-task training on the BAGEL-zebra-CoT backbone, producing MindBlock-Random and MindBlock-Semantic. The models are trained using a two-stage curriculum jointly supervised on forward assembly and reverse deconstruction operators. To enforce robust viewpoint invariance across iterative steps, each discrete state transition is supervised simultaneously across four camera views, compelling the latent representations to internalize 3D geometry rather than memorizing 2D visual tokens.
Key Experimental Results¶
Main Results¶
Evaluation of leading general-purpose UMMs and image editing models across single-step Spatial Assembly and Spatial Structure tasks demonstrates a clear distinction between 2D perceptual metrics and 3D volumetric metrics.
| Task | Model | 2D Similarity (CLIP-S ↑) | 3DGS Appearance (PSNR ↑) | 3DGS Structure (SSIM ↑) | 3DGS Depth Err. (Depth Err. ↓) | 3DGS Chamfer Dist. (CD ↓) |
|---|---|---|---|---|---|---|
| Spatial Assembly | BAGEL | 0.77 | 13.83 | 0.73 | 75.97 | 11.72 |
| Blip3-o-next | 0.81 | 11.90 | 0.69 | 83.20 | 13.05 | |
| ILLUME | 0.68 | 9.94 | 0.64 | 99.67 | 18.20 | |
| Emu3.5 | 0.75 | 13.10 | 0.71 | 79.20 | 12.10 | |
| Step1X-Edit | 0.86 | 15.20 | 0.78 | 68.50 | 9.10 | |
| Ovis-U1 | 0.76 | 14.10 | 0.75 | 73.10 | 10.40 | |
| Omni-Gen | 0.70 | 8.70 | 0.59 | 108.40 | 17.80 | |
| Spatial Structure | BAGEL | 0.74 | 12.26 | 0.76 | 82.64 | 10.73 |
| Blip3-o-next | 0.79 | 10.80 | 0.70 | 90.50 | 12.90 | |
| ILLUME | 0.68 | 8.90 | 0.64 | 107.33 | 18.58 | |
| Emu3.5 | 0.72 | 11.50 | 0.73 | 86.70 | 11.80 | |
| Step1X-Edit | 0.83 | 13.80 | 0.80 | 74.20 | 9.30 | |
| Ovis-U1 | 0.73 | 12.90 | 0.77 | 79.60 | 10.20 | |
| Omni-Gen | 0.68 | 7.90 | 0.57 | 118.60 | 19.30 |
Ablation Study¶
Three targeted diagnostic experiments evaluate long-horizon stability, the role of textual CoT as a symbolic scaffold, and the divide between discriminative perception and generative execution.
Table 1: Performance Across Multi-Step Forward Assembly and Reverse Deconstruction
| Task Phase | Model Configuration | CLIP-S ↑ | 3DGS-PSNR ↑ | 3DGS-SSIM ↑ | Depth Err. ↓ | Chamfer Dist. (CD) ↓ |
|---|---|---|---|---|---|---|
| Phase I: Forward Assembly | BAGEL-zebra-CoT | 0.78 | 13.5 | 0.74 | 71.3 | 11.8 |
| ThinkMorph | 0.76 | 15.1 | 0.79 | 77.4 | 9.6 | |
| MindBlock-Random (Ours) | 0.84 | 17.8 | 0.86 | 62.5 | 7.4 | |
| MindBlock-Semantic (Ours) | 0.90 | 20.4 | 0.91 | 55.8 | 5.9 | |
| Phase II: Reverse Deconstruction | BAGEL-zebra-CoT | 0.73 | 12.4 | 0.71 | 83.6 | 13.2 |
| ThinkMorph | 0.75 | 14.0 | 0.76 | 76.8 | 10.7 | |
| MindBlock-Random (Ours) | 0.82 | 16.3 | 0.83 | 67.9 | 8.5 | |
| MindBlock-Semantic (Ours) | 0.88 | 19.1 | 0.88 | 60.2 | 6.8 |
Table 2: Diagnostic Analysis on CoT as a Symbolic Scaffold (Single-Step Assembly)
| Generation Paradigm | Model | CLIP-S ↑ | 3DGS-PSNR ↑ | 3DGS-SSIM ↑ | Chamfer Dist. (CD) ↓ | Metric Change |
|---|---|---|---|---|---|---|
| Direct Image Generation | BAGEL (Direct) | 0.782 | 13.42 | 0.712 | 12.4 | Baseline |
| CoT-Augmented Generation | UniCoT | 0.814 | 20.88 | 0.785 | 6.5 | With symbolic coordinates & relations |
| Gain (\(\Delta_{SR}\)) | Improvement | +0.032 | +7.46 | +0.073 | -47.6% | Error nearly halved |
Table 3: The Perception-Execution Gap in BAGEL (4-Way MCQ vs. Pixel-Space Generation)
| Setting | Perception Accuracy (Perception Acc. ↑) | Generative Execution Match (Execution Match ↑) | The Gap (\(\Delta\) ↓) |
|---|---|---|---|
| Single-Step Assembly | 52.5% | 28.2% | 24.3% |
Key Findings¶
- The 2D-3D Consistency Gap: Blip3-o-next achieves high 2D semantic alignment with a CLIP-S score of 0.81 (outperforming BAGEL's 0.77), yet degrades severely under 3DGS-Eval, displaying higher depth error (83.20 vs. 75.97) and Chamfer Distance (13.05 vs. 11.72). Strong 2D perceptual metrics mask deep 3D structural incoherence.
- Geometric Equivariance Bottleneck: Across all evaluated models, performance consistently degrades on Spatial Structure compared to Spatial Assembly (e.g., Step1X-Edit drops from 15.20 to 13.80 PSNR). Current UMMs struggle to preserve invariant spatial topology under viewpoint shifts.
- Spatial Drift in Multi-Step Trajectories: General-purpose models such as ThinkMorph suffer from error accumulation over sequential steps, leading to structural divergence. Reverse deconstruction exhibits higher geometric errors than forward assembly (e.g., BAGEL reverse CD increases to 13.2) due to the demanding nature of causal stability checks.
- CoT Acts as a Symbolic Scaffold: UniCoT, which outputs explicit textual coordinates and support relations prior to image generation, reduces Chamfer Distance by 47.6% (from 12.4 to 6.5) and boosts 3DGS-PSNR by 7.46 dB over direct generation. Symbolic scaffolding effectively anchors latent spatial diffusion.
- The Perception-Execution Gap (24.3%): BAGEL correctly selects the valid spatial transition in 52.5% of multiple-choice queries, but successfully realizes that state in pixel space only 28.2% of the time. Discriminative spatial understanding does not automatically transfer to generative physical execution.
Highlights & Insights¶
- Repurposing 3DGS as a World Model Diagnostic Tool: Leveraging 3D Gaussian Splatting optimization across disjoint view subsets establishes an objective, gradient-based polygraph that uncovers multi-view geometric discrepancies invisible to 2D metrics.
- Formulation of Three Fundamental Spatial Failure Modes: Pinpointing the "2D-3D consistency gap," "spatial drift," and the "perception-execution gap" provides a clear conceptual vocabulary for advancing spatial intelligence in foundation models.
- Demystifying the Role of Chain-of-Thought: Proves that language CoT serves as an active geometric constraint rather than cosmetic commentary, grounding latent generative flows to consistent coordinate frames.
Limitations & Future Work¶
- Scope Restricted to Rigid Block Assemblies: The benchmark currently focuses on discrete rigid brick structures and does not yet address articulated objects, soft bodies, or continuous 3D environments.
- Evaluation Bottleneck for Interleaved Architectures: Most contemporary open-source UMMs support only single-turn visual generation, necessitating specialized fine-tuning to evaluate long-horizon interleaved trajectories.
- Future Directions: Introducing multi-view consistency losses and 3D-aware reinforcement learning rewards directly into UMM training could bridge the perception-execution gap and facilitate integration with robotic manipulation.
Related Work & Insights¶
- vs BrickGPT [33]: BrickGPT models LEGO assembly strictly via textual spatial coordinates within an LLM; MindBlock probes visual-spatial reasoning directly in pixel space with multimodal inputs and outputs.
- vs PhyBlock [29]: PhyBlock evaluates block assembly via multiple-choice VQA, which is susceptible to statistical shortcuts; MindBlock requires generative execution evaluated through volumetric 3D reconstruction.
- vs MVGBench [46]: MVGBench applies 3DGS to evaluate multi-view consistency; MindBlock extends this to dual-subset cross-validation integrated with causal physical assembly and geometric equivariance tasks.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Shifts spatial evaluation from passive question answering to active pixel-space construction, pioneering a 3DGS-based volumetric consistency protocol.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive evaluation across 7 models, single-step and long-horizon tasks, coupled with rigorous ablations on CoT scaffolding and the perception-execution gap.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear exposition, insightful narrative structure, and rigorous integration of empirical data and diagnostic frameworks.
- Value: ⭐⭐⭐⭐⭐ Provides an essential benchmark and diagnostic methodology for physical world modeling in unified multimodal models and embodied robotics.