GAP-MLLM: Geometry-Aligned Pre-training for Activating 3D Spatial Perception in Multimodal Large Language Models¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://gapmllm.github.io/
Area: Multimodal VLM
Keywords: 3D spatial perception, multimodal large language model, geometry-aligned pre-training, multi-level feature fusion, implicit geometric priors
TL;DR¶
GAP-MLLM introduces the first geometry-aligned pre-training paradigm for pure RGB-based multimodal large language models, activating metric 3D spatial perception under extremely sparse point supervision through visual-prompted joint pointmap-semantic prediction and layer-wise gated fusion with multi-modal DeepStack injection.
Background & Motivation¶
Multimodal Large Language Models (MLLMs) demonstrate exceptional cross-modal semantic reasoning and visual-linguistic capabilities. However, when grounded in the physical world for embodied AI and 3D visual perception, inferring accurate metric spatial structures from monocular RGB video sequences remains a formidable challenge. Due to the prohibitive acquisition and annotation costs of LiDAR and dense 3D point clouds, recent approaches leverage feed-forward 3D reconstruction models (such as VGGT and DUSt3R) to extract pixel-aligned implicit geometric priors. These implicit representations feed into LLMs to empower 3D visual grounding, 3D dense captioning, and 3D video object detection. Nevertheless, implicit prior-based pure RGB methods consistently exhibit a noticeable performance gap compared to approaches operating on explicit 3D point cloud inputs.
A systematic failure analysis reveals that this performance gap stems not from inadequate geometric priors provided by reconstruction models, but rather from a fundamental geometry–semantics misalignment in current training paradigms. Existing methods typically extract only the last-layer representations from geometric encoders, combine them with visual features via naive concatenation or addition, and fine-tune directly on downstream tasks supervised predominantly by natural language text. Under the strong gradient domination of next-token text prediction, unaligned implicit geometric representations are easily overwhelmed or marginalized. Moreover, geometric encoders exhibit distinct layer-wise attention patterns—shallow layers capture local structural details whereas deeper layers construct global correspondence. Merging only the final-layer features overlooks this hierarchical geometric distribution.
To eliminate these bottlenecks, the structural perception capabilities of MLLMs must be explicitly awakened before downstream task adaptation. Core idea: propose a geometry-aligned pre-training paradigm (GAP-MLLM) for RGB-only MLLMs that explicitly activates 3D metric spatial perception via visual-prompted joint sparse pointmap and semantic label prediction, and enables hierarchical integration of geometric priors through layer-wise gated fusion and multi-modal DeepStack injection.
Method¶
Overall Architecture¶
The GAP-MLLM architecture is designed to infer metric 3D scene structures and semantic contents end-to-end from monocular video sequences and natural language queries. The framework consists of parallel visual (Qwen3-VL visual encoder) and geometric (VGGT feed-forward reconstruction encoder) branches, a multi-level gated feature fusion module, and a video LLM decoder.
Given an input video sequence of \(n\) consecutive frames \(\{I_i\}_{i=1}^n\) and a textual query \(Q\), the visual and geometric encoders extract layer-wise feature tokens across all \(L\) layers (\(L=24\)). To ensure cross-modal spatial alignment, both branches group spatially adjacent \(2\times 2\) patches through a two-layer MLP for token merging. The merged tokens at each layer are fed into independent token-level gating networks to adaptively balance geometric and semantic contributions. When feeding into the LLM decoder, the final-layer (\(L\)-th) fused tokens serve as primary prefix embeddings concatenated with text embeddings, while intermediate-layer fused tokens (layers 5, 11, and 17) are directly added to the hidden states of early decoder blocks via a multi-modal DeepStack strategy, preventing structural signals from fading during causal self-attention decoding.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Image Sequence & Text Query"] --> B["Dual-Branch Parallel Encoding & Patch Merging<br/>Visual Branch + VGGT Geometry Branch"]
B --> C["Layer-Wise Gated Feature Fusion<br/>Token-Level Dynamic Geometry-Semantic Weighting"]
C --> D["Multi-Modal DeepStack Hierarchical Injection<br/>Intermediate Injections to Hidden States + Top Layer to Main Input"]
E --> F["Two-Stage Downstream Task Inference<br/>Keyframe Index Localization & Metric 3D Box Regression"]
Key Designs¶
1. Sparse Geometry-Semantics Joint Pre-training: Activating Metric Spatial Perception under a Unified Coordinate System
Inferring 3D structures from monocular RGB sequences suffers from scale ambiguity and inconsistent coordinate definitions across tasks. GAP-MLLM establishes a unified first-frame metric coordinate system, where all spatial coordinates are measured in meters and rounded to two decimal places. To awaken spatial perception before language-dominated fine-tuning, the framework introduces a visual-prompted joint pre-training task: given a four-frame image sequence with a red cross marking a specific pixel, the model is required to simultaneously predict the exact 3D metric coordinate \((x, y, z)\) and the semantic category label \(c\) of that prompted point in the first-frame coordinate system.
This design achieves remarkable training efficiency. While sampled across approximately 500K point instances from EmbodiedScan and ScanNet, the effective pixel-level supervision equates to roughly two \(680\times 480\) images. Predicting 3D coordinates compels the model to internalize the geometric projection from pixels to physical 3D space, while simultaneous semantic classification ensures that the pre-existing visual-linguistic feature manifold remains stable without degradation. The two objectives mutually regularize each other, activating robust metric geometric consciousness under sparse supervision.
2. Layer-Wise Gated Feature Fusion: Dynamic Multi-Scale Balancing across Depths
Attention map visualization reveals that geometric encoder tokens exhibit stark layer-wise distinction: intermediate layers capture broad structural depth and topology, while final layers encode task-specific abstractions. Prior methods relying on single-layer addition inevitably produce distorted spatial priors. To address this, GAP-MLLM incorporates an independent two-layer MLP gating network for each layer \(j \in \{1, 2, \dots, L\}\) after token merging:
Here, \(\sigma\) denotes the Sigmoid function, \([\cdot, \cdot]\) indicates concatenation, and \(\odot\) denotes element-wise multiplication. The gating weight \(g_{i,j} \in [0, 1]\) adaptively balances semantics and geometry across both network depth and spatial token locations. Empirical visualizations demonstrate that intermediate layers allocate significantly higher weights to geometric priors (reaching 60%–70%), whereas later layers shift focus toward visual semantic features, effectively eliminating cross-modal suppression.
3. Multi-Modal DeepStack Injection and Two-Stage Downstream Decoupling: Sustained Geometry and Structured Reasoning
Even with layer-wise gated representations, feeding tokens solely into the LLM input layer risks gradual information dilution across deep Transformer blocks. Inspired by DeepStack, fused intermediate tokens \(\mathcal{T}_{i,m}^S\) at layers \(m \in \{5, 11, 17\}\) are directly injected into early decoder hidden states via residual addition, sustaining geometric cues throughout text decoding.
For downstream adaptation, predicting complex 3D bounding boxes in long video sequences in a single pass is prone to temporal error accumulation. GAP-MLLM introduces a decoupled two-stage inference procedure: for 3D visual grounding, Stage 1 predicts the target frame index \(I_t\) where the object prominently appears; Stage 2 reorganizes the sequence with \(I_t\) as the primary reference frame and regresses the 9-DoF 3D bounding box \((x, y, z, w, h, d, \psi, \theta, \phi)\). Decoupling temporal selection from bounding box regression drastically reduces decoding complexity and ensures metric consistency.
Loss & Training¶
The framework is optimized in two stages: sparse geometry-semantics joint pre-training followed by mixed-task object-level fine-tuning. Throughout both stages, the visual encoder and VGGT geometric encoder are kept frozen to maintain stable representations, while the multi-level gated fusion module and the LLM backbone are updated. Optimization is conducted using Adam with a peak learning rate of \(1\times 10^{-5}\), a linear warm-up ratio of 0.03, and 1 training epoch for each stage. The batch size is 32 for joint pre-training and 16 for downstream task fine-tuning.
Key Experimental Results¶
Main Results¶
GAP-MLLM is evaluated across 3D visual grounding (ScanRefer), 3D dense captioning (Scan2Cap), and 3D video object detection (EmbodiedScan), comparing against leading RGB-only baselines and explicit 3D-input methods.
Table 1: Comparison on ScanRefer and Scan2Cap
| Dataset / Task | Model | 3D Input | Primary Metric 1 | Primary Metric 2 | Primary Metric 3 |
|---|---|---|---|---|---|
| ScanRefer (Grounding) | Video-3D LLM [59] | Yes (Point Cloud) | [email protected]: 58.1 | [email protected]: 51.7 | - |
| ScanRefer (Grounding) | SPAR [57] | No (RGB) | [email protected]: 31.9 | [email protected]: 12.4 | - |
| ScanRefer (Grounding) | VG-LLM-4B [58] | No (RGB) | [email protected]: 36.4 | [email protected]: 11.8 | - |
| ScanRefer (Grounding) | VG-LLM-4B (w/ GAP) | No (RGB) | [email protected]: 49.7 | [email protected]: 23.2 | - |
| ScanRefer (Grounding) | GAP-MLLM-3B (Ours) | No (RGB) | [email protected]: 53.1 | [email protected]: 26.0 | - |
| Scan2Cap (Captioning) | Video-3D LLM [59] | Yes (Point Cloud) | [email protected]: 80.0 | [email protected]: 40.2 | [email protected]: 61.7 |
| Scan2Cap (Captioning) | VG-LLM-4B [58] | No (RGB) | [email protected]: 78.6 | [email protected]: 40.9 | [email protected]: 62.4 |
| Scan2Cap (Captioning) | VG-LLM (w/ GAP) | No (RGB) | [email protected]: 80.6 | [email protected]: 41.1 | [email protected]: 62.8 |
| Scan2Cap (Captioning) | GAP-MLLM-3B (Ours) | No (RGB) | [email protected]: 84.7 | [email protected]: 42.1 | [email protected]: 63.1 |
Table 2: 3D Video Object Detection Results on EmbodiedScan (IoU 0.25)
| Model | 3D Input | 4-Frame P25 | 4-Frame R25 | 4-Frame F1 | 6-Frame P25 | 6-Frame R25 | 6-Frame F1 |
|---|---|---|---|---|---|---|---|
| SpatialLM [35] | Yes (Explicit 3D) | 40.8 | 23.7 | 29.1 | 42.1 | 26.1 | 31.1 |
| VG-LLM-4B [58] | No (RGB) | 41.7 | 35.7 | 38.2 | 39.7 | 34.0 | 36.4 |
| VG-LLM-8B [58] | No (RGB) | 43.4 | 39.6 | 41.2 | 43.5 | 38.7 | 40.8 |
| VG-LLM-4B (w/ GAP) | No (RGB) | 47.0 | 41.2 | 43.5 | 48.8 | 40.2 | 43.6 |
| GAP-MLLM-3B (Ours) | No (RGB) | 54.2 | 48.1 | 50.6 | 52.9 | 45.4 | 48.5 |
Ablation Study¶
Table 3: Ablation on Core Components in 3D Video Object Detection (4-Frame Setting)
| Model Configuration | Sparse Joint Pre-training (A) | Multi-Level Gated Fusion (B) | Precision (%) | Recall (%) | F1 (%) | Note |
|---|---|---|---|---|---|---|
| VG-LLM-4B Baseline | ✗ | ✗ | 41.7 | 35.7 | 38.2 | Naive fusion with direct fine-tuning |
| VG-LLM-4B + A | ✓ | ✗ | 44.1 | 39.6 | 41.4 | Pre-training alone (+3.2 F1) |
| VG-LLM-4B + B | ✗ | ✓ | 43.3 | 36.9 | 39.4 | Gated fusion alone (+1.2 F1) |
| VG-LLM-4B + A + B | ✓ | ✓ | 47.0 | 41.2 | 43.5 | Combined setup (+5.3 F1) |
| GAP-MLLM-3B Baseline | ✗ | ✗ | 48.8 | 42.1 | 44.7 | Default Qwen3-VL baseline |
| GAP-MLLM-3B + A | ✓ | ✗ | 52.7 | 45.9 | 48.7 | Pre-training alone (+4.0 F1) |
| GAP-MLLM-3B + B | ✗ | ✓ | 51.8 | 44.6 | 47.5 | Gated fusion alone (+2.8 F1) |
| GAP-MLLM-3B (Full) | ✓ | ✓ | 54.2 | 48.1 | 50.6 | Optimal combination (+5.9 F1) |
Table 4: Ablation on Feature Fusion Strategies (GAP-MLLM-3B)
| Fusion Strategy | Precision (%) | Recall (%) | F1 (%) |
|---|---|---|---|
| Element-wise Addition (Add) | 51.7 | 45.3 | 47.9 |
| Static Weighted Addition (Weighted) | 51.4 | 45.5 | 47.8 |
| Cross-Attention Mechanism | 52.7 | 46.3 | 49.1 |
| Layer-Wise Gated Fusion (Ours) | 54.2 | 48.1 | 50.6 |
Key Findings¶
- Sparse pre-training acts as the primary driver of performance: On a pure Qwen3-VL model without any geometric encoder, sparse joint pre-training alone improves detection F1 from 39.7 to 42.4 (+2.7 points). With the VGGT encoder added, pre-training boosts GAP-MLLM-3B F1 by an additional 4.0 points, demonstrating that geometric activation is essential.
- Joint geometric and semantic supervision exhibits mutual reinforcement: When evaluating ScanNet metric reconstruction, removing semantic classification causes metric evaluation error to deteriorate from 0.0616m to 0.0630m, and detection F1 drops from 41.4 to 40.1, verifying that high-level semantics regularize metric spatial coordinates.
- Hierarchical gating distribution matches intuitive perception: Gating visualizations reveal that intermediate layers route 60%–70% of feature weight to geometric tokens, whereas the output layer shifts focus back to semantic representations.
Highlights & Insights¶
- Sparse supervision achieves remarkable spatial activation: Annotating a single prompted 3D point coordinate and semantic label per sample (totaling effective supervision of merely two standard images) successfully activates metric 3D reasoning in MLLMs, presenting a highly data-efficient paradigm.
- Plug-and-play cross-architecture versatility: When applied directly to VG-LLM-4B without changing the underlying backbone, GAP pre-training lifts ScanRefer [email protected] from 36.4% to 49.7%, confirming broad generalizability.
- Two-stage decoupling simplifies spatial regression: Disentangling temporal keyframe identification from local 3D bounding box regression effectively circumvents multi-frame cumulative drift in unconstrained camera coordinate systems.
Limitations & Future Work¶
- Reliance on the first-frame reference frame: Representing all 3D bounding boxes relative to the initial frame's camera coordinate system risks geometric degradation in long video sequences subject to large camera displacements.
- Coarse semantic boundaries from sparse supervision: Because pre-training supervision is applied only to sparse prompt points, dense semantic segmentations generated by the model exhibit slight boundary smoothing.
- Future directions: Integrating pose-graph SLAM optimization for long-horizon spatial reasoning, and coupling autoregressive pointmap completion with higher-resolution scene understanding.
Related Work & Insights¶
- vs VG-LLM [58]: VG-LLM relies on single-layer addition and direct downstream fine-tuning where geometry is suppressed by text supervision. GAP-MLLM introduces multi-level gated fusion and explicit sparse pre-training, surpassing VG-LLM by over 12 F1 points on 3D detection.
- vs SpatialLM [35] & Video-3D LLM [59]: While these methods mandate dense, expensive explicit 3D point cloud inputs, GAP-MLLM proves that implicit priors from feed-forward reconstruction under monocular RGB sequences can match or exceed explicit 3D models when properly activated.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Visual-prompted sparse joint pre-training provides an elegant, low-cost solution to 3D spatial activation.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarks across grounding, dense captioning, detection, and metric reconstruction.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous methodology with lucid motivation and comprehensive ablation studies.
- Value: ⭐⭐⭐⭐⭐ Establishes a highly efficient and scalable pre-training blueprint for RGB-based 3D multimodal foundation models.