RGB-Pointmap Pretraining for Unified 3D Scene Understanding¶

Conference: ECCV 2026
Paper: ECCV Paper
Code: https://yebulabula.github.io/UniScene3D/
Area: 3D Vision
Keywords: 3D Scene Understanding, Pointmap Pretraining, RGB-Pointmap, Geometric Alignment, Multimodal Learning

TL;DR¶

UniScene3D proposes a unified 3D scene representation pretraining framework based on multi-view images and world-coordinate pointmaps (RGB-Pointmap), inheriting 2D foundation model priors via patch-level early fusion and enforcing cross-view geometric and grounded view alignments to achieve state-of-the-art performance across viewpoint grounding, scene retrieval, and 3D VQA.

Background & Motivation¶

Pretraining 3D encoders through alignment with pretrained 2D vision-language foundation models like CLIP has emerged as a promising paradigm for generalizable 3D scene understanding. However, the choice of 3D input modality fundamentally dictates the ceiling of learned representations. Explicit point clouds retain precise 3D geometry, but their irregular, unordered nature makes them structurally incompatible with standard grid-based vision architectures, preventing the direct reuse of rich 2D pretraining priors. Conversely, multi-view images and depth maps conform to 2D image grids but lack globally consistent 3D coordinate geometry across different camera viewpoints, severely limiting complex spatial reasoning. Pointmaps have recently emerged as an appealing alternative by encoding 3D world coordinates within an image-like grid; nevertheless, pure pointmaps only encode coordinates and omit color and texture appearance cues, rendering them inadequate when reasoning about visual attributes.

Furthermore, existing multi-view 3D pretraining paradigms typically treat different viewpoints independently. They fail to leverage the inherent cross-view geometric overlap and shared object visibility across multi-view pointmaps, leaving models vulnerable to view-dependent occlusions and partial scene observability. This dual tension between decoupled appearance-geometry modeling and isolated view-by-view processing hinders the development of unified 3D scene representations.

UniScene3D addresses these limitations by jointly modeling appearance and geometry from paired multi-view RGB images and world-frame pointmaps while enforcing explicit cross-view consistency. Core idea: fuse multi-view RGB images and world-coordinate pointmaps at the patch embedding stage to preserve 2D foundation priors, while pretraining with Chamfer-distance-based soft geometric alignment and 3D-mask-based grounded view alignment to build a geometrically consistent and semantically rich unified 3D scene representation.

Method¶

Overall Architecture¶

UniScene3D takes as input $V$ aligned multi-view image-pointmap pairs (RGB-Pointmap). For each view, the pointmap is generated by back-projecting depth maps into a shared world coordinate frame using known camera intrinsics and extrinsics. The framework performs early token fusion at the patch embedding stage and appends a learnable class token before processing through a stack of standard Transformer blocks to output per-view representations, which are mean-pooled to form scene-level embeddings. During pretraining, the model is optimized under four complementary multimodal objectives: cross-view geometric alignment, grounded view alignment, view-level text alignment, and scene-level text alignment.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Multi-view RGB-Pointmap Input<br/>(RGB Images + World-Frame Pointmaps)"] --> EarlyFusion["1. Patch-Level Early Fusion<br/>Element-wise summation preserving 2D priors"]
    EarlyFusion --> Backbone["Transformer Encoder Backbone<br/>Multi-view class token and patch interactions"]
    Backbone --> Lgeo["2. Cross-View Geometric Alignment<br/>Chamfer distance soft-target spatial consistency"]
    Backbone --> Lground["3. Grounded View Alignment<br/>3D mask visibility cross-view text alignment"]
    Backbone --> GlobalAlign["4. Batch-Level Multimodal Alignment<br/>View-level and scene-level global contrastive loss"]
    Lgeo --> TotalLoss["Pretraining Total Loss L_total"]
    Lground --> TotalLoss
    GlobalAlign --> TotalLoss

Key Designs¶

1. Patch-Level Early Fusion: Zero-Parameter Transfer of 2D Foundation Priors While pure pointmaps miss critical visual appearance, late fusion or adding linear projection layers can corrupt pretrained weights or hurt generalization due to cold-start training. UniScene3D initializes from the pretrained 2D image encoder of FG-CLIP. Given each view, the RGB image and world-coordinate pointmap are projected by image patch embedding $\phi_I$ and pointmap patch embedding $\phi_P$ (which shares identical architecture and initialization with $\phi_I$). The resulting patch tokens are merged via element-wise summation: $$z_v = \phi_I(I_v) + E_{pos} + \phi_P(P_v)$$ This design injects 3D geometric coordinates into the visual token stream right at the input stage without introducing uninitialized bottleneck layers, allowing standard Transformer blocks to jointly process appearance and geometry while fully maintaining 2D model generalization.

2. Cross-View Geometric Alignment: Rank-Aware Soft-Target Metric Learning An effective 3D embedding space must reflect physical spatial proximity across different viewpoints: overlapping and spatially adjacent views should reside close to each other in the latent space, while distant views are separated. UniScene3D measures geometric dissimilarity between pointmaps $P_v$ and $P_u$ within the same scene using the symmetric Chamfer distance $CD(P_v, P_u)$ and sorts views to assign ordinal proximity rank indices $r_v(u)$. To prevent penalizing highly overlapping neighbor views as false negatives, it introduces a rank-aware soft target distribution controlled by temperature $\tau_r$: $$p_{v,u}^{\text{soft}} = \frac{\exp(-r_v(u)/\tau_r)}{\sum_{k\in\mathcal{V}_S\setminus\{v\}}\exp(-r_v(k)/\tau_r)}$$ This soft target is linearly interpolated with a hard nearest-neighbor target $p_{v,u}^{\text{hard}}$ via weighting factor $\alpha$: $p_{v,u} = \alpha p_{v,u}^{\text{hard}} + (1-\alpha) p_{v,u}^{\text{soft}}$. Optimizing the soft cross-entropy loss $\mathcal{L}_{\text{geo}}$ against similarity logits enforces smooth, continuous geometric topology in the representation space.

3. Grounded View Alignment: 3D Mask Visibility-Based Semantic Association While geometric alignment ensures spatial consistency, multiple views also share semantic relationships whenever they observe the same 3D objects. To capture shared object semantics, UniScene3D introduces grounded view alignment. For an object $o$ with text embedding $t_o$, the algorithm tests whether the view's 3D pointmap $P_v$ geometrically intersects the ground-truth 3D object mask. All intersecting pairs form valid positive correspondences $(v, o) \in \mathcal{P}$, trained via symmetric multi-positive contrastive loss: $$\mathcal{L}_{\text{ground}} = -\frac{1}{2|\mathcal{P}|} \sum_{(v,o)\in\mathcal{P}} \left[ \log\frac{\exp(s_{v,o})}{\sum_{o'\in\mathcal{O}_S}\exp(s_{v,o'})} + \log\frac{\exp(s_{v,o})}{\sum_{v'\in\mathcal{V}_S}\exp(s_{v',o})} \right]$$ This objective binds fine-grained referring descriptions with all views witnessing that object, pulling together views that share object semantics and drastically boosting viewpoint grounding.

4. Batch-Level Multimodal Alignment: View-Level and Scene-Level Global Discrimination To ensure distinct representation across different scenes across the dataset, UniScene3D complements intra-scene alignment with batch-wide contrastive learning. View-level alignment $\mathcal{L}_{\text{view}}$ matches single-view embeddings $h_v$ with view captions $t_u^V$, while scene-level alignment $\mathcal{L}_{\text{scene}}$ mean-pools per-view embeddings into a global scene vector $\bar{h}_s$ and contrasts it against whole-scene text descriptions $t_j^S$.

Loss & Training¶

The overall pretraining objective is formulated as a weighted combination of all four alignment losses: $$\mathcal{L}_{\text{total}} = \lambda \mathcal{L}_{\text{geo}} + \mathcal{L}_{\text{ground}} + \mathcal{L}_{\text{view}} + \mathcal{L}_{\text{scene}}$$ where $\lambda = 0.1$. Pretraining is conducted on 6,562 indoor scenes from ScanNet, 3RScan, and ARKitScenes using ViT-B/16 at 224×224 resolution. For each scene, 32 views are selected via maximum coverage sampling. The geometric interpolation weight is set to $\alpha = 0.7$ with rank temperature $\tau_r = 0.35$. The model is trained for 80 epochs using AdamW with cosine learning rate decay from $1 \times 10^{-4}$ down to $1 \times 10^{-5}$.

Key Experimental Results¶

Main Results¶

The framework is evaluated across zero-shot viewpoint grounding, zero-shot scene retrieval ($n=5$ and $n=10$ caption utterances), scene type classification, and task-specific fine-tuning for 3D VQA.

Task / Dataset	Input Modality	Metric	UniScene3D (Ours)	Prev. SOTA (POMA-3D / Baseline)	Gain
Viewpoint Grounding (ScanRefer)	RGBP vs PM	R@1 (%)	38.6	16.4 (POMA-3D) / 22.2 (SigLIP2)	+22.2 / +16.4
Viewpoint Grounding (Nr3D)	RGBP vs PM	R@1 (%)	25.2	13.6 (POMA-3D) / 18.8 (SigLIP2)	+11.6 / +6.4
Viewpoint Grounding (Sr3D)	RGBP vs PM	R@1 (%)	23.6	10.2 (POMA-3D) / 13.4 (FG-CLIP)	+13.4 / +10.2
Scene Retrieval (ScanRefer, $n=5$)	RGBP vs PM	R@1 (%)	22.4	13.8 (POMA-3D)	+8.6
Scene Retrieval (ScanRefer, $n=10$)	RGBP vs PM	R@1 (%)	33.4	20.4 (POMA-3D)	+13.0
Scene Classification (ScanNet 21-cls)	RGBP vs PM	0-shot Acc (%)	70.7	63.9 (POMA-3D) / 49.5 (FG-CLIP)	+6.8 / +21.2
Scene Classification (ScanNet 21-cls)	RGBP vs PM	10-shot Acc (%)	83.7	74.1 (POMA-3D) / 77.9 (SigLIP2)	+9.6 / +5.8
3D VQA (ScanQA)	RGBP vs PM	EM@1 (%)	23.2	22.3 (POMA-3D) / 20.9 (FG-CLIP)	+0.9 / +2.3
3D VQA (SQA3D)	RGBP vs PM	EM@1 (%)	52.5	51.1 (POMA-3D) / 49.5 (FG-CLIP)	+1.4 / +3.0
3D VQA (Hypo3D)	RGBP vs PM	EM@1 (%)	35.2	33.4 (POMA-3D)	+1.8

Ablation Study¶

The contribution of each design component (Table 8) and early fusion alternatives (Table 7) is evaluated on viewpoint grounding, scene retrieval, and classification.

Model Variant	ScanRefer R@1	Nr3D R@1	Sr3D R@1	Scene Cls 0-shot	Note / Core Takeaway
UniScene3D (Full Model)	38.6	25.2	23.6	70.7	Optimal representation with all components
w/o image input	37.6	23.7	23.1	67.2	Misses appearance and color texture cues
w/o pointmap input	28.8	21.3	18.5	68.9	Misses 3D coordinates; grounding drops heavily
w/o $\mathcal{L}_{\text{geo}}$	38.2	23.2	23.0	68.7	Lacks cross-view geometric rank regularization
w/o $\mathcal{L}_{\text{ground}}$	18.5	15.6	11.5	68.5	ScanRefer R@1 suffers catastrophic drop (-20.1%)
Fusion Ablations (ScanRefer)	-	-	-	-	-
Full Model (Element-wise Sum)	38.6	-	-	-	Preserves pretrained 2D weight structure
Random Init PM Patch Embed	38.0	-	-	-	Without 2D patch embed initialization
w/o Positional Encoding	37.2	-	-	-	Loses 2D grid patch spatial order
Concat + Projection Layer	36.5	-	-	-	Extra randomly initialized linear layer hurts priors

Key Findings¶

Grounded view alignment $\mathcal{L}_{\text{ground}}$ is critical for spatial grounding: removing $\mathcal{L}_{\text{ground}}$ cuts ScanRefer R@1 by more than half (from 38.6% to 18.5%), proving that intersecting 3D masks to associate referring text with matching views is indispensable for fine-grained viewpoint localization.
Appearance and geometry are strongly complementary: discarding pointmaps causes ScanRefer R@1 to fall by 9.8% (38.6% $\to$ 28.8%), confirming that RGB alone lacks spatial scale; meanwhile, omitting image appearance harms tasks requiring color reasoning (e.g., identifying green chairs).
Element-wise summation outperforms concatenation: concatenating modalities followed by a projection layer drops R@1 to 36.5%, worse even than the pointmap-only variant, because learning an uninitialized projection layer disturbs the pretrained 2D feature manifold.
Robustness to view density: analysis on view numbers shows that UniScene3D with only 16 views matches or exceeds POMA-3D with 32 views on scene retrieval, demonstrating high information efficiency.

Highlights & Insights¶

Seamless 2D-3D architectural synergy: using pointmaps in the world frame formats 3D geometry into 2D image grids, enabling early patch summation that seamlessly unlocks 2D foundation model priors without adding unaligned parameters.
Rank-aware Chamfer soft targets: replaces naive binary view pairing with an ordinal Chamfer-distance soft distribution, ensuring the embedding space continuously preserves smooth geometric proximity.
Transferable multi-view pretraining pattern: establishing cross-view positive associations via 3D mask intersection is an elegant weak supervision mechanism directly applicable to multi-camera autonomous driving and embodied robotics.

Limitations & Future Work¶

Fixed input resolution and model capacity: the model was trained strictly with ViT-B/16 at 224×224; extending UniScene3D to higher resolutions and larger foundation backbones (e.g., ViT-L/G) remains unexplored.
Limited scale of 3D scene datasets: training on 6,562 indoor scenes remains small compared to billions of 2D image pairs; expanding pretraining to large-scale diverse indoor and outdoor environments is an open frontier.
Sensitivity to camera calibration accuracy: constructing world-frame pointmaps relies on clean depth maps and accurate camera poses, meaning severe sensor noise or SLAM drift could degrade performance.

vs POMA-3D: POMA-3D was an early pointmap pretraining pioneer, but it only ingested pure geometric coordinates without appearance and relied on complex JEPA-style multi-stage pretraining. UniScene3D models RGB-Pointmaps jointly and nearly doubles grounding performance.
vs Point cloud models (Uni3D / 3D-VisTA / SceneVerse): Point cloud architectures cannot natively exploit 2D grid vision backbones, leading to severe overfitting and poor zero-shot scene generalization. UniScene3D leverages grid pointmaps and 2D CLIP weights to dominate low-shot tasks.
vs Multi-view 2D methods (DFN / SigLIP2): Pure image models lack physical metric scale, struggling on spatial extent reasoning (e.g., finding "the longest seating area"). UniScene3D overcomes this by injecting explicit 3D world coordinates.

Rating¶

Novelty: ⭐⭐⭐⭐ [Clever early summation fusion of RGB and pointmaps, coupled with Chamfer-based geometric and mask-based grounded view alignment]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive low-shot evaluation on grounding, retrieval, classification, and 3 3D VQA benchmarks with thorough ablations]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, structured explanations, clean mathematics, and well-designed qualitative figures]
Value: ⭐⭐⭐⭐ [Provides an effective blueprint for unified 3D scene foundation models bridging 2D vision and 3D geometry]