DINOv3D: 2D-3D Joint Optimization for Unified Spatial Understanding

Conference: ECCV 2026
Paper: ECCV Official Poster
Code: https://github.com/bobochow/DINOv3D
Area: 3D Vision
Keywords: 3D Gaussian Splatting, Vision Foundation Models, 2D-3D Joint Optimization, Representation Learning, Unified Spatial Understanding

TL;DR

DINOv3D presents a homologous teacher-student joint optimization framework based on DINOv3, where a frozen teacher anchors native 2D visual priors into a 3D Gaussian regularization field while a LoRA-adapted student ingests long-context 3D geometric consistency via shifted-window dual-memory attention and ray-depth-semantic alignment, establishing state-of-the-art results across 3D scene understanding benchmarks with only 10% trainable parameters and reciprocally enhancing 2D spatial features.

Background & Motivation

Adapting powerful 2D vision foundation models (VFMs, such as DINOv3 and SAM) to 3D spatial understanding represents a foundational path toward spatial intelligence and embodied AI. However, current adaptation paradigms face a critical dilemma: unconstrained end-to-end fine-tuning of 2D foundation backbones on 3D geometric tasks inevitably causes catastrophic forgetting of native 2D semantic representations learned from billions of images. Conversely, freezing the 2D backbone and simply attaching task-specific 3D projection heads severely restricts 3D geometric awareness, resulting in multi-view inconsistencies, geometric conflicts, and severe semantic blurring under dramatic viewpoint changes or occlusions.

The core tension stems from the optimization imbalance between geometric structural regularization and 2D prior preservation. While standard teacher-student distillation offers a potential remedy against catastrophic forgetting, conventional formulations typically operate on short-context sequences (e.g., 2–16 frames) with noisy geometric pseudo-labels. Under heavy occlusions or sparse viewpoints, these noisy constraints inject erroneous distillation gradients that corrupt the student model. Achieving stable regularized adaptation demands the capability to process long-context multi-view inputs (up to 128 frames) where spatial-temporal physical consistency serves as an intrinsic filter against 2D perceptual noise.

This paper's angle of attack is to adopt explicit 3D Gaussian Splatting (3DGS) as a continuous differentiable bridge between 2D continuous feature spaces and 3D physical structures, while designing an efficient local-window memory mechanism to capture long-context multi-view consistency. Core idea: establish a homologous teacher-student dual-branch framework based on DINOv3, where a frozen teacher distills native 2D visual priors into a 3D Gaussian regularization field as an anti-forgetting anchor, while a LoRA-tuned student model processes long-context sequences via dual-memory shifted window attention and ray-depth-semantic alignment to collaboratively absorb 3D geometry and reinforce 2D spatial awareness.

Method

Overall Architecture

The input to DINOv3D is a sequence of multi-view images \(I = \{I_k \in \mathbb{R}^{H \times W \times 3}\}_{k=1}^K\). The framework employs a homologous teacher-student collaborative architecture: 1. Teacher Branch: The DINOv3 backbone \(E_\psi\) is completely frozen to extract reference 2D visual feature maps \(F_{gt}\), guiding a unified DPT decoder to predict a 3D Gaussian regularization field with features \(f_{reg}\). Differentiable Gaussian rasterization renders a 2D regularization feature map \(F_{reg}\) that is explicitly aligned with \(F_{gt}\), establishing a rigid 2D prior anchor. 2. Student Branch: The student backbone is adapted using parameter-efficient LoRA fine-tuning. It incorporates a Shifted Window Attention with Dual Memory mechanism to process up to 128 frames with linear computational complexity, capturing long-range cross-view spatial-temporal dependencies. 3. Unified Decoding and Ray-Depth-Semantic Alignment: A centralized DPT decoder fuses multi-scale features \(F_{shared}\) to jointly predict a depth map \(D\), a dense ray map \(R\), and a multi-attribute 3D Gaussian field \(G\) (containing appearance, semantic features \(f_{sem}\), instance features \(f_{ins}\), and regularization features \(f_{reg}\)). The ray-depth-semantic alignment module utilizes the predicted 3D rays and metric depth to establish cross-view geometric correspondence, enforcing multi-view consistency across rendered semantic representations under depth-confidence filtering.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-view Image Sequence<br/>128-frame Long Context"] --> B["Dual-Memory Shifted Window Attention<br/>Global/Local Memory + Shifted Windows"]
    B --> C["Homologous Teacher-Student Architecture<br/>Frozen Teacher Anchors 2D + LoRA Student Absorbs 3D"]
    C --> D["Unified Geometric-Semantic DPT Decoder<br/>Multi-scale Feature Fusion into Fshared"]
    D --> E["3D Gaussian Field Prediction<br/>Position/Rotation/Scale/Opacity + fsem/fins/freg"]
    E --> F["Ray-Depth-Semantic Cross-View Alignment<br/>Ray Unprojection + Depth Confidence Masking"]
    F --> G["Multi-task Outputs<br/>Novel View Synthesis / Depth Estimation / Open-Vocabulary Segmentation"]

Key Designs

1. Homologous Teacher-Student Architecture and Regularization 3D Gaussian Field: Anchoring Native 2D Priors Against Forgetting
Direct fine-tuning on 3D geometry tasks inevitably degrades the general 2D perceptual capabilities of foundation backbones. To lock in native 2D visual knowledge while learning spatial geometry, DINOv3D introduces a homologous teacher-student dual-branch framework, where both branches originate from the same pretrained DINOv3 model. In the teacher branch, the backbone \(E_\psi\) remains completely frozen, and only the DPT decoder and regularization head \(\mathcal{H}_{reg}\) are trained to predict the regularization attributes \(f_{reg}\) attached to each 3D Gaussian primitive. These primitives are rendered via differentiable Gaussian rasterization into a 2D regularization feature map \(F_{reg}\): $\(F_{reg} = \sum_{i=1}^{N_g} f_{reg, i} \, \alpha_i \prod_{j=1}^{i-1} (1 - \alpha_j)\)$ By minimizing the feature-metric alignment loss \(\mathcal{L}_{reg}\) against the frozen teacher's native representations \(F_{gt} = E_\psi(I)\) over valid pixels, the 3D Gaussian field is grounded in the foundational 2D feature manifold. Crucially, during the subsequent student branch optimization, the regularization head \(\mathcal{H}_{reg}\) is strictly frozen to define an invariant projection subspace, preventing feature space drift and catastrophic forgetting as the student backbone updates via 3D geometry gradients.

2. Shifted Window Attention with Dual Memory: Linear Complexity for Long-Context Spatial-Temporal Resonance
Standard multi-view models struggle with short contexts (2–16 frames), lacking global structural context, while global self-attention across 128 frames incurs prohibitive quadratic complexity. DINOv3D introduces a Shifted Window Attention with Dual Memory mechanism. It integrates learnable scene-level global memory tokens \(T_{global}\), window-level local memory tokens \(T_{local}\), and the original image class token \(T_{cls}\). The transformer layers are divided into \(L_s\) intra-frame self-attention layers and \(L_g\) cross-view window interaction layers. In cross-view layers, the sequence is partitioned into local windows of size \(W\). Successive layers shift the partition boundaries by \(\lfloor W/2 \rfloor\) views. The global memory token \(T_{global}\) serves as an information conduit propagating semantic context across all views, while \(T_{local}\) models fine-grained local variations within individual windows, strictly preserving linear computational complexity.

3. Ray-Depth-Semantic Cross-View Alignment: Geometry-Guided Rectification of Multi-View Semantic Inconsistencies
Lifting 2D foundation features directly into 3D space causes severe semantic blurring because 2D foundation models exhibit view-dependent fluctuations. DINOv3D resolves this ambiguity using explicit 3D geometry. The unified DPT decoder predicts a dense ray map \(R\) (origins \(o_u\) and directions \(d_u\)) alongside metric depth \(D\). Together, they establish exact geometric pixel correspondence between source pixel \(u_s\) and target pixel \(u_{s \to t}\): $\(u_{s \to t} = \mathcal{P}_t \left( o_{u_s} + D(u_s) \cdot d_{u_s} \right)\)$ Because early geometric predictions can be noisy, unconstrained alignment risks injecting corruption into the backbone. The framework applies a depth-confidence metric \(M_{u_s}\) to modulate the semantic alignment loss \(\mathcal{L}_{align}\): $\(\mathcal{L}_{align} = \frac{1}{|Q|} \sum_{(u_s, u_t) \in Q} M_{u_s} \left( 1 - \frac{F_{sem}(u_s)^\top F_{sem}(u_t)}{\|F_{sem}(u_s)\|_2 \|F_{sem}(u_t)\|_2} \right)\)$ This explicitly forces semantic boundaries to coincide with physical depth discontinuities, leveraging rigid 3D spatial consistency to filter out 2D semantic noise.

Loss & Training

DINOv3D is trained end-to-end via a composite objective: $\(\mathcal{L}_{total} = \mathcal{L}_{reg} + \mathcal{L}_{rgb} + \mathcal{L}_{depth} + \mathcal{L}_{ray} + \mathcal{L}_{sem} + \mathcal{L}_{ins} + \mathcal{L}_{align}\)$ - \(\mathcal{L}_{rgb}\): Combines L1 loss and perceptual LPIPS loss between rendered views \(\hat{I}\) and ground-truth images \(I\). - \(\mathcal{L}_{depth}\): Confidence-weighted depth L1 loss coupled with spatial finite-difference gradient regularization. - \(\mathcal{L}_{ray}\): Supervises ray origin and direction vectors, penalizing the reprojection error between reconstructed 3D points and ground-truth point maps \(X_{gt}\). - \(\mathcal{L}_{sem}\) & \(\mathcal{L}_{ins}\): The semantic branch aligns rendered features \(F_{sem}\) with frozen LSeg representations \(F_{lseg}\) via cosine similarity; the instance branch uses SAM segmentation masks for contrastive margin supervision. - Training is executed on 8 NVIDIA H20 GPUs with DINOv3 ViT-L as the backbone. The student backbone is adapted via LoRA (\(r=16\), only 0.1B trainable parameters). Input resolution is \(256 \times 256\), window size \(W=16\), context length 128 frames, optimized with AdamW (learning rate \(1 \times 10^{-4}\), weight decay 0.05, cosine annealing schedule) over 100 epochs.

Key Experimental Results

Main Results

Quantitative evaluation on the ScanNet indoor benchmark across Novel View Synthesis (NVS), Open-Vocabulary Semantic Segmentation (OVSS), and Depth Estimation (DE) across varying input views is summarized below (from Table 1 of the original paper):

Input Views	Method	Total / Trainable Params	NVS PSNR (dB) ↑	NVS SSIM ↑	OVSS mIoU (%) ↑	OVSS Acc (%) ↑	DE Abs Rel ↓	DE \(\tau_{<1.25}\) (%) ↑
2 views	Feature3DGS [73]	None	24.49	0.8132	42.23	71.74	12.95	21.07
	LSM [15]	0.6B / 0.6B	24.39	0.8072	50.78	76.86	3.38	67.77
	Uni3R [53]	1.3B / 1.0B	25.53	0.8727	55.84	82.68	3.87	61.37
	DINOv3D (Ours)	0.4B / 0.1B	25.67	0.8823	57.68	85.12	3.02	70.52
8 views	Feature3DGS [73]	None	18.17	0.6740	19.50	72.40	17.28	13.31
	Uni3R [53]	1.3B / 1.0B	24.12	0.8290	55.40	80.70	4.46	56.88
	DINOv3D (Ours)	0.4B / 0.1B	25.43	0.8420	56.30	83.25	3.38	66.74
16 views	Feature3DGS [73]	None	17.09	0.6490	19.80	67.20	23.71	10.57
	Uni3R [53]	1.3B / 1.0B	23.24	0.8081	52.20	79.50	5.88	42.88
	DINOv3D (Ours)	0.4B / 0.1B	24.68	0.8230	54.10	82.86	3.59	64.28
32 views	Feature3DGS [73]	None	16.53	0.6127	17.60	62.10	28.95	7.89
	Uni3R [53]	1.3B / 1.0B	22.54	0.7863	50.50	77.20	6.21	37.52
	DINOv3D (Ours)	0.4B / 0.1B	24.32	0.8020	52.90	81.31	3.86	60.15

Ablation Study

Incremental ablation of the proposed core modules on ScanNet (3D tasks) and ADE20k (2D semantic segmentation) is presented below (from Table 6 of the original paper):

Configuration	Dual Branch	Semantic-Geometric Align.	Long Context (128 frames)	ScanNet PSNR ↑	ScanNet rel ↓	ScanNet mIoU ↑	ADE20k mIoU ↑	Note
Baseline (Frozen)	-	-	-	23.45	3.85	51.2	54.9	Frozen DINOv3 backbone
+ Dual Br.	✓	-	-	24.62	3.42	54.5	55.0	Teacher-student regularization prevents forgetting
+ Align.	✓	✓	-	25.41	3.15	55.6	55.2	Ray-depth-semantic multi-view alignment
+ Long Context	✓	✓	✓	25.58	3.11	56.5	55.3	Shifted-window dual memory over 128 frames
Full Model (+ Uni3D)	✓	✓	✓	25.67	3.02	57.7	55.4	Multi-source consistency-filtered Uni3D dataset

Key Findings

• High Parameter Efficiency Surpassing Billion-Scale Models: With only 0.1B trainable parameters via LoRA, DINOv3D achieves a PSNR of 25.67 dB under the sparse 2-view setting on ScanNet, outperforming Uni3R (25.53 dB with 1.0B trainable parameters), while cutting depth error from 3.87 to 3.02 and raising segmentation mIoU from 55.84% to 57.68%.
• Robust Long-Context Scalability Without View-Inconsistency Degradation: When scaling from 2 to 32 views, prior per-scene or unregularized methods (such as Feature3DGS) suffer severe performance collapse (PSNR dropping from 24.49 to 16.53 dB; mIoU dropping from 42.23% to 17.60%). DINOv3D maintains high fidelity (24.32 dB PSNR and 52.90% mIoU at 32 views) thanks to shifted-window temporal consistency.
• Bi-directional Reciprocity: 3D Geometry Regularizes 2D Perception: As reported in Table 5 of the original paper, DINOv3D outperforms the original DINOv3 baseline across 2D linear probing tasks: ImageNet-R classification accuracy improves from 88.1% to 89.4% (+1.3%), and NYUv2 depth estimation RMSE decreases from 0.352 to 0.324, demonstrating that multi-view 3D consistency acts as a superior regularizer for 2D representations.

Highlights & Insights

• 3D Gaussian Fields as Physical Invariant Regularization Anchors: Instead of using 3DGS purely for image rasterization, DINOv3D establishes an immutable 2D-to-3D projection subspace in the teacher branch, turning the continuous spatial Gaussian field into a physical anchor that preserves 2D foundational knowledge.
• Dual Memory Decoupling for Long-Range Sequence Modeling: The combination of a scene-level global token \(T_{global}\) and window-level local tokens \(T_{local}\) paired with \(\lfloor W/2 \rfloor\) shifting provides long-range receptive fields across 128 frames at strictly linear computational cost.
• Reciprocal 2D-3D Foundation Adaptation: Rather than treating 2D-to-3D transfer as a one-way distillation pipeline, DINOv3D demonstrates that parameter-efficient 3D training with explicit ray-depth constraints simultaneously injects metric geometric awareness into 2D foundation representations.

Limitations & Future Work

• Limitations Acknowledged by Authors: The ray-depth-semantic formulation assumes rigid scenes; dynamic moving objects or non-rigid structural deformations can violate the geometric unprojection assumptions, leading to erroneous alignment pairs.
• Identified Limitations: In unconstrained outdoor scenes with vast specular reflections, transparent materials, or featureless surfaces, depth estimation confidence drops significantly, causing the confidence-filtered alignment mask to become overly sparse.
• Future Directions: Extending the teacher-student Gaussian field to dynamic 4D Gaussians, and exploring real-time closed-loop control integration for embodied robotics and spatial navigation.

Related Work & Insights

• vs Uni3R [53] / LSM [15]: Uni3R and LSM train full models with billions of parameters over short view windows, resulting in severe GPU memory footprints and multi-view drift; DINOv3D utilizes a parameter-efficient LoRA setup (0.1B trainable parameters) and dual-memory shifted windows to process 128 frames with superior consistency.
• vs Feature3DGS [73]: Feature3DGS relies on per-scene optimization without cross-view geometric rectification, leading to severe semantic blurring as view count grows; DINOv3D is a generalized feed-forward framework with explicit ray-depth alignment.
• vs FiT3D [68] / Pri3D [23]: Previous 3D-to-2D prior transfer frameworks required slow per-scene optimization or offline point-cloud matching; DINOv3D provides a unified feed-forward paradigm that simultaneously excels at 3D perception and enhances 2D foundation features.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Elegant homologous teacher-student 3D Gaussian regularization field combined with linear-complexity dual-memory shifted window attention]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Comprehensive evaluations across novel view synthesis, depth estimation, open-vocabulary segmentation, and 9 downstream 2D linear probing tasks]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear structural narrative, rigorous problem formulation, and consistent mathematical notation]
• Value: ⭐⭐⭐⭐⭐ [Provides an impactful, highly parameter-efficient blueprint for adapting 2D foundation models to unified 3D spatial intelligence]

Related Papers

• [ECCV 2026] RGB-Pointmap Pretraining for Unified 3D Scene Understanding
• [ECCV 2026] Drop-In Perceptual Optimization for 3D Gaussian Splatting
• [ECCV 2026] FlexiAvatar: Unified 3D Gaussian Human Avatars Under Arbitrary Body Visibility
• [ICLR 2026] Joint Optimization for 4D Human-Scene Reconstruction in the Wild
• [CVPR 2026] GS²: Graph-based Spatial Distribution Optimization for Compact 3D Gaussian Splatting