Geometric Foundation Model Distillation for Efficient Lunar 3D Reconstruction¶

Conference: ECCV 2026
Paper: CVF Open Access
Code: https://clementinegrethen.github.io/publications/ECCV.html
Area: 3D Vision
Keywords: Geometric Foundation Model / Knowledge Distillation / Lunar 3D Reconstruction / SVD Weight Initialization / Cross-view Stereo Matching

TL;DR¶

Addressing the stringent onboard compute limitations in planetary exploration, this paper presents a compact knowledge distillation framework for large 3D foundation models (MASt3R) on uncalibrated lunar stereo reconstruction, leveraging truncated SVD decoder weight initialization and margin-filtered feature alignment to compress the model by 4.4× to 7.3× with a >2× speedup while incurring only a ~15% Chamfer error increase and outperforming sparse ground-truth supervision.

Background & Motivation¶

Recent vision foundation models for dense multi-view 3D reconstruction, such as DUSt3R, MASt3R, and VGGT, have achieved state-of-the-art accuracy by directly regressing globally consistent 3D geometry and relative camera poses from uncalibrated image pairs without explicit camera calibration or traditional Structure-from-Motion (SfM) pipelines. However, these models rely heavily on massive transformer backbones (ViTs) comprising hundreds of millions to over a billion parameters, demanding substantial memory bandwidth and computational budget. Consequently, these systems remain restricted to high-end terrestrial compute clusters and are inaccessible for direct onboard deployment on resource-constrained edge platforms.

This disparity becomes acutely problematic in planetary exploration. Onboard processing units on lunar landers and rovers operate under strict thermal, power, memory, and latency constraints. Simultaneously, precise 3D reconstruction is essential for safety-critical tasks including autonomous hazard detection, terrain navigation, and descent guidance. The lunar operating environment further compounds this difficulty due to unique domain properties rarely represented in terrestrial training sets, such as low-texture regolith, harsh lighting with extreme shadowed relief, and restricted baseline parallax during descent trajectories. Directly running a 688M-parameter MASt3R model onboard is impractical, whereas training compact student networks directly with sparse ground truth frequently results in degraded geometry and training instability.

Rather than designing increasingly heavy universal models, the authors propose a domain-tailored compression strategy: distilling a domain-adapted geometric foundation model into a lightweight student family. By transferring the dense representations learned by a fine-tuned MASt3R teacher on the StereoLunar benchmark, the student models can learn dense 3D geometry without requiring ground-truth camera pose labels. Core idea: distill a 688M lunar-adapted MASt3R teacher into compact student models using truncated SVD initialization to project decoder weights into reduced latent dimensions, combined with margin-filtered feature alignment and pseudo-ground-truth supervision, demonstrating that preserving transformer encoder capacity while aggressively pruning the decoder provides the optimal trade-off for resource-constrained 3D reconstruction.

Method¶

Overall Architecture¶

The framework establishes an end-to-end stereo 3D knowledge distillation pipeline. It takes an uncalibrated lunar stereo image pair $(I_1, I_2)$ as input and outputs dense 3D pointmaps expressed in the coordinate frame of the first camera, pixel-wise confidence maps, and 24-dimensional local descriptors for cross-view feature matching.

The architecture comprises a domain-adapted teacher model, an intermediate feature projection module, an SVD-based structured decoder initialization strategy, and a dual-supervision distillation loss. The teacher is a fine-tuned MASt3R model (ViT-Large encoder with 12 cross-attention decoder blocks, totaling 688.6M parameters). During training, multiple student variants share a single teacher forward pass to acquire dense pseudo-ground-truth (Pseudo-GT) supervision without ground-truth pose annotations. To alleviate the slow convergence caused by random decoder initialization, a truncated Singular Value Decomposition (SVD) projection warm-starts the student decoder from the teacher weights. Joint optimization is then performed using margin-filtered feature alignment and output-level geometric regression.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Lunar Stereo Pair (I₁, I₂)"] --> B["Multi-Student Shared Teacher Forward & Pseudo-GT<br/>MASt3R-Large generates pointmaps/confidence/descriptors"]
    A --> C["Asymmetric Architecture: Light Encoder & Compressed Decoder<br/>Preserve ViT expressiveness + prune decoder depth/width"]
    B -.->|SVD principal truncation + uniform layer mapping| D["Truncated SVD Decoder Initialization<br/>Retain dominant singular components & project latent space"]
    D --> C
    C --> E["Margin-Filtered Feature Alignment<br/>Learnable projector + α=0.9 hard-sample cosine loss"]
    B -.->|Provides intermediate feature targets| E
    C --> F["Dense Geometry & Matching Distillation<br/>Scale-normalized 3D regression + InfoNCE descriptor loss"]
    B -.->|Provides dense pseudo-GT labels| F
    F --> G["Output: Dense 3D Pointmaps, Confidence, Relative Pose"]

Key Designs¶

1. Multi-Student Shared Teacher Forward & Pseudo-GT: Annotation-Free Dense Supervision In planetary exploration, reliable metric ground truth and relative camera poses are notoriously difficult to acquire and often sparse. The framework leverages a 688.6M-parameter MASt3R teacher fine-tuned on the StereoLunar benchmark to produce dense, view-consistent 3D pointmaps, confidence scores, and local feature descriptors as pseudo-ground truth. To circumvent the high computational cost of querying a large teacher model across diverse student configurations, all student networks are trained synchronously within the same iteration loop. A single teacher forward pass produces the supervisory signals shared across all student backbones, amortizing the teacher inference cost.

2. Truncated SVD Decoder Initialization: Structural Knowledge Projection Across Latent Spaces Initializing the student decoder weights randomly results in sluggish optimization, as the student must first learn cross-view feature correlation from scratch before benefiting from fine-grained distillation. To provide an effective warm start, the authors develop a structured initialization scheme based on truncated Singular Value Decomposition (SVD). Given a teacher weight matrix $W_T \in \mathbb{R}^{d_T^{out} \times d_T^{in}}$, its SVD factorizes as: $$W_T = U \Sigma V^\top$$ where $\Sigma = \operatorname{diag}(\sigma_1, \ldots, \sigma_k)$ with singular values arranged in descending order. By the Eckart-Young-Mirsky theorem, retaining the leading $r = \min(d_S^{out}, d_S^{in})$ singular components yields the optimal low-rank approximation in the Frobenius norm. The student weight matrix $W_S \in \mathbb{R}^{d_S^{out} \times d_S^{in}}$ is initialized as: $$W_S = U_{[:d_S^{out}, :r]} \operatorname{diag}(\sigma_1, \ldots, \sigma_r) V_{[:r, :d_S^{in}]}^\top$$ To compress the decoder depth from $T=12$ layers to $S=6$ layers, a uniform stride mapping $\phi(i) = \lfloor \frac{i \cdot T}{S} \rfloor$ is employed, selecting teacher blocks $[0, 2, 4, 6, 8, 10]$. One-dimensional bias parameters are sliced directly ($b_S = b_T[:d_S]$), while convolutional filters are flattened into 2D matrices, compressed via truncated SVD, and reshaped back. This projects the dominant directional transformations of the teacher directly into the student parameter space.

3. Margin-Filtered Feature Alignment: Focusing Representation Distillation on Challenging Regions Output-level regression alone struggles to bridge the substantial capacity gap between teacher and student. To guide internal representations, the framework introduces an intermediate feature distillation loss. Because the student encoder channel dimension $C_s$ differs from the teacher's $C_t$, a learnable linear projector $W \in \mathbb{R}^{C_t \times C_s}$ maps student features into the teacher representation space. Both projected student representations and detached teacher features are $\ell_2$-normalized across channels, yielding spatial unit vectors $\hat{\mathbf{f}}_s^{(i)}$ and $\hat{\mathbf{f}}_t^{(i)}$. The feature distillation loss incorporates a tolerance margin $\alpha$: $$\mathcal{L}_{feat} = 1 - \frac{1}{|\mathcal{M}|} \sum_{i \in \mathcal{M}} \hat{\mathbf{f}}_s^{(i)} \cdot \hat{\mathbf{f}}_t^{(i)}, \quad \mathcal{M} = \left\{i \;\middle|\; \hat{\mathbf{f}}_s^{(i)} \cdot \hat{\mathbf{f}}_t^{(i)} < \alpha\right\}$$ Setting $\alpha = 0.9$ filters out spatial regions where cosine similarity already exceeds 0.9, preventing well-aligned low-frequency regions from diluting the gradient and forcing the optimization to focus strictly on fine geometric boundaries, steep terrain slopes, and extreme shadows.

4. Asymmetric Architecture: Light Encoder & Compressed Decoder Capacity Trade-off The empirical analysis investigates CNN backbones (MobileNetV3) versus Vision Transformers (DINOv2 ViT-Small, ViT-Tiny) alongside variable decoder dimensions. The findings establish that dense cross-view geometry estimation fundamentally requires the long-range relational reasoning of self-attention mechanisms; the MobileNet baseline fails to produce accurate parallax correspondences. Furthermore, the capacity bottleneck lies in the encoder rather than the decoder. Student S3 (ViT-Small encoder + reduced decoder 256/4/4, 94.9M parameters) significantly outperforms student S4 (ViT-Tiny encoder + standard decoder 512/6/4, 138.5M parameters), demonstrating that preserving encoder representation capacity while compressing cross-view decoder width and depth yields the superior efficiency-accuracy frontier. Additionally, encoder weights must remain unfrozen during distillation to adapt pretrained representations to lunar photometric distributions.

Loss & Training¶

The total distillation loss is formulated as: $$\mathcal{L} = \mathcal{L}_{geo} + \gamma \mathcal{L}_{feat} = \mathcal{L}_{conf} + \beta \mathcal{L}_{match} + \gamma \mathcal{L}_{feat}$$ where: - $\mathcal{L}_{conf}$ represents the scale-normalized 3D point regression loss, normalized by the median ground-truth scene depth with a logarithmic confidence penalty to avoid trivial predictions; - $\mathcal{L}_{match}$ denotes the block-wise InfoNCE contrastive loss over cross-view descriptors; - Models are trained for 50 epochs across 7 NVIDIA A100 GPUs using the AdamW optimizer with a cosine learning rate decay schedule, combined with data augmentations including color jitter, random cropping, grayscale tinting, bilateral filtering, and contrast perturbations.

Key Experimental Results¶

Main Results¶

Evaluation was conducted on a test partition of StereoLunar containing 1,200 stereo image pairs spanning nadir, pitched, dynamic orbital trajectories, and various altitudes. Metric scale is recovered via RANSAC similarity alignment with ground-truth point clouds. Reconstruction fidelity is evaluated using Chamfer distance, Accuracy, and Completeness normalized by median ground-truth depth (AbsRel). Landing terrain safety is measured by Slope Mean Absolute Error (MAE) and Profile MAE. Pose estimation accuracy is assessed using Virtual Correspondence Reprojection Error (VCRE).

Model	Encoder Backbone	Decoder (d/L/h)	Params (M)	Chamfer ↓	Accuracy ↓	Completeness ↓	VCRE Med. (% diag) ↓	VCRE Prec@5% ↑	Profile MAE (m) ↓	Slope MAE (°) ↓
Teacher (MASt3R)	ViT-Large	768 / 12 / 12	688.6	0.46%	0.46%	0.46%	1.15	98.6%	53.63	7.22
S1	MobileNetV3	512 / 6 / 4	135.7	1.02%	1.09%	0.95%	8.75	36.8%	161.85	8.37
S2 (Best Student)	ViT-Small	512 / 6 / 4	154.9	0.53%	0.56%	0.51%	3.56	82.8%	75.69	7.59
S3 (Reduced Dec.)	ViT-Small	256 / 4 / 4	94.9	0.68%	0.70%	0.65%	3.57	79.2%	85.82	7.97
S4	ViT-Tiny	512 / 6 / 4	138.5	0.74%	0.80%	0.68%	4.04	76.5%	93.41	8.03

Inference efficiency measured on a single NVIDIA A100 GPU across various batch sizes ($N$ image pairs):

Model	Params (M)	Compression	Latency (N=2)	Latency (N=32)	Latency (N=64)	Latency (N=128)	Speedup (N=128)
Teacher	688.6	1.0×	0.07 s	2.04 s	4.17 s	8.36 s	1.0×
S1	135.7	5.1×	0.04 s	0.91 s	1.84 s	3.69 s	2.27×
S2	154.9	4.4×	0.04 s	0.98 s	1.99 s	4.01 s	2.08×
S3	94.9	7.3×	0.04 s	0.86 s	1.73 s	3.47 s	2.41×
S4	138.5	5.0×	0.03 s	0.87 s	1.77 s	3.55 s	2.35×

Ablation Study¶

Ablation analysis on student S2 (DINOv2 ViT-Small backbone) isolating the impact of distillation supervision, intermediate feature alignment, SVD initialization, and encoder fine-tuning:

Experiment	Distillation (KD)	Feature Loss $\mathcal{L}_{feat}$	SVD Init	Unfrozen Encoder	Chamfer ↓	Accuracy ↓	Completeness ↓	Description
Teacher	–	–	–	–	0.46%	0.46%	0.46%	Upper bound teacher baseline
Exp. A (Full Method)	✓	✓	✓	✓	0.53%	0.56%	0.51%	Proposed complete distillation pipeline
Exp. B (w/o SVD)	✓	✓	✗	✓	0.68%	0.69%	0.67%	Random initialization (+28.3% error)
Exp. C (Frozen Enc.)	✓	✓	✓	✗	0.87%	0.85%	0.89%	Severe domain mismatch on lunar data
Exp. D (w/o Feat. Loss)	✓	✗	✓	✓	0.67%	0.70%	0.63%	Missing intermediate representation guidance
Exp. E (Pure GT Superv.)	✗	✗	✗	✓	0.65%	0.69%	0.61%	Fully-supervised baseline without teacher (+22.6% error)

Key Findings¶

Teacher Pseudo-GT Outperforms Ground Truth: Full distillation (Exp. A, 0.53% Chamfer) outperforms direct ground-truth supervision (Exp. E, 0.65% Chamfer) by ~18.5%. The dense geometric priors embedded in the teacher provide a smoother, more informative gradient surface than sparse, noise-prone terrain ground truth.
Critical Impact of SVD Weight Initialization: Omitting SVD initialization (Exp. B) causes Chamfer error to degrade from 0.53% to 0.68%. Retaining the dominant singular components of the teacher decoder weights prevents early training divergence and provides a robust warm start.
Encoder Capacity Trumps Decoder Capacity: S3 (ViT-Small, 256/4/4 decoder, 94.9M) beats S4 (ViT-Tiny, 512/6/4 decoder, 138.5M) on Chamfer distance (0.68% vs 0.74%) and VCRE pose accuracy (Prec@5% of 79.2% vs 76.5%) despite having 31% fewer parameters. Preserving strong visual representations is far more critical than allocating capacity to cross-view decoding.
CNN Backbones Underperform on Multi-View Geometry: S1 (MobileNetV3, 135.7M) shows severe degradation with a Chamfer error of 1.02% and VCRE Prec@5% falling to 36.8%, confirming that standard convolutional receptive fields struggle with uncalibrated cross-view matching.

Highlights & Insights¶

SVD Cross-Dimensional Weight Transfer: By applying the Eckart-Young-Mirsky theorem to truncate teacher matrices and projecting singular components into student dimensions, the method provides an elegant mathematical solution to structural parameter transfer between non-identical transformer architectures.
Margin-Filtered Hard-Region Distillation: Excluding spatial locations with cosine similarity $> 0.9$ dynamically focuses intermediate representation learning on difficult shadow boundaries and complex slopes, avoiding gradient saturation on uninformative flat regions.
Practical Asymmetric Allocation Rule: The empirical discovery that decoder capacity is largely redundant while encoder representational power must be conserved establishes a practical rule of thumb for deploying stereo reconstruction models on compute-bound systems.
Broad Domain Transferability: While validated on lunar terrain, this pipeline directly translates to other edge-constrained robotics domains facing extreme lighting or feature degradation, such as underground mining navigation, UAV search-and-rescue, and laparoscopic surgical stereo imaging.

Limitations & Future Work¶

Author-Acknowledged Limitations: The evaluation relies on the photorealistic synthetic StereoLunar benchmark; validation on real lunar flight data subject to flight-sensor noise, radiation artifacts, and camera motion blur remains an open objective.
Independent Observations: At low batch sizes ($N=2$), the absolute speedup is 1.75× (0.07 s to 0.04 s) despite a 7× parameter reduction, reflecting hardware memory bandwidth saturation typical of small-batch attention operations. Furthermore, the performance gap between MobileNet and ViT is partially confounded by disparities between ImageNet and DINOv2 self-supervised pretraining regimes.
Future Directions: Exploring low-bit post-training quantization (INT8/FP4) combined with FlashAttention kernel optimization could further reduce memory footprint; integrating monocular elevation priors could further stabilize extremely reduced decoders.

vs DUSt3R / MASt3R: While DUSt3R and MASt3R pioneer uncalibrated dense 3D pointmap regression, their 688M+ footprint renders them unsuitable for onboard space platforms. This work preserves their core dual-decoder formulation while achieving up to 7.3× parameter compression.
vs VGGT: VGGT expands 3D foundation modeling to video sequences with over 1 billion parameters, requiring extensive multi-GPU clusters. This paper demonstrates that mission-specific operational domains can be effectively addressed by compact distilled student networks.
vs DUNE / Distill3R: DUNE aggregates multiple heterogeneous teachers into an encoder without an end-to-end 3D reconstruction head; Distill3R targets indoor scenes without structured weight transfer. This paper introduces the first holistic encoder-decoder compression pipeline leveraging SVD initialization for deep-space planetary mapping.

Rating¶

Novelty: ⭐⭐⭐⭐☆ (Pioneering distillation pipeline for dense geometric foundation models tailored to extreme planetary terrain, featuring SVD-based cross-dimensional weight transfer.)
Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Comprehensive evaluation spanning 4 student backbones, detailed ablations on supervision/loss/initialization, full geometric and pose metrics, and multi-batch latency benchmarking.)
Writing Quality: ⭐⭐⭐⭐⭐ (Exemplary clarity, mathematically sound formulation, and coherent technical narrative.)
Value: ⭐⭐⭐⭐⭐ (Provides an actionable engineering blueprint and clear architectural design guidelines for deploying foundation 3D vision on compute-constrained edge robotics.)