SkySplat-OV: Generalizable Language Gaussian Splatting for Open-Vocabulary Scene Understanding from Sparse Satellite Views¶
Conference: ECCV 2026
Paper: ECCV Official
Project Page: https://skysplat-ov.github.io/
Area: 3D Vision
Keywords: 3D Gaussian Splatting / Open-Vocabulary Scene Understanding / Satellite Imagery / Sparse-View Reconstruction / Foundation Model Distillation
TL;DR¶
The first self-supervised, generalizable language 3DGS framework designed for sparse satellite views that incorporates rational polynomial coefficient (RPC) camera models into a feed-forward pipeline, leverages Geometry–Radiance–Semantics Fusion (GRS-Fusion) alongside relative height supervision, and employs single-stage SAM3 distillation to achieve up to 1070× inference speedup over per-scene optimization while delivering state-of-the-art open-vocabulary 3D semantic segmentation.
Background & Motivation¶
Reconstructing 3D geometry and interpreting open-vocabulary semantic concepts from multi-view satellite imagery is essential for urban planning, ecological monitoring, and global geographic information systems. Conventional earth observation pipelines rely heavily on fully supervised learning frameworks, necessitating expensive dense ground-truth annotations for elevation models and semantic masks. Furthermore, these methods are confined to rigid, closed-set label spaces that fail to generalize to arbitrary novel classes encountered in the real world. While recent advances in Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have demonstrated impressive capability in lifting 2D vision–language representations into 3D spaces via feature distillation, representative models such as Feature-3DGS, LangSplatV2, Sat-NeRF, and SatGS depend strictly on per-scene optimization. This paradigm requires several hours of iterative gradient descent per scene and demands dense multi-view observations, severely hindering scalable deployment across large-scale geographical areas.
Recent generalizable 3DGS architectures (such as MVSplat, LSM, and Uni3R) have emerged to predict Gaussian parameters and lift features into 3D in a single feed-forward pass, drastically accelerating reconstruction and improving transferability. However, directly applying these models to sparse satellite imagery exposes three fundamental challenges: first, high-resolution satellite imagery typically follows a line-scan pushbroom imaging geometry characterized by rational polynomial coefficients (RPCs), completely violating the standard pinhole projection model assumed by mainstream 3D vision methods; second, constrained orbital revisits usually provide only two or three sparse observation perspectives, leading to severely under-constrained multi-view geometry and causing catastrophic geometric collapse or floating artifacts in directly predicted Gaussians; third, satellite imagery covers vast geographic scales where object dimensions vary across orders of magnitude—from small vehicles to expansive water bodies—often forcing prior frameworks into complex, slow, multi-stage cascading distillation routines to preserve multi-scale details.
To resolve these interconnected bottlenecks, SkySplat-OV seamlessly incorporates the rigorous RPC camera projection model into an end-to-end feed-forward language 3DGS architecture while constructing a unified multi-modal fusion mechanism and a fully self-supervised training paradigm. Core idea: integrate satellite RPC projection geometry directly into feed-forward 3DGS cost volume construction, unify geometric, radiometric, and semantic features through a GRS-Fusion module, and supervise the language Gaussian field using monocular relative height from DA3 and single-stage SAM3 feature distillation without requiring any ground-truth annotations.
Method¶
Overall Architecture¶
SkySplat-OV takes \(N\) sparse satellite images \(\{I_i\}_{i=1}^N\) alongside their corresponding RPC parameters as inputs. The full pipeline unfolds across four tightly coordinated phases: multi-view deep feature extraction and RPC-guided initial height regression, multi-modal feature integration via Geometry–Radiance–Semantics Fusion (GRS-Fusion) for Gaussian parameter estimation, vision foundation model (VFM) aligned self-supervised multi-task training, and downstream open-vocabulary scene understanding driven by a frozen SAM3 decoder.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Sparse Satellite Images & RPC Parameters"] --> B["RPC-Based Geometry Reconstruction<br/>Epipolar Cost Volume & Height Regression"]
B --> C["GRS-Fusion Module<br/>Multi-Modal U-Net & Gaussian Parameter Heads"]
C --> D["Single-Stage SAM3 Distillation & Alignment<br/>RGB Loss + SAM3 Feature Distillation + DA3 Relative Height"]
D --> E["Open-Vocabulary Scene Understanding<br/>Injecting SAM3 FPN & Decoder for 3D DSM Output"]
At inference time, the model executes a single feed-forward pass to predict all geometric, radiometric, and language parameters for the 3D Gaussians. The rendered 3D language features directly substitute for the SAM3 image encoder outputs and feed into SAM3's feature pyramid network (FPN) and cross-attention detection heads, generating highly view-consistent open-vocabulary semantic masks and digital surface model (DSM) level 3D segmentations from arbitrary text prompts.
Key Designs¶
1. RPC-Based Geometry Reconstruction: Cost Volume Construction and Height Regression under Pushbroom Geometry
Addressing the breakdown of standard pinhole epipolar constraints in pushbroom satellite imaging, the framework explicitly leverages the rational polynomial coefficient (RPC) model to establish rigorous forward and backward mappings between the 3D geographic object space and 2D image coordinates. After extracting initial features via a lightweight CNN and enriching them with a multi-view Transformer to form \(\{F_i\}_{i=1}^N\), the algorithm samples \(M\) discrete candidate height planes \(\{h_m\}_{m=1}^M\) (defaulting to 64 layers) for every reference feature pixel \((u_i, v_i)\). Using the RPC projection operator, candidate 3D positions are reprojected onto source feature maps \(F_j\) to obtain corresponding coordinates: $$ (u_j^{h_m}, v_j^{h_m}) = \mathrm{RPC_Proj}(u_i, v_i, h_m) $$ Source feature representations are sampled via bilinear interpolation and concatenated with reference features along the channel dimension to form the raw cost volume. A 3D CNN regularizes the volume across the height dimension to output regularized cost volumes \(C_i\) and normalized probability volumes \(P_i\). Finally, continuous initial height maps \(H_i = \sum_{m=1}^M h_m P_i(h_m)\) are derived through expectation-based height regression. Unprojecting \(H_i\) back into physical space via the RPC model provides accurate 3D Gaussian centroid positions \(\mu_m\).
2. Geometry–Radiance–Semantics Fusion (GRS-Fusion): Multi-Modal Harmonization for Gaussian Parameter Estimation
Under sparse observation conditions with only two or three views, relying on pure visual radiance leads to ill-posed Gaussian parameter estimation and severe geometric drift. To overcome this, the GRS-Fusion module acts as a centralized multi-modal representation hub, integrating three distinct information streams: geometric cues (regularized cost volumes \(C_i\), predicted height maps \(H_i\), and matching confidence maps derived from the maximum values of \(P_i\)), radiometric cues (multi-view raw images \(I_i\)), and semantic cues (language features \(F_i^{\text{sam}}\) extracted from frozen SAM3 and concatenated CNN/Transformer features).
These heterogeneous features are channel-wise concatenated and processed by a lightweight 2D U-Net to produce a compact, unified feature representation \(F_i^{\text{grs}} \in \mathbb{R}^{\frac{H}{s} \times \frac{W}{s} \times C_{\text{grs}}}\). Dedicated prediction heads then estimate the full suite of Gaussian parameters: - Language-Aligned Semantic Head: Processes \(F_i^{\text{grs}}\) alongside context features and incorporates a residual skip connection from the frozen SAM3 feature \(F_i^{\text{sam}}\) to reliably estimate language embeddings \(f_m\); - Covariance and Color Head: Combines \(F_i^{\text{grs}}\) with radiometric inputs to regress scale factors \(s_m\), quaternion-based rotation matrices \(R(\theta_m)\), and RGB colors \(C_m\); - Opacity Head: Concatenates \(F_i^{\text{grs}}\) with geometric confidence features to predict opacity values \(\alpha_m\) via convolutional layers. Multi-modal coupling firmly anchors Gaussian attributes, effectively eliminating floating artifacts.
3. Single-Stage SAM3 Distillation and Monocular Relative Height Self-Supervision: Cross-Modal 3D Alignment
Instead of using complex multi-stage cascading distillation schemes to bridge scale disparities in satellite imagery, SkySplat-OV distills foundation model capabilities directly into the 3D Gaussian field in a single stage. During volumetric splatting (approximating local ray pencils with equivalent pinhole projections), color images \(I^{\text{render}}\) and language feature maps \(F^{\text{render}}\) are rasterized simultaneously using identical opacity blending weights: $$ I^{\text{render}} = \sum_{m=1}^M C_m \alpha_m \prod_{n=1}^{m-1}(1 - \alpha_n), \quad F^{\text{render}} = \sum_{m=1}^M f_m \alpha_m \prod_{n=1}^{m-1}(1 - \alpha_n) $$ To train the entire pipeline without ground-truth geometry or semantic labels, three complementary self-supervision signals are deployed: 1. Appearance Reconstruction: Combines pixel-level MSE with perceptual LPIPS loss, formulated as \(\mathcal{L}_{\text{rgb}} = \mathrm{MSE}(I^{\text{gt}}, I^{\text{render}}) + \mathrm{LPIPS}(I^{\text{gt}}, I^{\text{render}})\); 2. Language Feature Alignment: Rendered feature maps are projected through two-layer MLPs into aligned language features \(\hat{F}^{\text{lang}}\), matching frozen SAM3 features \(F^{\text{lang}}\) via combined MSE and cosine similarity: \(\mathcal{L}_{\text{lang}} = \mathrm{MSE}(F^{\text{lang}}, \hat{F}^{\text{lang}}) + (1 - \cos(F^{\text{lang}}, \hat{F}^{\text{lang}}))\); 3. Monocular Relative Height Alignment: Relative depth/height maps \(hei\) are predicted from reference images via the pre-trained monocular model Depth Anything v3 (DA3). To handle scale and shift ambiguities, a Pearson correlation loss enforces structural geometric consistency: $$ \mathcal{L}_{\mathrm{hei}} = \frac{\mathrm{Cov}(hei, H_i)}{\sqrt{\mathrm{Var}(hei)\mathrm{Var}(H_i)}} $$ The composite self-supervised objective \(\mathcal{L} = \mathcal{L}_{\mathrm{rgb}} + \mathcal{L}_{\mathrm{lang}} + \mathcal{L}_{\mathrm{hei}}\) unifies appearance fidelity, geometric plausibility, and open-vocabulary semantic grounding into a cohesive 3D representation.
Loss & Training¶
The entire framework is optimized end-to-end using the composite self-supervised loss \(\mathcal{L} = \mathcal{L}_{\mathrm{rgb}} + \mathcal{L}_{\mathrm{lang}} + \mathcal{L}_{\mathrm{hei}}\). Optimization is performed using AdamW with an initial learning rate of \(2 \times 10^{-4}\). During cost volume construction, 64 candidate height planes are uniformly sampled. Experiments are trained on a workstation equipped with eight NVIDIA GeForce RTX 4090 GPUs (24 GB memory each) running Ubuntu 22.04, without any manual 3D ground truth or semantic mask supervision.
Key Experimental Results¶
Main Results¶
Quantitative evaluations are conducted across the DFC19 and SpaceNet4 benchmarks, assessing novel view synthesis quality (PSNR, SSIM, LPIPS) alongside open-vocabulary novel view semantic segmentation (mF1, mIoU). Competing baselines include two-stage cascading approaches that segment HiSplat-rendered views with 2D open-vocabulary models (ProxyCLIP, Trident, SegEarth-OV) and generalizable 3D vision–language models (LSM, Uni3R).
| Dataset | Paradigm | Method | mF1 (%) ↑ | mIoU (%) ↑ | PSNR (dB) ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|---|---|---|---|
| DFC19 | 2D OV Cascade | ProxyCLIP (HiSplat rendered) | 29.12 | 20.07 | 16.88 | 0.381 | 0.564 |
| 2D OV Cascade | Trident (HiSplat rendered) | 29.93 | 20.79 | 16.88 | 0.381 | 0.564 | |
| 2D OV Cascade | SegEarth-OV (HiSplat rendered) | 31.80 | 22.37 | 16.88 | 0.381 | 0.564 | |
| Generalizable 3D | LSM-2V | 31.42 | 19.95 | 15.71 | 0.402 | 0.603 | |
| Generalizable 3D | LSM-3V | 32.19 | 20.58 | 16.52 | 0.431 | 0.600 | |
| Generalizable 3D | Uni3R | 34.18 | 22.20 | 16.56 | 0.450 | 0.604 | |
| Ours | SkySplat-OV | 62.42 | 48.12 | 17.63 | 0.531 | 0.443 | |
| SpaceNet4 | 2D OV Cascade | ProxyCLIP (HiSplat rendered) | 53.07 | 37.64 | 16.46 | 0.321 | 0.488 |
| 2D OV Cascade | Trident (HiSplat rendered) | 47.86 | 34.08 | 16.46 | 0.321 | 0.488 | |
| 2D OV Cascade | SegEarth-OV (HiSplat rendered) | 53.66 | 38.11 | 16.46 | 0.321 | 0.488 | |
| Generalizable 3D | LSM-2V | 46.76 | 31.22 | 17.05 | 0.257 | 0.655 | |
| Generalizable 3D | Uni3R | 43.08 | 28.58 | 16.54 | 0.258 | 0.654 | |
| Ours | SkySplat-OV | 67.40 | 51.02 | 18.49 | 0.377 | 0.400 |
In cross-dataset zero-shot transfer evaluations, models trained exclusively on US urban scenes (DFC19 or SpaceNet4) were directly evaluated on the MVS3D benchmark located in San Fernando, Argentina. For DFC19 \(\to\) MVS3D, SkySplat-OV reduces reconstruction MAE to 4.28m (compared to 18.01m for HiSplat and 11.37m for LSM) and improves mIoU to 43.22% (vs. 36.17% for the best baseline). For SpaceNet4 \(\to\) MVS3D, SkySplat-OV achieves 4.37m MAE and 45.26% mIoU, demonstrating robust geographic transferability.
Ablation Study¶
Ablation experiments conducted on the DFC19 dataset dissect the individual contributions of the GRS-Fusion module, relative height supervision (R.H.S.), and single-stage feature distillation (F.D.):
| GRS-Fusion Module | Relative Height Supervision (R.H.S.) | Feature Distillation (F.D.) | mF1 (%) ↑ | mIoU (%) ↑ | PSNR (dB) ↑ | SSIM ↑ |
|---|---|---|---|---|---|---|
| ✗ | ✗ | ✗ | — | — | 15.90 | 0.425 |
| ✗ | ✗ | ✓ | 50.20 | 37.28 | 15.68 | 0.420 |
| ✓ | ✗ | ✓ | 57.31 | 43.12 | 17.64 | 0.496 |
| ✗ | ✓ | ✓ | 60.65 | 46.38 | 17.23 | 0.526 |
| ✓ | ✓ | ✓ | 62.42 | 48.12 | 17.63 | 0.531 |
Comparing distillation paradigms further confirms that rendering novel views via pure geometric 3DGS followed by 2D SAM3 segmentation (SkySplat + SAM3) achieves only 22.44% mIoU. In contrast, embedding SAM3 features into 3D Gaussians (SkySplat-OV) boosts mIoU to 48.12% (+25.68%) while raising novel view PSNR from 17.53 dB to 17.63 dB.
Key Findings¶
- GRS-Fusion Stabilizes Multi-Modal Parameter Learning: Introducing GRS-Fusion into the feature-distilled pipeline elevates mIoU from 37.28% to 43.12% (+5.84%) and increases PSNR by nearly 2.0 dB, demonstrating that joint multi-modal conditioning prevents Gaussian attribute degradation under sparse observations.
- Monocular Relative Height Solves Sparse Geometric Ambiguity: Incorporating DA3 relative height supervision with Pearson correlation loss provides strong local smoothness constraints without requiring metric depth, lifting mIoU to 48.12% and slashing cross-dataset elevation MAE by more than 70%.
- Massive Speedup over Per-Scene Optimization: Across eight representative areas of interest (AOIs), per-scene optimization baselines require extensive computation: S-NeRF and Sat-NeRF take 6–7 hours, Feature-3DGS takes 3.82 hours, and LangSplatV2 takes 2.92 hours. SkySplat-OV completes scene reconstruction and understanding in 9.82 seconds, achieving a 1070× speedup over LangSplatV2 while cutting mean MAE from 4.68m (Sat-NeRF) to 2.77m and improving mean mIoU from 22.71% (Feature-3DGS) to 42.15%.
Highlights & Insights¶
- Direct RPC Integration into Feed-Forward 3DGS: Avoids the systematic geometric distortions incurred by naively approximating pushbroom satellite imagery with pinhole cameras, maintaining rigorous photogrammetric precision throughout cost volume construction and Gaussian placement.
- Unified 3D Lifting Guarantees Multi-View Consistency: Directly embedding foundation model features into 3D Gaussians guarantees temporal and cross-view consistency during novel view rendering, completely eliminating the view-inconsistency flicker characteristic of 2D open-vocabulary inference.
- Synergistic Foundation Model Distillation: Creatively leverages complementary 2D foundation models—DA3 for monocular geometric prior regularization and SAM3 for multi-scale open-vocabulary semantics—creating a robust self-supervised learning loop without manual 3D labels.
Limitations & Future Work¶
- Dependency on SAM3 Discriminative Boundaries: The open-vocabulary segmentation quality is bounded by SAM3's pre-trained representations, occasionally struggling with tiny, visually ambiguous satellite structures or dark shadow regions.
- Degradation over Specular Surfaces and Water Bodies: Multi-view stereo cost volume matching degrades in textureless calm water bodies and specular metallic roofs, resulting in occasional local height noise.
- Future Directions: Integrating temporal multi-date satellite observations and incorporating shape-from-shading constraints to better resolve structural details in complex urban shadows.
Related Work & Insights¶
- vs LSM / Uni3R: Both LSM and Uni3R are designed for pinhole cameras and struggle when deployed directly on pushbroom satellite observations. SkySplat-OV explicitly integrates the RPC model and retains clear performance superiority even in controlled pinhole ablation settings, confirming the strength of its multi-modal architecture.
- vs LangSplatV2 / Feature-3DGS: These methods require lengthy per-scene optimization taking several hours per AOI and require dense input views. SkySplat-OV achieves 1070× faster execution via feed-forward inference while delivering superior geometric accuracy and semantic segmentation.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Pioneering integration of rigorous RPC satellite pushbroom geometry with generalizable language 3DGS via GRS-Fusion and self-supervised foundation model distillation.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarks spanning DFC19, SpaceNet4, and MVS3D across 8 AOIs, featuring main comparisons, cross-dataset generalization, per-scene optimization benchmarks, and extensive component ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Methodological progression is rigorous, mathematical derivations are consistent, and empirical analysis is structured and lucid.
- Value: ⭐⭐⭐⭐⭐ Establishes a highly practical, near-real-time foundation for large-scale open-vocabulary satellite 3D reconstruction and geographic scene understanding.