Skyfall-GS: Synthesizing Immersive 3D Urban Scenes from Satellite Imagery¶

Conference: ECCV2026
Paper: ECCV Paper
Official ID: 3179
Project: Skyfall-GS
Area: 3D Vision
Keywords: Gaussian Splatting, multi-date satellite imagery, urban scene synthesis, diffusion refinement, curriculum learning

TL;DR¶

Skyfall-GS anchors coarse urban geometry and appearance in satellite observations, then progressively transfers diffusion-generated facade details into a 3D Gaussian scene through a descending view curriculum, achieving FIDCLIP 27.03 and CMMD 2.110 on DFC2019 and enabling real-time exploration after offline optimization.

Background & Motivation¶

Satellite imagery provides broad coverage of rooftops, roads, and urban layouts, but it is not an ideal observation set for street-level reconstruction. Satellite cameras have limited parallax, facades are frequently occluded, and multi-date images introduce illumination changes, seasonal variation, and transient objects. Directly fitting 3D Gaussian Splatting (3DGS) or Sat-NeRF can produce acceptable overhead views, yet lowering the camera exposes underconstrained Gaussians as floaters, smearing, and blurry facades.

More reconstruction iterations cannot fully resolve this problem because the training images do not contain complete low-viewpoint details. CityDreamer and GaussianCity can generate cities from layouts, but rely more heavily on height fields, semantic maps, and domain-specific training data, without necessarily preserving the appearance of a particular satellite observation. The paper therefore addresses observation-conditioned scene synthesis rather than claiming to recover every real window from distant imagery: a generative prior must supply missing information without changing the observed layout and texture into another city.

Diffusion can improve an individual degraded rendering, but starting with the worst street-level views may amplify errors, while independently edited views may disagree. Core Idea: first establish a satellite-constrained 3D scene, then descend from more reliable high-elevation views and jointly supervise a shared Gaussian representation with multiple diffusion samples per view, progressively reconciling observed structure with generated detail.

Method¶

Overall Architecture¶

The inputs are multi-view, multi-date satellite RGB images of the same area and their camera information; the output is a city-block-scale 3DGS scene supporting free-flight exploration. The pipeline has reconstruction and synthesis stages: Multi-date Reconstruction and Geometric Constraints establish the initial scene, the Descending View Curriculum determines the synthesis order, and Multi-sample Iterative Refinement feeds each round of edited images back into the 3D representation.

The method does not train a new urban diffusion model. It optimizes Gaussian parameters for each scene while using pretrained FLUX.1 [dev] and FlowEdit to provide editing supervision. Real-time exploration renders the optimized Gaussian model, rather than invoking diffusion for every frame.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-view satellite imagery<br/>and camera information"] --> B["Multi-date Reconstruction"]
    B --> C["Geometric Constraints"]
    C --> D["Descending View Curriculum"]
    D --> E["Multi-sample Iterative Refinement"]
    E -->|Next episode after updating the scene| D
    E --> F["Real-time explorable<br/>3D Gaussian scene"]

Key Designs¶

1. Multi-date Reconstruction: separate capture-condition changes from scene structure

Satellite imagery commonly uses a rational polynomial camera (RPC) model and cannot simply be treated as ordinary perspective photographs. SatelliteSfM approximates perspective intrinsics and extrinsics, and its sparse SfM points initialize the Gaussians. This connects satellite observations to standard 3DGS optimization while ensuring that the initial spatial structure comes from observations rather than unrestricted diffusion generation.

The same rooftop may have different colors across capture dates. Forcing a single appearance onto every image can encourage optimization to explain lighting changes through incorrect geometry. Following WildGaussians, the method learns a global embedding per image and a local embedding per Gaussian; a lightweight MLP combines them with zeroth-order spherical harmonic color to predict an affine color transformation. Only zeroth- and first-order spherical harmonics are retained to avoid excessively encoding capture-condition changes as view-dependent color. At inference, a fixed image embedding is selected and appearance is baked into static Gaussian colors; the embeddings and MLP are discarded, retaining compatibility with standard 3DGS renderers.

2. Geometric Constraints: remove floaters while constraining low-viewpoint surfaces

Many floaters in the initial reconstruction have low opacity. Entropy-based opacity regularization encourages a more binary opacity distribution, making low-opacity Gaussians easier to remove during densification pruning. Rather than uniformly reducing opacity, it limits the freedom to explain images with translucent structures suspended away from actual surfaces.

A second constraint uses pseudo-cameras placed closer to the ground. The current Gaussian model renders both RGB and alpha-blended depth, while MoGe predicts monocular depth from the rendered RGB; agreement between these depth structures supervises geometry. MoGe does not introduce additional real street photographs: it supplies a pretrained geometric prior conditioned on the current rendering. Since monocular depth does not guarantee absolute scale, supervision uses the absolute Pearson correlation coefficient instead of a direct metric-depth error.

The equation below is reconstructed from the legible prose surrounding Eq. (3). Formula layout is damaged in the text cache, so the corrupted extraction is not presented as an intact original equation:

\[ \mathcal{L}_{\mathrm{depth}}=1-\left|\operatorname{PCorr}(\hat D_{\mathrm{GS}},\hat D_{\mathrm{est}})\right|, \qquad \operatorname{PCorr}(X,Y)=\frac{\operatorname{Cov}(X,Y)}{\sigma_X\sigma_Y}. \]

The two depth maps come from Gaussian rendering and MoGe prediction, with covariance and standard deviations computed over corresponding pixels. This objective constrains depth correlation, not absolute scale, and predicted priors are not measurements; it works alongside original satellite reconstruction supervision. The reconstruction ablation also validates digital surface models (DSMs) against LiDAR, distinguishing geometric evaluation from visual impressions alone.

3. Descending View Curriculum: repair easier views before exposing occluded regions

The initial model is more reliable at high elevations; low-elevation errors can cause the editor to misinterpret building boundaries. The method places look-at points uniformly across the scene, samples orbital cameras around each point, and controls their trajectories through episode-specific radii and elevations. The curriculum starts high and gradually descends, rather than randomly reusing original training cameras or rendering all difficult near-ground views immediately.

Crucially, the 3D scene changes between episodes: the next round receives a representation that has already absorbed previous refinements, not the original degraded model. Facades become visible gradually, so each editing step faces smaller missing regions and can better preserve existing structure. A reverse curriculum processes the worst low-elevation views first and subsequently moves upward; its substantially poorer ablation results show that the contribution is the ordering of information, not merely additional editing rounds.

4. Multi-sample Iterative Refinement: turn 2D edits into supervision for a shared 3D representation

Each episode renders its curriculum-selected cameras and edits the images using FlowEdit with FLUX.1 [dev]. Source prompts describe blur and artifacts, while target prompts describe clear, plausible building appearance. The objective is to improve existing content rather than freely fill facades treated as entirely unknown. Oblique satellite views already contain some facade color and structure, making preservation of input constraints important. Edited images serve as pseudo ground truth for Gaussian optimization, followed by re-rendering in an Iterative Dataset Update (IDU) loop.

With only one diffusion result per view, random window patterns or edges can be overfitted by the Gaussians and conflict with other views. Multiple independently edited samples instead constrain the same model through the photometric loss. This "consensus" emerges through optimization of the shared 3D representation; it neither explicitly matches windows across images nor simply averages images before training. It mitigates stochastic high-frequency geometric noise without guaranteeing strict multi-view consistency. The default uses 2 samples per view, and increasing the count does not monotonically improve every metric.

A Worked Example¶

Consider a city block with visible rooftops and only partial oblique facade coverage. Satellite images first fit the initial Gaussians, while appearance modeling prevents changing shadows from being explained as suspended structures. Geometric constraints then remove floaters and encourage more regular depth structures on rooftops and roads.

During synthesis, each look-at point receives 6 orbital cameras per episode and 2 edited samples per camera, producing 12 refined images per look-at point per episode. Higher views first repair contours and local textures; Gaussian updates precede the next reduction in elevation, gradually revealing facades over 5 episodes. The number 12 is calculated from the reported configuration, not an additional performance measurement reported by the paper.

The final facade appearance reflects both satellite constraints and information generated by the diffusion prior. Looking coherent from nearby positions does not establish that newly synthesized doors and windows correspond to real building details.

Loss & Training¶

Reconstruction runs for 30,000 iterations, combining color reconstruction, opacity regularization, and pseudo-camera depth supervision. The reported weights are 0.2 for D-SSIM, 10 for opacity, and 0.5 for depth. The color objective combines pixel reconstruction with structural similarity, while geometric constraints address underconstrained structure caused by limited parallax.

Synthesis uses 5 episodes of 10,000 iterations each. Training images are sampled 75% from IDU-refined views and 25% from original satellite views, so generated supervision never completely replaces the real inputs. Retaining original images continues to constrain appearance and reduces drift away from the satellite observations during repeated editing.

Experiments use one RTX A6000 (48GB). The full JAX_214 pipeline takes approximately 6 hours 45 minutes: 1 hour 35 minutes for reconstruction and 5 hours 10 minutes for synthesis. These are per-scene offline costs. After appearance baking, the model renders at 60 FPS at 1920×1080 on a MacBook Pro M4 Pro; this rendering speed must not be reported as generation speed.

Key Experimental Results¶

Main Results¶

DFC2019 uses 4 Jacksonville AOIs: JAX_004, JAX_068, JAX_214, and JAX_260. GoogleEarth uses NYC scenes 004, 010, 219, and 336, with training views at 80° elevation; this source is also the training domain for CityDreamer and GaussianCity.

The table selects distribution metrics from Tables 1 and 2, both lower-is-better. FIDCLIP compares distributions of CLIP features, while CMMD uses maximum mean discrepancy on CLIP representations; these assess generated distributions rather than building-by-building geometry. Their magnitudes do not support a direct comparison of difficulty between the datasets.

Dataset	Method	FIDCLIP ↓	CMMD ↓
DFC2019	Sat-NeRF	86.52	4.788
DFC2019	EOGS	87.67	5.291
DFC2019	CoR-GS	84.95	5.692
DFC2019	Mip-Splatting + proposed appearance modeling	86.72	5.404
DFC2019	Skyfall-GS	27.03	2.110
GoogleEarth	CityDreamer	36.66	4.200
GoogleEarth	GaussianCity	28.76	2.915
GoogleEarth	CoR-GS	26.35	3.758
GoogleEarth	Mip-Splatting	16.09	2.086
GoogleEarth	Skyfall-GS	10.29	1.959

Reference images come from Google Earth Studio (GES), with test elevations of 17° for DFC2019 and 45° for GoogleEarth. Besides different viewing protocols, WorldView-3 and GES differ in lighting and color, so the paper treats pixel metrics as secondary references. On GoogleEarth, Skyfall-GS obtains PSNR 14.42, SSIM 0.302, and LPIPS 0.393; Mip-Splatting has a better LPIPS of 0.379, ruling out a claim of superiority on every metric.

Ablation Study¶

The following selection from Table 4 compares synthesis configurations on JAX_068. Times are in hours; unreported curriculum-variant times remain "Not reported." These timings are not the full JAX_214 pipeline measurement.

Config	FIDCLIP ↓	CMMD ↓	Time (hours)
1 sample per view	34.11	3.189	3.44
2 samples per view (default)	28.35	2.875	6.37
3 samples per view	28.64	2.769	7.19
5 samples per view	29.17	2.677	9.80
Random elevations, no curriculum	33.79	3.361	Not reported
Reverse curriculum, low to high	53.03	4.170	Not reported

Key Findings¶

Curriculum direction matters beyond simply adding edits: reverse-curriculum FIDCLIP is 53.03 versus 28.35 for the default. Sending the most degraded images to diffusion first makes subsequent geometric recovery harder.
Multi-sample supervision trades quality against cost: 2 samples achieve the lowest FIDCLIP in the table, whereas 5 yield the lowest CMMD but increase time from 6.37 to 9.80 hours. The paper chooses 2 for visual quality and cost, not because it wins every metric.
The reconstruction ablation in Table 3 uses higher evaluation elevations and should not be mixed with the main low-viewpoint results. Against LiDAR-derived DSM references, MAE decreases from 3.542 meters with appearance modeling alone to 2.250 meters with all constraints; RMSE decreases from 5.218 to 3.483 meters.
Two user studies each involve 44 participants. Reported preference win rates are approximately 90–94% on DFC2019 and 79–82% on GoogleEarth, supporting perceptual quality rather than independently establishing pointwise geometric truth.

Highlights & Insights¶

The camera curriculum determines what evidence the generative prior receives. For scene completion from sparse observations, exposing reliable regions before progressively revealing missing areas may be more stable than beginning with the largest viewpoint gap.
Multiple samples do more than increase output diversity: they reduce the dominance of an individual stochastic edit over the 3D representation. The transferable idea is to let a shared representation absorb multiple supervisory hypotheses while checking whether the compromise loses detail.
Appearance modeling addresses deployment as well as optimization stability. Keeping embeddings and the MLP in training, then baking them into standard Gaussians, avoids making the generative toolchain a per-frame rendering dependency.

Limitations & Future Work¶

The authors exclude strict pedestrian-level synthesis: elevations below 10° and facade-close views remain difficult to generate and evaluate reliably. Fixed heuristic camera trajectories may also miss blind spots behind complex geometry.
Inputs require off-nadir observations that expose some facade information; purely nadir imagery remains an open problem. "Satellite imagery only" does not imply that any single overhead image is sufficient.
Evaluation covers a limited set of static scenes from two sources, and refinement requires expensive per-scene optimization. Combining adjacent AOIs demonstrates an approximately 1 km × 512 m scene in approximately 9 hours, not general validation at whole-city scale.
This note's risk assessment: generated windows, facade textures, and occluded regions are plausible hypotheses, not direct evidence for surveying, navigation safety, or physical simulation. Visibility or uncertainty annotations and separate validation against real near-ground observations would help establish appropriate use boundaries.
This note's evaluation recommendation: test more regions, materials, and extreme views, and report observed-region geometry, unseen-region perceptual quality, and cross-view consistency separately rather than making one generative metric support every conclusion.

vs Sat-NeRF / EOGS: these methods emphasize reconstruction under satellite observations. Skyfall-GS adds generative refinement to improve low-viewpoint exploration, so its advantage concerns synthesis capability rather than unconditional measurement accuracy.
vs CityDreamer / GaussianCity: these rely on structural conditions such as layouts or heights and domain-specific training. Skyfall-GS conditions on actual multi-view satellite textures and an open-domain diffusion prior, but still requires per-scene optimization and suitable camera coverage.
vs Instruct-NeRF2NeRF: Skyfall-GS adopts the IDU render-edit-update paradigm and addresses satellite-to-low-viewpoint degradation through a descending curriculum and multi-sample supervision; it does not introduce IDU itself.
vs WildGaussians / CoR-GS: appearance separation and sparse-view overfitting are important foundations. Selecting cameras through geometric uncertainty instead of a fixed elevation sequence is a promising extension, but remains a research suggestion rather than an existing component of this paper.

Rating¶

Novelty: 4/5. The combination of existing reconstruction, editing, and IDU components is well targeted, with the main advance in the satellite-to-low-viewpoint curriculum and constraints.
Experimental Thoroughness: 4/5. Two data sources, reconstruction and generation baselines, component ablations, and user studies provide useful evidence, but strict street-level and broader geographic validation remain limited.
Writing Quality: 4/5. The two-stage motivation connects clearly to curriculum ablations, though readers must distinguish geometric accuracy from plausible generation.
Value: 4/5. Useful for interactive urban scenes and simulation assets, provided deployment preserves the uncertainty boundary around generated details.