SkyLume: A Large-Scale Multi-Illumination Aerial Benchmark for Urban Scene Reconstruction and Beyond¶

Conference: ECCV 2026
Paper: ECCV Official
Code: https://skylume.github.io
Area: 3D Vision / Autonomous Driving
Keywords: Large-Scale 3D Reconstruction, 3D Gaussian Splatting, Multi-Illumination, Inverse Rendering, UAV Dataset

TL;DR¶

Addressing the failure of photometric consistency caused by shifting illumination and moving cast shadows across multi-temporal aerial flights, SkyLume establishes the first large-scale real-world UAV benchmark comprising over 100K 6K five-directional oblique images across 10 urban regions, coupled with LiDAR ground truth and the Temporal Consistency Coefficient (TCC) to rigorously benchmark 3D reconstruction, novel view synthesis, and inverse rendering.

Background & Motivation¶

UAV oblique photogrammetry has become an indispensable technology for city-scale 3D modeling and smart city mapping. Over recent years, neural radiance representations—notably Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS)—have substantially improved rendering fidelity and surface reconstruction efficiency. Nonetheless, in real-world large-scale mapping missions, drone flights cannot cover an entire urban district in a single sortie. Data acquisition inevitably spans multiple flight sessions across different hours of the day or across consecutive days. Such multi-temporal capture fundamentally violates the photometric consistency assumption that underpins classical Multi-View Stereo (MVS) and contemporary radiance field optimization. Modern pipelines persistently bake transient lighting, moving penumbras, and variable sun exposures directly into scene geometry or radiance fields, resulting in severe color artifacts, blurred details, and geometric distortions when merging multi-session results.

This foundational challenge has remained largely underexplored because of a critical void in benchmark datasets. Existing UAV corpora fall into two distinct but suboptimal categories: synthetic datasets generated via video game engines (e.g., Matrix City), which offer flexible lighting control but suffer from significant Sim-to-Real domain gaps in sensor noise, optics, and atmospheric scattering; and real-world aerial datasets (e.g., Mill 19, UrbanScene3D, GauU-Scene), which almost universally capture scenes during a single flight session without revisiting regions under distinct sun angles. Furthermore, existing real-world datasets typically lack high-precision active LiDAR scans and per-frame depth/normal ground truth, preventing researchers from isolating material reflectance errors from geometric reconstruction biases.

To overcome these barriers, this paper breaks away from synthetic simulations and single-flight captures to construct a comprehensive real-world multi-temporal aerial benchmark. Core idea: by flying identical RTK-guided waypoint routes across early morning, midday, and late afternoon with synchronized five-directional 6K cameras and active LiDAR, we isolate lighting as the sole independent variable, establish sub-millimeter joint multi-temporal alignment, and introduce the Temporal Consistency Coefficient (TCC) to benchmark appearance decoupling and geometric fidelity without requiring ground-truth physical albedos.

Method¶

Overall Architecture¶

The SkyLume pipeline encompasses four interconnected stages: survey-grade multi-temporal flight acquisition, LiDAR-guided cross-temporal joint SfM registration, multi-modal MVS fusion with water-surface topological completion, and the standardized TCC evaluation framework. The input consists of synchronized 6K nadir and oblique imagery collected along identical RTK routes at three distinct daily time slots along with active LiDAR scans; the output includes COLMAP-compatible multi-period camera poses aligned in WGS 84 / UTM coordinates, per-frame metric depth/normal ground truth, solar geometry annotations, and standardized benchmark leaderboards across inverse rendering, surface extraction, and novel view synthesis.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Survey-Grade Multi-Temporal UAV Capture<br/>Morning / Noon / Evening + Identical RTK Route + 5-Direction 6K Imagery"] --> B["LiDAR-Guided Cross-Temporal Joint SfM Registration"]
    B --> C["Multi-Modal MVS Fusion and Water Geometry Completion"]
    C --> D["Standardized Multi-Modal Ground Truth Export<br/>Unified Metric Poses / Dense Depth & Normals / Solar Geometry"]
    D --> E["Temporal Consistency Coefficient (TCC) Protocol"]
    E --> F["Downstream Multi-Task Benchmarks<br/>Inverse Rendering / Surface Mesh Extraction / Novel View Synthesis"]

Key Designs¶

1. LiDAR-Guided Cross-Temporal Joint SfM Registration: Sub-Pixel Multi-Modal Alignment Across Drastic Illumination Shifts

Between low-angle morning direct sunlight, harsh noon diffuse irradiance, and late afternoon backlight, architectural façades undergo extreme radiometric changes and non-linear gradient shifts, causing standard independent feature matching to drift or segment into disjoint coordinate frames. To achieve seamless cross-temporal and cross-modal registration, SkyLume leverages an active LiDAR point cloud (acquired via a DJI Zenmuse L2) georeferenced to WGS 84 / UTM as a metric anchor. Virtual camera poses are defined to render pseudo-RGB anchor views directly from the LiDAR point cloud. Subsequently, an anchor-constrained joint bundle adjustment is executed: all five-directional RGB images across all three time slots are optimized simultaneously alongside the LiDAR anchor views. RTK GNSS positions serve as soft regularization priors, and manual Ground Control Points (GCPs) are incorporated in low-texture areas. The global SfM solution achieves an average reprojection error of 0.79 pixels and an average track length of 3.3, keeping relative camera position uncertainties \(\le 0.001\text{ m}\) to \(0.002\text{ m}\) across X, Y, and Z axes. Once converged, time-specific subsets are exported with matched camera parameters.

2. Multi-Modal MVS Fusion and Water Geometry Completion: Constructing Illumination-Invariant Geometric Ground Truth

Varying lighting not only alters photometric texture but also severely degrades classical MVS depth estimation, frequently causing dark building shadows to be misconstrued as physical hollows or artificial surface extrusions. To establish absolute geometric ground truth uncorrupted by lighting, SkyLume performs dense surface reconstruction by fusing the multi-temporal 5-directional imagery with active LiDAR scans, using the LiDAR as a metric scaffold to regularize depth in shadowed and weakly textured areas. Furthermore, urban water bodies (rivers, lakes, fountains) routinely cause catastrophic MVS failure due to view-dependent specular reflections and translucency. SkyLume introduces a targeted water-surface completion algorithm: for each water body, four shoreline boundary points are sampled to calculate the mean elevation \(z_{\text{mean}}\) with sample variance constrained to \(\le 2\text{ cm}\). The surface normals of the water patch are strictly constrained to be horizontal, and planar mesh patches are stitched into the global scene model. Based on the completed watertight geometry, per-frame dense mesh depth and surface normals matching the native 6K resolution are generated alongside occupancy-filtered sparse LiDAR depth maps.

3. Temporal Consistency Coefficient (TCC): Quantifying Material Decoupling Without Physical Ground-Truth Albedo

Acquiring dense ground-truth physical albedos at city scale is practically intractable. To evaluate whether inverse rendering algorithms achieve genuine material-illumination decoupling rather than merely memorizing observed views, SkyLume establishes the Temporal Consistency Coefficient (TCC) framework. The physical premise is that intrinsic surface albedo must remain invariant over time despite diurnal sun angle progression. Taking advantage of the unified coordinate system, \(K\) virtual camera viewpoints \(\{v_k\}_{k=1}^K\) are uniformly sampled along the global trajectory. For each viewpoint \(v_k\), models trained independently on \(T=3\) distinct time slots render predicted albedo maps \(A_t^{(k)} \in [0, 1]^{H \times W \times 3}\), and the cross-temporal mean albedo is computed:

\[\bar{A}^{(k)} = \frac{1}{T}\sum_{t=1}^T A_t^{(k)}\]

To prevent single-metric bias, the TCC framework formulates structural stability via \(TCC_{\text{SSIM}}\), perceptual distance via \(TCC_{\text{LPIPS}}\), and pixel-level absolute radiometric deviations via \(TCC_{\text{MAE}}\) and \(TCC_{\text{RMSE}}\):

\[TCC_{\text{metric}} = \frac{1}{KT} \sum_{k=1}^K \sum_{t=1}^T \text{metric}\left(A_t^{(k)}, \bar{A}^{(k)}\right), \quad \text{metric} \in \{\text{SSIM}, \text{LPIPS}, \text{MAE}, \text{RMSE}\}\]

Inverting error metrics to similarity scores yields the composite overall metric \(TCC_{\text{overall}} \in [0, 1]\), where a theoretically perfect decomposition across changing lighting scores exactly 1:

\[TCC_{\text{overall}} = \frac{1}{4}\left( TCC_{\text{SSIM}} + (1 - TCC_{\text{LPIPS}}) + (1 - TCC_{\text{MAE}}) + (1 - TCC_{\text{RMSE}}) \right)\]

Key Experimental Results¶

Main Results¶

All benchmarks were systematically executed on an NVIDIA A800 80GB GPU across six representative urban regions, evaluating leading inverse rendering algorithms (GS-IR, Ref-Gaussian, Ref-GS), surface geometry extractors (2DGS, PGSR, GOF, CityGaussianV2), and novel view synthesis pipelines (3DGS, Abs-GS, Mip-Splatting, Octree-GS).

Table 1: Quantitative evaluation of inverse-rendering baselines using the TCC framework (corresponds to original Table 4) All metrics lie in \([0, 1]\). Higher is better for all metrics except Std (standard deviation).

Metric	Stat	GS-IR	Ref-Gaussian	Ref-GS	Analysis / Takeaway
TCC-LPIPS	Mean ↑	0.826	0.866	0.874	Perceptual consistency is relatively high across models; Ref-GS leads
	Min ↑	0.759	0.778	0.755	Worst-case perceptual stability under extreme illumination disparity
	Max ↑	0.867	0.919	0.979	Near-perfect perceptual alignment under favorable diffuse conditions
	Std ↓	0.020	0.027	0.029	Cross-scene dispersion remains compact
TCC-SSIM	Mean ↑	0.905	0.883	0.928	High structural stability (\(\ge 0.88\)), indicating texture contours persist
	Min ↑	0.864	0.832	0.878	Worst-case structural retention
	Max ↑	0.936	0.928	0.985	High-contrast structural boundaries remain stable across slots
	Std ↓	0.014	0.019	0.017	Most stable sub-metric across all scenes
TCC-MAE	Mean ↑	0.700	0.513	0.766	Color drift is the core bottleneck; Ref-Gaussian drops to 0.513
	Min ↑	0.212	0.262	0.587	Severe albedo corruption in shadow-edge regions; Ref-GS is noticeably more resilient
	Max ↑	0.800	0.653	0.977	Unshadowed flat façades exhibit decent radiometric stability
	Std ↓	0.074	0.078	0.063	Large variance highlights vulnerability to hard shadows
TCC Overall	Mean ↑	0.721	0.658	0.775	Ref-GS achieves the best overall score (0.775); Ref-Gaussian trails (0.658)
	Min ↑	0.563	0.575	0.622	Minimum bound under difficult aerial lighting shifts
	Max ↑	0.765	0.735	0.913	Best-case scene performance
	Std ↓	0.033	0.026	0.036	Overall composite variance remains within 0.036

Table 2: Surface geometry benchmark under varying illumination across distance tolerances \(\tau \in \{0.25, 0.50, 0.75\}\text{ m}\) (corresponds to original Table 5) Reports Precision (Pre), Recall (Rec), and F-1 scores against LiDAR ground truth for Period 1 (sunlit) and Period 2 (diffuse), plus cross-temporal pairwise mesh consistency F-1 (TCC-Geometry).

Method	Period 1 (Sunlit) F-1 ↑	Period 1 (Sunlit) F-1 ↑	Period 1 (Sunlit) F-1 ↑	Period 2 (Diffuse) F-1 ↑	Period 2 (Diffuse) F-1 ↑	Period 2 (Diffuse) F-1 ↑	TCC-Geometry F-1 ↑	TCC-Geometry F-1 ↑	TCC-Geometry F-1 ↑
Tolerance \(\tau\)	0.25 m	0.50 m	0.75 m	0.25 m	0.50 m	0.75 m	0.25 m	0.50 m	0.75 m
2DGS	0.227	0.508	0.704	0.232	0.552	0.735	0.570	0.675	0.750
PGSR	0.152	0.428	0.719	0.171	0.420	0.720	0.554	0.662	0.749
GOF	0.103	0.372	0.672	0.116	0.400	0.689	0.431	0.610	0.712
CityGaussianV2	0.291	0.610	0.808	0.250	0.566	0.770	0.547	0.719	0.790

Table 3: Novel view synthesis comparison under Period 1 (direct sunlight) on four representative scenes (corresponds to original Table 6)

Scene Name	Metric	3DGS	Abs-GS	Mip-Splatting	Octree-GS	GOF	2DGS	PGSR
Gym (Low density, no glass)	PSNR ↑	21.144	21.183	21.131	20.931	20.830	20.350	20.110
	SSIM ↑	0.672	0.681	0.669	0.648	0.626	0.583	0.580
	LPIPS ↓	0.318	0.305	0.325	0.336	0.360	0.425	0.424
iPark (High-rise, glass façades)	PSNR ↑	23.250	22.958	22.708	22.681	23.172	22.722	21.953
	SSIM ↑	0.707	0.711	0.701	0.697	0.681	0.648	0.622
	LPIPS ↓	0.355	0.335	0.362	0.358	0.382	0.440	0.467
Tec School (Medium density, water)	PSNR ↑	21.652	21.523	21.364	21.765	21.224	21.103	20.926
	SSIM ↑	0.631	0.639	0.621	0.607	0.619	0.549	0.537
	LPIPS ↓	0.384	0.375	0.398	0.360	0.401	0.489	0.500
High School (Medium density, shadows)	PSNR ↑	20.570	20.413	OOM	20.618	OOM	19.531	19.398
	SSIM ↑	0.626	0.634	OOM	0.624	OOM	0.497	0.490
	LPIPS ↓	0.389	0.382	OOM	0.380	OOM	0.539	0.547

Key Findings¶

Inverse rendering exhibits a pronounced decoupling asymmetry between structure and color: Across all inverse-rendering baselines, structural stability remains relatively high (\(TCC_{\text{SSIM}} \ge 0.88\)), indicating that geometry contours and texture gradients are reasonably maintained across sessions. In contrast, pixel-level absolute consistency drops drastically (\(TCC_{\text{MAE}}\) drops to 0.513 for Ref-Gaussian and hits a minimum of 0.212 for GS-IR). Qualitative inspection reveals that current pipelines fail to untangle transient shadows from surface reflectance, permanently burning building shadows into estimated albedos.
Harsh sunlight degrades surface reconstruction more severely than diffuse illumination: Geometry extracted under overcast or diffuse conditions (Period 2) exhibits far greater structural completeness than under direct sunlight (Period 1). Under direct sunlight, elongated moving shadows induce massive holes and fragmented topology across lawns and building bases. Methods relying on opacity fields (GOF, CityGaussianV2) achieve superior F-1 scores on flat structures under favorable light (CityGaussianV2 reaches 0.808 at \(\tau=0.75\text{ m}\)), but degrade precipitously under strong shadows; conversely, TSDF-fusion methods based on alpha-blended depth (2DGS, PGSR) provide smoother topology and achieve superior cross-temporal geometric consistency (\(TCC_{\text{Geometry}}\) F-1 of 0.570 at \(\tau=0.25\text{ m}\) for 2DGS).
Surface geometry regularization incurs a distinct photometric penalty in NVS: Rendering-centric methods (Abs-GS, 3DGS, Octree-GS) consistently outperform geometry-oriented pipelines (2DGS, PGSR) by 1.0–1.5 dB in PSNR and 0.10–0.15 in LPIPS across all scenes. Introducing explicit surface planarity or normal regularization limits the capacity of anisotropic Gaussians to fit view-dependent and shadow-dominated aerial appearances.
Exploratory tasks reveal fundamental domain gaps for aerial vision: Experiments on monocular depth/normal estimators, classical MVS (COLMAP), and feed-forward 3D foundation models (\(\pi^3\), VGGT) show that classical MVS suffers structural disconnection under multi-temporal shadow shifts, monocular estimators over-smooth architectural roof boundaries, and feed-forward models collapse under raw, wide-angle distorted UAV trajectories.

Highlights & Insights¶

Survey-Grade Multi-Temporal Flight Protocol: By automating identical RTK flight trajectories and five-camera synchronized triggering across three daily time windows, SkyLume strictly isolates illumination as the sole environmental variable, providing an uncorrupted empirical foundation for multi-temporal research.
Ground-Truth-Free Decoupling Evaluation via TCC: Leveraging sub-millimeter multi-temporal registration, TCC compares rendered albedos from cross-temporal models at identical virtual test viewpoints, bypassing the intractable challenge of acquiring physical ground-truth albedos at city scale.
LiDAR-MVS Synergistic Fusion with Water Surface Inpainting: Active LiDAR provides a metric scaffold that stabilizes SfM bundle adjustment and regularizes shadowed MVS depths, while an elevation-constrained planar repair algorithm eliminates specular MVS topological corruption over urban water bodies.

Limitations & Future Work¶

Author-Acknowledged Limitations: Current data captures are confined to daytime illumination variations (clear sky, direct sunlight, overcast diffuse light) and do not yet encompass nocturnal artificial illumination or adverse weather (fog, rain, snow); annotations currently lack multimodal thermal infrared channels and per-pixel semantic segmentations.
Additional Identified Limitations: The water completion algorithm assumes a strictly static planar elevation, which may introduce minor boundary discrepancies in tidal rivers or fluctuating reservoirs; furthermore, because TCC evaluates consistency relative to the cross-period mean albedo, it could overestimate decoupling performance if all baselines share a systematic, correlated bias in predicting shadow regions.
Future Research Directions: Integrating thermal infrared imagery could provide illumination-independent thermal inertia cues to disambiguate shadows from surface albedo changes; in neural radiance modeling, future work should develop shadow-aware 3DGS paradigms featuring explicit spherical harmonic decoupling, differentiable ambient occlusion, and sun-position-guided inverse rendering.

vs GauU-Scene [46] / UrbanScene3D [20]: Prior real-world aerial benchmarks collect scenes during a single flight sortie, making it impossible to evaluate algorithm robustness against solar diurnal variations; SkyLume provides the first multi-temporal aligned benchmark specifically isolating illumination shifts.
vs Matrix City [15]: Matrix City synthesizes flexible weather and illumination via Unreal Engine 5 but carries noticeable domain gaps in vegetation geometry, sensor noise, and atmospheric scattering; SkyLume offers 100% real-world sensor captures with survey-grade LiDAR ground truth.
vs GS-IR [18] / Ref-GS [57]: Existing inverse rendering techniques were predominantly conceived for object-centric or indoor scenes; SkyLume demonstrates that these methods struggle severely at city scale, where complex occlusion, non-Lambertian glass façades, and moving penumbras lead to heavy shadow imprinting on albedo estimates.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ (Pioneering large-scale real-world multi-illumination aerial benchmark with over 100K 6K images, paired LiDAR geometry, and the novel TCC evaluation metric)
Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Extensive evaluation across 11 SOTA methods spanning inverse rendering, geometry reconstruction, and novel view synthesis, complemented by exploratory downstream pilot studies)
Writing Quality: ⭐⭐⭐⭐⭐ (Exemplary clarity in pipeline description, photogrammetric parameters, mathematical formulation of TCC, and insightful empirical analysis)
Value: ⭐⭐⭐⭐⭐ (Foundational resource for large-scale 3D vision, urban digital twins, autonomous aerial mapping, and illumination-invariant neural radiance fields)