WildCity: A Real-World Dataset for City-Scale Rendering and Beyond¶

Conference: ECCV 2026
Paper: ECCV official page
Project: WildCity
Area: 3D Vision / City-scale reconstruction and neural rendering
Keywords: continuous street-view data, 3D Gaussian Splatting, rigid camera rig, ground regularization, off-trajectory views

Source note: this note uses the official title supplied in the task; the PDF body is titled WildCity: A Real-World City-Scale Testbed for Rendering, Simulation, and Spatial Intelligence. The cached text contains the full method, experiments, and discussion, but not the supplementary six-city experiments referenced in the paper. The project link is not a verified code repository.

TL;DR¶

WildCity organizes 18 continuous driving logs from six U.S. cities into 1,507.1 km of surround-view RGB–LiDAR data and introduces a 3DGS baseline combining rig-aware pose optimization, separate sky modeling, and ground regularization, showing that city reconstruction requires not only room for more Gaussians but also control of pose drift, geometric ambiguity, and off-trajectory rendering errors.

Background & Motivation¶

City-scale reconstruction is not simply room reconstruction with more data: driving routes are long and narrow, viewpoints stay close to roads, and distant buildings and road surfaces often lack sufficient intersecting observations. Vehicles, pedestrians, lighting, and localization errors also change throughout the journey. A model may render attractive street views along the recorded route while placing geometry at the wrong depth. Once a simulated agent leaves that route, errors that were inconspicuous in projection become floaters, broken roads, or unstable backgrounds.

Existing datasets leave different gaps. Synthetic datasets such as MatrixCity offer control but introduce domain differences; real driving datasets such as nuScenes and Waymo are mainly organized into short clips; KITTI-360 and Oxford RobotCar provide longer routes but have limitations in multi-city coverage, surround-view sensing, or reconstruction protocols. Block-NeRF and city-scale Gaussian methods improve scale through partitioning, hierarchies, and parallelization, but increasing capacity does not automatically calibrate poses or recover true geometry under narrow baselines.

WildCity preserves the continuity and uncertainty of long real-fleet logs and supplies reproducible multi-scale segments and an urban reconstruction baseline. Core idea: move city digital-twin evaluation beyond whether images look right on the recorded route to whether geometry remains trustworthy across scales and useful off the route, using continuous multimodal data and structural constraints to expose problems that partitioning and capacity alone cannot solve.

Method¶

Overall Architecture¶

The paper contributes a dataset, a reconstruction baseline, and a closed-loop demonstration. Six surround-view cameras, a roof-mounted LiDAR, an IMU, and GPS collect the data; calibration and synchronization are followed by SLAM/GPS pose initialization and masks for sky, ground, and moving objects. Reconstruction represents the static environment with 3D Gaussian Splatting, jointly optimizes geometry and camera-rig poses, and applies distinct modeling constraints to sky and roads. Optionally, diffusion-repaired novel views are fed back into training.

The result is a city environment that can be queried from different viewpoints, not a complete traffic-dynamics world model. Moving objects are filtered, while stationary instances such as parked vehicles are retained where possible. Alpamayo 1 then proposes the next target pose from the task and current rendered observation, and the simulator renders that pose to return a new observation. This loop demonstrates interface feasibility, but the main text provides no comprehensive evaluation of navigation success, collision rates, or responses from other traffic participants.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Surround RGB, LiDAR, IMU, GPS"] --> B["Continuous multimodal processing<br/>poses, masks, scale-based segments"]
    B --> C["Rig-aware pose optimization<br/>joint scene and pose updates"]
    C --> D["Region-specific sky and ground constraints<br/>static city Gaussians"]
    D --> E["Off-trajectory repair feedback<br/>render, repair, re-optimize"]
    E -.->|Optional pseudo-supervision feedback| D
    D --> R["Pose queries and rendering"]
    E --> R
    R -->|Current observation| V["VLA target-pose prediction"]
    V -->|Next target pose| R

Key Designs¶

1. Continuous multimodal processing: preserve real noise while providing reusable geometric references

Three forward-facing narrow-angle cameras and three lateral/rear wide-angle cameras form the surround-view system, alongside LiDAR, IMU, and GPS. AprilCal calibrates camera intrinsics and distortion, while extrinsics map sensors into the ego-vehicle frame. Initial ego poses come from onboard SLAM and are refined with GPS to improve global consistency and revisited loops. Different sensor frequencies are handled through original-timestamp alignment rather than resampling everything to a common frequency; LiDAR sweeps also receive motion compensation to reduce motion-induced distortion. These processing steps do not make the poses error-free ground truth, so subsequent joint optimization remains useful.

SAM3 uses text prompts to produce masks for ground, sky, and potentially movable categories. Onboard 3D tracking cuboids then distinguish genuinely moving objects from stationary instances. A vehicle is not automatically something to remove: parked vehicles and standing pedestrians may be retained, while moving targets are filtered to avoid baking motion traces into static Gaussians. Overall mask mIoU is 91.58% on 100 manually annotated images across four cities, supporting this coarse regional supervision without guaranteeing every occlusion boundary or rare category is correct.

Keyframes are spatially sampled every 0.5 m and organized into continuous 5 km chunks, with 50 m, 250 m, 500 m, 1 km, 2.5 km, and 5 km sub-trajectories for studying local reconstruction through long-range drift. Original acquisition rates and spatial subsampling are distinct: cameras, LiDAR, and GPS operate at 10 Hz, and the IMU at 100 Hz, whereas reconstruction keyframe spacing is determined by traveled distance. Shared city-level coordinates let segments be concatenated directly, but compatible coordinates do not guarantee seamless appearance or geometry after concatenation.

2. Rig-aware pose optimization: constrain six cameras through shared ego motion

Optimizing each camera pose independently may reduce local rendering error while violating the rigid relationship between simultaneous camera views. The paper decomposes each camera pose into a per-keyframe ego pose and camera extrinsics shared across keyframes. The composition clearly specified in the text is:

\[ T_{t,c}=T^{\mathrm{ego}}_t T^{\mathrm{rig}}_c. \]

Here \(t\) denotes a keyframe and \(c\) a camera. The renderer uses the composed camera pose to generate images, and reconstruction error jointly updates Gaussian parameters, ego poses, and rig extrinsics. A regularizer penalizes departures from initial localization and calibration. Rather than letting each view independently choose the pose that looks best, the views must share motion consistent with one rigid sensor platform, reducing cross-camera inconsistency.

This is not a replacement for a complete SLAM pipeline: it refines an existing localization initialization using rendering supervision. It requires a sufficiently good initialization and remains affected by lighting changes and geometric ambiguity. Removing pose optimization increases Atlanta depth L1 from 13.26 m to 14.29 m in the ablation, although some 2D metrics improve slightly, so its value cannot be summarized by a single PSNR number.

3. Region-specific sky and ground constraints: different poorly observed regions need different priors

Unlike nearby surfaces, the sky has no reliably triangulated finite depth; representing it directly with Gaussians can create distant floaters. A lightweight view-dependent MLP predicts sky colors as an infinite background, composed with the foreground according to its remaining transmittance: the Gaussian image plus sky color multiplied by one minus Gaussian opacity. This separates sky appearance from finite-depth scene geometry and reserves Gaussians for physical scene structure rather than spurious distant surfaces.

Roads face a different problem: weak texture and restricted viewing angles. For Gaussians classified as ground, the method samples local slices along camera depth and penalizes the standard deviation of camera-frame vertical coordinates within each slice. It additionally encourages the shortest Gaussian axis to align vertically and promotes sufficient opacity. These priors favor stable local surfaces rather than transparent or undulating geometry that compensates for rendering errors. Because they depend on ground classification and camera coordinates, possible oversmoothing on slopes, curbs, or erroneous masks requires separate evaluation.

The two treatments do not amount to flattening every region. Sky becomes an infinite background, whereas ground remains renderable 3D Gaussians with local structural priors. In Table 5, removing sky modeling raises Atlanta depth error to 148.54 m, strongly supporting sky separation. Ground regularization does not improve average depth error in that table; the authors primarily motivate it with qualitative road geometry, and this distinction is preserved here.

4. Off-trajectory repair feedback: generated pseudo-supervision, not new real observations

A simulator must query viewpoints outside the training route, yet long, narrow driving observations constrain lateral road views and occluded regions poorly. The method renders the current Gaussians at an extrapolated pose, passes the rendering, reference image, and prompt to Difix3D+, and adds the repaired image paired with its pose to the training set. Gaussians are re-optimized and the process repeats. This progressive render–repair–augment loop is more than a final-image filter because repaired outputs change the underlying scene representation.

The feedback supplies a generative model's inferred supervision, not new geometric ground truth. It can reduce broken surfaces and floaters, but it may invent nonexistent structures when the starting reconstruction is weak; the authors explicitly acknowledge hallucination and high cost. The main text primarily provides qualitative off-trajectory results rather than a complete independent numerical ablation of repair, so it does not establish safe navigation at every extrapolated position.

Multi-GPU Gaussian parameter sharding, with synchronization of necessary statistics and gradients, provides the engineering support for large-scale training following GrendelGS and GSplat. Sharding addresses whether the representation fits in memory; pose and structural constraints address whether it reconstructs correctly. These are complementary requirements, and stated support for billion-level primitives should not be described as a completed accuracy evaluation at that scale.

Loss & Training¶

Confirmed training components include image-rendering loss, pose-distance regularization relative to initialization, and three ground terms covering height variation, shortest-axis alignment, and opacity. Several loss equations are corrupted in the cached text, and the main text does not fully specify weights, optimizer, learning rates, or a uniform iteration count. This note therefore retains the mechanisms and clearly recoverable pose-composition equation without inventing an exact total loss.

Comparisons use identical, sufficiently provisioned H200 hardware and train to stable validation performance where feasible. VGGT-Long outputs are aligned to metric point clouds through a global similarity transform. VGGT-Long+CityGS replaces CityGS's default SLAM poses and sparse-point initialization. Color metrics cover only mask-defined static regions; depth L1 further restricts evaluation to pixels with valid LiDAR depth in those regions. These are not full-scene or dynamic-object metrics.

Key Experimental Results¶

Main Results¶

The dataset reports 3.01M keyframes, 18 logs across six cities, and 1,507.1 km of cumulative driving, averaging approximately 83.7 km and 2.5 hours per log. Cumulative distance can include revisits and must not be described as 1,507.1 km of distinct roads. Table 3's resource references grow with trajectory length: 0.5/1/2.5/5 km correspond to about 6M/12M/30M/60M Gaussians and minimum VRAM configurations of 1×24G, 1×40G, 1×80G, and 2×80G.

Table 4 is reproduced below. The labels 0.5k and 5k count timestamps, corresponding to 0.25 km and 2.5 km, not 0.5 km and 5 km. Each cell contains PSNR / SSIM / LPIPS / depth L1 (m); the first two are higher-is-better, and the latter two lower-is-better.

Method	Ann Arbor, 0.25 km	Atlanta, 2.5 km
3DGS	20.00 / 0.829 / 0.581 / 32.704	19.39 / 0.715 / 0.492 / 15.447
H-3DGS	27.59 / 0.849 / 0.318 / 18.450	21.40 / 0.747 / 0.380 / 14.368
CityGS	24.41 / 0.856 / 0.409 / 17.053	21.27 / 0.762 / 0.536 / 8.349
VGGT-Long	— / — / — / 29.632	— / — / — / 19.720
VGGT-Long+CityGS	20.98 / 0.663 / 0.535 / 22.695	13.40 / 0.666 / 0.745 / 20.934
WildCity baseline	29.99 / 0.917 / 0.240 / 15.158	23.14 / 0.799 / 0.477 / 6.622

The proposed baseline leads all four metrics on the short route. On the long route it leads PSNR, SSIM, and depth L1, but its LPIPS of 0.477 is worse than H-3DGS's 0.380. Against CityGS on the long route, PSNR improves by 1.87 dB and depth L1 drops by 1.727 m. Replacing CityGS initialization with VGGT-Long lowers PSNR to 13.40 dB, indicating that these feedforward geometry priors cannot directly replace sensor-localization initialization on this long sequence.

A numerical conflict must remain visible: §5.2 states a short-route depth error of 11.75 m, whereas Table 4 reports 15.158 m for the proposed method. Comparisons here follow the table, without substituting 11.75 or speculating about a different experimental version.

Ablation Study¶

Table 5 compares ground regularization, sky modeling, and pose optimization. Arlington covers 0.1 km and Atlanta 0.25 km, unlike the main table's long route, so their absolute errors should not be compared directly.

Configuration	Arlington PSNR ↑	SSIM ↑	LPIPS ↓	Atlanta PSNR ↑	SSIM ↑	LPIPS ↓	Depth L1 (m) ↓
Without ground regularization	30.14	0.924	0.186	26.35	0.864	0.288	13.11
Without sky model	28.69	0.903	0.230	24.72	0.829	0.362	148.54
Without pose optimization	29.33	0.913	0.199	25.51	0.843	0.320	14.29
Full model	29.60	0.910	0.217	25.42	0.815	0.342	13.26

Sky modeling contributes most clearly to this depth metric: error falls from 148.54 m without sky modeling to 13.26 m in the full model, a difference of 135.28 m. Ground regularization's benefit is not in aggregate scores: removing it improves all 2D metrics in both scenes and slightly reduces Atlanta depth L1 from 13.26 to 13.11 m. The authors show qualitative evidence of more stable roads, but the table does not establish better average depth. This design needs complementary local road-geometry metrics rather than being described as improving everything.

Key Findings¶

Scaling is not only a memory problem. With Gaussian budgets matched in the scale analysis, real data degrade more than synthetic Carla, implicating accumulated appearance variation, localization errors, and limited overlap. The main text does not provide a full numerical table at every scale, so values are not guessed from extracted figure text.
Geometry and 2D quality can diverge. Removing sky modeling causes enormous depth degradation despite comparatively modest changes in some 2D metrics; ground regularization illustrates a different trade-off between aggregate scores and qualitative structure.
On-trajectory performance does not establish extrapolation quality. Every method degrades off the recorded path. Diffusion repair helps but can hallucinate, and offline image quality is not evidence of simulation safety.
City-scale data do not mean city-wide validation in the main table. The main comparison covers 0.25 km and 2.5 km subsequences from two cities. The referenced supplementary six-city results were not verified in the current cache.

Highlights & Insights¶

Rig structure is an inexpensive consistency constraint. Sharing ego poses across simultaneous cameras and camera extrinsics across time respects the acquisition platform more closely than unrestricted per-image pose refinement.
Semantic regions determine geometric assumptions. Sky and roads both lack strong depth constraints, yet require infinite-background modeling and local-surface priors respectively, not simply greater model capacity.
Positive and negative results remain in the same table. Ground regularization's aggregate scores are not superior, making visible which task-specific metrics are still missing from the argument for simulation usefulness rather than relying only on better-looking screenshots.

Limitations & Future Work¶

Annotations and poses remain imperfect. The 91.58% mask mIoU comes from 100 images across four cities and cannot cover every boundary or rare class. The authors report residual vertical error after GPS anchoring and describe sub-centimeter horizontal and centimeter-level vertical drift per kilometer in loop-closure analysis, without fully expanding the measurement details.
Depth accuracy does not yet establish a usable digital twin. Even the best long-route depth L1 is 6.622 m. A mean over static pixels with valid LiDAR cannot directly establish road traversability, collision geometry, or safety throughout the scene.
Dynamic behavior is not fully modeled. The baseline filters moving objects, and the closed-loop demonstration does not report traffic interaction, rewards, collision constraints, or task success. It is not a validated complete autonomous-driving simulator.
Diffusion repair introduces pseudo-supervision risks. Visual coherence is not geometric truth, and repeated feedback may solidify a generative prior into scene content; independent geometry or uncertainty estimates are still needed.
Reproduction and version boundaries. Corrupted cached equations, missing training hyperparameters and complete split details, and the source's 11.75/15.158 m short-route discrepancy remain unresolved rather than filled in. Future work could add road-specific geometry metrics, quantitative off-route tasks, and consistent cross-city protocols.

vs. MatrixCity and SS3DM: WildCity retains real noise and long-range multi-city observations for studying real reconstruction degradation, while synthetic data remain complementary for controlled evaluation with known ground truth.
vs. Block-NeRF, H-3DGS, and CityGaussianV2: partitioning and hierarchies address storage and computation; this paper additionally emphasizes sensor-pose consistency and semantic structural constraints rather than replacing scalable frameworks with an entirely new representation.
vs. VGGT-Long: feedforward priors perform poorly in this long-sequence initialization-replacement experiment, but the result should not be generalized to every feedforward model or urban scenario; sensor configuration and alignment procedures also matter.
vs. Difix3D+ and Alpamayo: these are existing components used for repair and a closed-loop proof of concept. WildCity's main additions are the data, the urban reconstruction baseline, and the analysis, not a newly invented diffusion repair model or driving policy.

Rating¶

Novelty: 4/5. Continuous multi-city real-world surround data and simulation-oriented analysis are valuable, while the baseline largely integrates existing modules with urban structural constraints.
Experimental Thoroughness: 3/5. Includes main comparisons, component ablations, mask validation, and scale analysis, but main-table scenes are limited, closed-loop and extrapolation quantification is incomplete, and depth values conflict in the source.
Writing Quality: 3/5. The problem and data pipeline are clear, but overall-best claims and geometric benefits must be separated from individual table metrics; cached equations and title versions also complicate verification.
Value: 4/5. Provides a stricter real-city reconstruction testbed than short clips and encourages joint evaluation of images, geometry, and off-route usability.