Grounding World Simulation Models in a Real-World Metropolis¶

Conference: ECCV2026
Paper: ECCV Paper · Paper PDF
Area: Video Generation
Keywords: world models, street-view retrieval, cross-temporal pairing, geometric conditioning, long video generation

TL;DR¶

Seoul World Model (SWM) trains a retrieval-augmented video diffusion model on cross-temporal street-view data and constrains autoregressive generation with geometric and semantic conditioning plus a continuously updated virtual lookahead sink, producing more faithful real streets in unseen Busan and Ann Arbor rather than merely plausible fictional cities.

Background & Motivation¶

Existing video world models generate future scenes from an initial image, camera actions, and text, but resembling a street is different from depicting the actual street on a map. As the camera advances, buildings, intersections, and facades outside the original view are often supplied by the generative prior. Locally coherent videos can therefore drift progressively away from the real location. City-scale NeRF or Gaussian Splatting systems reconstruct real scenes, but primarily support static rendering rather than video generation that can change weather, traffic, or events.

Street-view photographs appear to supply the missing location information, yet introduce three mismatches. First, vehicles and pedestrians change across capture times, so enforcing reference-image consistency can copy obsolete transient objects. Second, vehicle-mounted street views are usually sampled 5–20 meters apart: they are neither continuous videos nor a source of diverse pedestrian and free-camera motion. Third, autoregressive errors propagate between chunks. Keeping the starting image as a permanent attention sink becomes less useful once the camera reaches a distant neighborhood.

SWM consequently changes training pairs, video construction, and reference injection together, rather than simply attaching more reference images to a conventional world model. The model must preserve stable structures while allowing transient content to change. Core Idea: learn persistent urban structure through cross-temporal street-view supervision, then re-ground every generated chunk using complementary spatial and appearance references and a future anchor that advances with the route.

Method¶

Overall Architecture¶

Inputs comprise a map location, the current chunk's camera trajectory, a text prompt, and previously generated history; the output is a dynamic video following the specified route. SWM fine-tunes Cosmos-Predict2.5-2B, using a 3D VAE to compress video into a latent space and a Diffusion Transformer (DiT) to generate successive chunks.

Training first constructs cross-temporal reference–target pairs, then uses interpolation and synthetic trajectories to address sparse temporal sampling and restricted motion coverage. During generation, a geo-indexed database retrieves nearby street views that supply geometrically aligned layouts, unwarped appearance details, and a virtual lookahead sink positioned after the current chunk. The dashed edge denotes training-data support, not data construction repeated at inference time.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Cross-Temporal Pairing"] --> B["Temporal Continuity and<br/>Trajectory Expansion"]
    B -.->|Training pairs and supervision| G["Autoregressive DiT"]
    U["Location, trajectory, text"] --> R["Nearby street-view retrieval"]
    R --> C["Geometric and Semantic<br/>Reference Pathways"]
    R --> D["Virtual Lookahead Sink"]
    C --> G
    D --> G
    H["Previously generated history"] --> G
    U --> G
    G --> O["Current chunk and updated history"]

Key Designs¶

1. Cross-Temporal Pairing: separate location structure from transient objects

The authors collect 1.2M panoramic images of Seoul from a map provider with permission for research use, including GPS coordinates and capture timestamps. After processing, 440K images participate in training; license plates and pedestrians are blurred. Each training sequence uses consecutive street views along a route as targets, with spatially nearby panoramas captured at different times as references. Target views face forward with random horizontal rotations up to 90 degrees, while reference panoramas are rendered into pinhole images aligned with the corresponding target viewing directions. Qwen2.5-VL-72B captions the videos, augmented with camera actions such as straight movement, stopping, and left or right turns.

The constraint is more than generic data augmentation: it removes the shortcut of reproducing every vehicle visible in a reference. With simultaneous captures, buildings and vehicles both match, leaving no reason to distinguish them. Across timestamps, building layouts remain predictive while transient vehicles become unreliable, encouraging reliance on persistent structure. There is no additional dynamic-object separation network or explicit decomposition loss. The attention visualization in Figure 6 qualitatively supports this interpretation but does not establish rigorous causal disentanglement.

2. Temporal Continuity and Trajectory Expansion: turn sparse vehicle captures into usable video supervision

Large gaps between street views create viewpoint jumps that conflict with a pretrained video model's continuous-motion prior. Conventional keyframe conditioning concatenates isolated image latents along the channel dimension, but the underlying 3D VAE compresses every 4 frames into one latent, so a single keyframe does not form a complete temporal group. The Intermittent Freeze-Frame strategy repeats each keyframe 4 consecutive times when training the interpolation model, making it correspond to one latent while preserving continuous motion elsewhere. During interpolation, input keyframes are likewise repeated and encoded, and their latents replace the corresponding positions in the noisy sequence. After decoding, the extra 3 frames per keyframe are discarded, producing smooth training videos for the world model. The freeze segments are therefore an intermediate representation adapted to VAE compression, not periodic pauses required in the final simulation.

Temporal interpolation cannot supply motion types absent from vehicle-mounted captures. The authors additionally render 10K videos from 5 CARLA urban maps covering approximately 431,500 square meters, including pedestrian paths, vehicle paths, and free-camera trajectories that avoid scene collisions. Reference street views are sampled every 10 meters along roads with 8 directions per location. Synthetic references and targets also use different simulated timestamps, retaining the cross-temporal learning constraint; Waymo driving videos add scenario diversity. Depth Anything V3 estimates real street-view depth and camera poses, which GPS metadata then aligns to metric scale, rather than treating approximate coordinates as accurate camera extrinsics.

3. Geometric and Semantic Reference Pathways: distinguish placement from appearance

For each chunk, the model first performs nearest-neighbor location retrieval along the target trajectory, then checks depth-reprojection coverage in the nearest target view to retain up to K references. This avoids conditioning on nearby images that barely observe the relevant region. For each target frame, the geometric pathway selects only the spatially nearest reference, unprojects it into 3D using depth, and forward-splats it into the target view using the relative pose. A single reference avoids conflicts between multiple depth estimates. The warped video is VAE-encoded and concatenated with noisy target latents along channels, indicating where roads and buildings should appear. This pathway is not a complete city reconstruction and cannot recover unseen occluded content through projection alone.

Because reprojection loses details through depth errors and occlusions, the semantic pathway retains original reference images. Each image is independently encoded and patch-embedded, then appended to the DiT latent sequence along time, allowing target tokens to attend to all K references. Reference RoPE positions lie beyond the generation window, separated by a large gap G so these appearance observations are not treated as contiguous video frames. Camera poses for targets, references, and the sink are represented with Plücker ray embeddings, convolutionally encoded, and concatenated into latent channels, giving attention access to both appearance and viewpoint relationships. Geometry supplies alignment while semantics restores details; both can contain outdated vehicles, so cross-temporal pairing remains necessary for learning which information to trust.

4. Virtual Lookahead Sink: move the long-horizon anchor forward with the route

For every chunk, SWM retrieves a street view near the trajectory endpoint as the Virtual Lookahead (VL) Sink, encodes it into one clean latent, and assigns it a RoPE time position after the current target chunk. The model thus sees both previous history and a forward reference from a real database, rather than relying only on generated history with accumulated errors. Unlike a fixed first-frame sink, its spatial location is refreshed and stays relevant after traversing multiple neighborhoods. Its temporal role as a future reference also differs from placing a retrieved image at the beginning of the sequence.

The anchor is not a final frame to reconstruct, need not lie exactly on the user's future trajectory, and does not require copying its vehicles or lighting. Training samples a ground-truth future frame at a random temporal offset, exposing the model to different lookahead distances. Inference replaces it with retrieved street imagery and fixes the lookahead offset to 5. This supplies a constraint for continuing toward a relevant location rather than exact endpoint pixel matching. The cached extraction of the sequence formula in Equation (1) and projection formula in Equation (2) is damaged; this note explains the mechanism from adjacent prose instead of presenting guessed expressions as the authors' equations.

A Worked Example¶

Suppose a user starts at an intersection, requests forward driving, and changes the environment to sunset. This illustrates the pipeline rather than reporting an additional experiment. The TF configuration retrieves up to 5 street views for a 77-frame chunk, projects the nearest image into each target view to constrain road and building positions, and allows attention to inspect all references for facade details. A separate street view near the chunk endpoint becomes the lookahead sink, while the last 5 history latents from the preceding chunk also enter the model.

Text controls sunset lighting, history provides continuity for existing dynamic content, and references primarily constrain persistent structures. For the next chunk, history becomes the newly generated tail, while references and the lookahead sink are refreshed for the new location. The model need not memorize every detail of an entire city, but its database must provide relevant observations along the route. Cross-city generalization is therefore not a purely parametric ability independent of imagery from the new city.

Loss & Training¶

The main configuration uses AdamW with a learning rate of 4.8e-5, a total batch size of 48, and 10K training iterations across 24 NVIDIA H100 GPUs. Teacher Forcing (TF) uses 77-frame chunks, 5 ground-truth history latents, 5 references, and a reference time gap of 50 during training. Inference replaces ground-truth history with generated history, retaining a history-distribution mismatch between training and inference.

The Self-Forcing (SF) variant starts from the TF checkpoint, performs ODE initialization with 1K pairs for 6K steps, and then fine-tunes for 10K iterations. It uses self-generated history in a KV cache, with 3 history latents, 12-frame chunks, and 1 reference. The reported throughput is 15.2 fps on one H100, which is not the speed of the TF configuration; the main text also does not establish full end-to-end retrieval-system latency. Both variants use a VL Sink offset of 5. The main text does not provide a complete independently reproducible loss formulation, so no additional objective is invented here.

Key Experimental Results¶

Main Results¶

Training uses Seoul; Busan-City-Bench and Ann-Arbor-City-Bench, the latter derived from MARS, are unseen during training. Each contains 30 test sequences of 365 frames, approximately 100 meters per sequence. Evaluation permits nearby local street-view references but excludes images belonging to the test sequence itself, measuring cross-city generalization with external references.

Table 1 excerpts the paper's Table 1; every entry follows the order "Busan / Ann Arbor." FID and FVD measure image- and video-feature distribution differences, respectively, with lower values preferred. Higher mPSNR and lower mLPIPS are better; both are evaluated only on static regions after SAM3 segments dynamic objects, avoiding penalties for plausible vehicles or pedestrians that differ from the recorded scene.

Method	FID ↓	FVD ↓	mPSNR ↑	mLPIPS ↓
Aether	141.24 / 132.77	1096.50 / 1214.84	11.10 / 13.03	0.671 / 0.635
DeepVerse	130.32 / 182.95	892.63 / 1524.97	12.20 / 13.43	0.679 / 0.727
Yume1.5	54.82 / 85.62	425.24 / 993.62	12.09 / 14.15	0.667 / 0.623
HY-World1.5	49.63 / 67.02	544.04 / 864.76	11.87 / 14.26	0.588 / 0.575
FantasyWorld	83.51 / 67.72	783.11 / 917.57	10.01 / 11.97	0.654 / 0.592
Lingbot	62.14 / 57.99	717.44 / 1039.50	10.48 / 12.51	0.645 / 0.641
SWM (TF)	28.43 / 56.61	301.76 / 640.17	14.56 / 15.18	0.392 / 0.481
SWM (SF)	32.50 / 43.97	325.87 / 779.94	13.52 / 14.20	0.478 / 0.573

Baselines receive only the inputs each supports and cannot all exploit the same reference interface as SWM. This is therefore not a single-module comparison under identical inputs and resource budgets. TF achieves a Busan FID of 28.43 versus 49.63 for HY-World1.5, the strongest external baseline on that metric. SF achieves the lowest Ann Arbor FID of 43.97, but its static-structure metrics and FVD are worse than TF's; neither configuration should be described as universally superior on every metric.

Ablation Study¶

Table 2 excerpts the paper's Table 2, using Busan-City-Bench and the full configuration corresponding to SWM (TF). TransErr measures translation error between generated camera motion and the target trajectory, with lower values preferred. The main text does not sufficiently specify its normalization or units, so it should not be labeled as meters.

Config	FID ↓	FVD ↓	TransErr ↓	mPSNR ↑	mLPIPS ↓
Full model	28.43	301.76	0.015	14.56	0.392
Without cross-temporal pairing	44.74	487.87	0.123	12.54	0.519
Without synthetic data	27.74	365.24	0.020	13.52	0.427
Without real street-view data	29.82	467.58	0.050	13.99	0.411
Without geometric referencing	33.01	398.74	0.051	12.33	0.525
Without semantic referencing	30.27	326.18	0.022	14.08	0.442
Without any attention sink	33.06	342.81	0.016	14.16	0.406
First-frame sink	32.71	378.92	0.018	14.25	0.388
Retrieved image as first-position sink	32.41	354.61	0.027	14.35	0.379

Key Findings¶

Cross-temporal pairing is particularly important for FID, FVD, and TransErr: removing it increases TransErr from 0.015 to 0.123. However, removing geometric referencing produces worse mPSNR and mLPIPS, so no component is the most important across every metric.
Synthetic data primarily benefits motion and structure: removing it improves FID to 27.74 but worsens FVD to 365.24, showing that image-distribution metrics cannot substitute for world-model evaluation.
The VL Sink improves FID and FVD over both conventional sink variants, but does not achieve the best mLPIPS. Figure 10 also shows long-sequence FID with a 200-frame sliding window, supporting slower degradation. The 2.48 km result in Figure 7 is qualitative and should not be conflated with the two approximately 100-meter quantitative benchmarks.

Highlights & Insights¶

The pairing rule is a distinctive contribution. Cross-temporal references make preserving buildings while changing transient objects a requirement of supervision itself, rather than relying solely on prompts to ignore vehicles.
The future anchor turns the city database into a continuing source of correction. It supplies real observations that generated history cannot provide by itself, rather than merely enlarging the context window.
A VAE's temporal stride can determine whether conditioning works. Intermittent freezing adapts keyframes into temporal units the model can encode correctly, a potentially transferable idea for video conditioning from sparse observations.

Limitations & Future Work¶

The authors explicitly discuss temporal mismatch, limited trajectory coverage, and long-horizon errors, but the main text lacks a dedicated comprehensive limitations section. The following analysis should not all be attributed to author-acknowledged failures.
This note's assessment: the method depends on local street-view coverage, usage permissions, and depth and pose estimation quality. Results in two new cities do not establish reliability without references or under prolonged construction and major neighborhood changes.
This note's assessment: static-region metrics primarily verify layout and appearance, not correct traffic behavior, object interactions, or physical laws. Text-driven scenario control is also demonstrated mainly qualitatively.
The cache does not include the cited appendices, preventing verification of interpolation ablations, per-baseline adaptations, and complete execution settings. Further evaluation should include matched reference inputs, kilometer-scale quantitative tests, and degradation under retrieval failures.

vs Aether / DeepVerse / FantasyWorld: these methods incorporate geometry or 3D prediction into world generation. SWM instead emphasizes external observations from a real-location database, rather than maintaining consistency only within generated content.
vs VMem / spatial memory retrieval: reusing generated history supports consistent revisits, but that history may already be wrong. Street-view references supply a separate source of location constraints at the cost of maintaining an external database.
vs CityGaussian / Block-NeRF: static city reconstruction emphasizes novel views of an existing scene. SWM supports dynamics and text-driven changes under location constraints, without promising the same deterministic geometric precision.
vs fixed attention sinks / Lookahead Anchoring: SWM adapts lookahead anchoring to city navigation by retrieving relevant street views near each chunk endpoint and continuously updating their spatial location.

Rating¶

Novelty: 4/5. Real-city retrieval, cross-temporal supervision, and a dynamic future anchor form a task-specific combination.
Experimental Thoroughness: 4/5. Two unseen cities and several ablations provide meaningful evidence, but input fairness, long-distance quantification, and appendix verification remain limited.
Writing Quality: 4/5. Problems map clearly to designs, although some reproduction details depend on appendices.
Value: 4/5. The work offers a concrete path to location-controlled video world models, with applications still constrained by data coverage and dynamic realism.