Skip to content

Dynamic World Generation Made Efficient

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Area: 3D Vision / Video Generation
Keywords: Dynamic World Generation, Spatial Token Memory, Cross-View Consistency, 3D Gaussian Splatting, Parallel Video Generation

TL;DR

To overcome the severe multi-hour computational bottleneck and cross-view motion drift caused by sequential autoregressive pipelines, DYNWORLD introduces a Spatial Token Memory that attaches estimated 3D spatial coordinates to intermediate latent tokens in transformer-based video generators, enabling parallel multi-view video generation and decoupled 4D Gaussian reconstruction with an order-of-magnitude speedup (from 3+ hours down to 15–30 minutes).

Background & Motivation

Synthesizing large-scale dynamic 3D worlds is a fundamental ambition in computer vision, robotics simulation, and immersive virtual reality. In an ideal dynamic world, an observer should be able to navigate freely through the environment while witnessing physically coherent object actions (such as moving vehicles and walking pedestrians) and detailed environmental textures (such as running water, drifting clouds, and fluttering flames). However, due to the severe scarcity of expansive, ground-truth 4D dynamic datasets, directly training native world-scale 4D generative models remains largely infeasible. Consequently, modern paradigms leverage pre-trained video diffusion or flow-matching models to synthesize multi-view video clips and fuse them into explicit 3D representations such as 3D Gaussian Splatting.

Nevertheless, this approach encounters a critical challenge in multi-view motion consistency. Standard video generators synthesize each camera view independently without intrinsic geometric cross-view constraints. Independent sampling leads to severe physical contradictions across cameras, including distorted trajectories, mismatched speeds, and appearance flickering. To bypass this, prevailing world-generation pipelines (such as DynamicVoyager) rely on sequential autoregressive outpainting: they generate a video from one camera pose, integrate it into an explicit 3D representation, and use that 3D proxy to guide generation for the next viewpoint. This rigid temporal dependency creates an immense computational bottleneck, taking 3 to 6 hours to construct a scene across a few viewpoints while still suffering from drift across non-overlapping distant cameras. Alternatively, score-stitching methods (e.g., SyncSDE) blend denoising trajectories only in overlapping regions, which frequently degrades synthesis quality and introduces visible boundary seams.

The key insight of this paper is that the latent tokens within transformer-based video generators already implicitly encode rich spatial semantics and dynamic information. Rather than waiting for full-frame generation to update an explicit point cloud sequentially, one can directly establish geometric correspondence between latent tokens across camera views in the attention space. Core idea: introduce a Spatial Token Memory that stores intermediate key-value video tokens augmented with estimated 3D world coordinates, re-projects them into novel camera views with Rotary Position Embedding (RoPE) during attention computation, and enables fully parallel, view-consistent video generation followed by decoupled dynamic 3D Gaussian world construction.

Method

Overall Architecture

DYNWORLD operates via three tightly integrated stages: 1. Static World Construction: Given an initial input image, a descriptive text prompt, and a camera path, the system predicts monocular depth and surface normals to instantiate an initial 3D Gaussian scene, and expands it along the path via iterative outpainting and Gaussian optimization to establish a globally consistent static environment. 2. Video Generation with Spatial Token Memory: Multiple camera viewpoints are sampled across the static world. Flow-matching transformer video generators synthesize fixed-viewpoint videos simultaneously across all cameras. A shared Spatial Token Memory maintains latent key-value tokens along with estimated 3D coordinates; by re-projecting these tokens and applying RoPE, cross-view dynamic context is directly exchanged inside attention layers during denoising. 3. Decoupled Dynamic World Construction: Generated multi-view videos are segmented into static backgrounds and moving foregrounds. Backgrounds are represented using 3D Gaussians with constant geometry but time-varying colors, while foreground objects are modeled with fully time-varying dynamic Gaussians, unified through joint optimization into an interactive 4D world.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Image + Path & Prompt"] --> B["Static World Construction<br/>Depth & Normal Estimation + Outpainting + 3D GS Optimization"]
    B --> C["Sample Multi-View Fixed Camera Poses"]
    C --> D["Parallel Video Generation<br/>Flow-Matching DiT with Strict Fixed-Camera Prompts"]
    D <--> E["Spatial Token Memory Update & Sharing<br/>Posterior-Mean Depth + 3D Back-Projection + RoPE Injection"]
    D --> F["Multi-View Consistent Videos"]
    F --> G["Dynamic World Construction<br/>FG/BG Video Decomposition + Static Geometry/Dynamic Color BG + Dynamic FG GS"]
    G --> H["Unified 4D Dynamic 3D World"]

Key Designs

1. Spatial Token Memory and Cross-View Geometric Attention: Parallel Generation via Latent Interaction

Existing systems require sequential view synthesis because novel viewpoints cannot reference intermediate generation states geometrically. DYNWORLD constructs a cross-view Spatial Token Memory \(\mathcal{M} = \{(k^{(\cdot,\tau)}, v^{(\cdot,\tau)}, p^{(\cdot,\tau)}_{\mathrm{3D}})\}_{\tau=0}^\Gamma\) across denoising steps \(\tau\). When evaluating an attention layer for camera \(\Pi^{(i)}\), the model retrieves accumulated key-value tokens and their 3D coordinates \(p^{(\cdot,\tau)}_{\mathrm{3D}}\) from memory \(\mathcal{M}\). The 3D coordinates are projected onto the 2D image plane of view \(\Pi^{(i)}\) using camera projection \(\varphi\): $\(p^{(i,\tau,*)}_{\mathrm{2D}} = \varphi(p^{(\cdot,\tau)}_{\mathrm{3D}}, \Pi^{(i)})\)$ To align with the spatial structure of the current viewpoint, Rotary Position Embedding (RoPE) is applied to the retrieved key tokens using the projected 2D coordinates: $\(k^{(i,\tau,*)} = \mathrm{RoPE}(k^{(\cdot,\tau)}, p^{(i,\tau,*)}_{\mathrm{2D}})\)$ Value tokens \(v^{(\cdot,\tau)}\) are directly preserved without positional encoding. These transformed key-value pairs are concatenated with the current layer's original tokens \([k^{(i,\tau)}, v^{(i,\tau)}]\) along the sequence dimension to perform multi-head attention. This enables video generators across distinct views to directly perceive and synchronize shared 3D dynamic trajectories, even across spatially connected views that do not share direct visual overlap.

2. Posterior-Mean Depth Estimation and Static Anchor Calibration: Accurate Token Geometrization

To back-project key-value tokens into 3D world space, accurate scalar depth values are mandatory. Waiting for fully denoised video frames would ruin parallel execution. DYNWORLD leverages the posterior mean property of linear flow-matching models to analytically approximate the clean video latent \(\hat{z}^{(i,0)}\) directly from the noisy latent \(z^{(i,\tau)}\) at step \(\tau\): $\(\hat{z}^{(i,0)} = \frac{1}{\dot{\sigma}_\tau \alpha_\tau - \sigma_\tau \dot{\alpha}_\tau} \left( \dot{\alpha}_\tau z^{(i,\tau)} - \alpha_\tau u_\theta(z^{(i,\tau)}, \tau) \right)\)$ After decoding \(\hat{z}^{(i,0)}\) into video frames, a pre-trained monocular video depth estimator extracts depth maps \(\hat{D}^{(i)}\). Capitalizing on the smooth evolution of latents across adjacent denoising steps, depth estimation is performed only at three anchor steps (early, intermediate, and late stages) and cached. To resolve monocular scale-and-shift ambiguity across viewpoints, the estimated depth maps are globally anchored to the rasterized depth maps \(D^{(i,\mathrm{st})}\) of the static 3D world via least-squares alignment: $\(\min_{\lambda^{(i)}, \mu^{(i)}} \sum_{f=0}^{F-1} \|\lambda^{(i)} \cdot \hat{D}^{(i)}_f + \mu^{(i)} \cdot \mathbf{1} - D^{(i,\mathrm{st})}\|_F^2\)$ The calibrated depth maps are downsampled to match the token grid \(t \times h \times w\). Each token indexes its scalar depth, allowing the camera-to-world transformation \(\phi\) to compute exact 3D coordinates \(p^{(\cdot,\tau)}_{\mathrm{3D}}\).

3. Decoupled 4D Gaussian Splatting: Disentangling Dynamic Appearance from Structural Geometry

Attempting to fit a unified dynamic Gaussian field across raw video sequences often leads to floating artifacts, background blurring, and geometric collapse. DYNWORLD decouples the scene into static backgrounds and dynamic foregrounds using video segmentation and completion models. For background Gaussians \(G_{\mathrm{bg}} = \{x_{\mathrm{bg}}, R_{\mathrm{bg}}, s_{\mathrm{bg}}, o_{\mathrm{bg}}, c_{\mathrm{bg}}\}\), spatial geometry parameters are held constant across time (initialized from temporally averaged depth and normal maps), whereas color attributes \(c_{\mathrm{bg}} \in [0, 255]^{F \times 3}\) vary across frames to capture lighting changes, rippling water, and moving cloud textures. In contrast, foreground Gaussians \(G_{\mathrm{fg}} = \{x_{\mathrm{fg}}, R_{\mathrm{fg}}, s_{\mathrm{fg}}, o_{\mathrm{fg}}, c_{\mathrm{fg}}\}\) have fully time-varying parameters across all \(F\) frames, tracking non-rigid deformations and motion trajectories. Both sets are jointly optimized via \(L_1\) and SSIM objectives, reconstructing a seamless, artifact-free 4D world.

Key Experimental Results

Main Results

DYNWORLD was evaluated on the WorldScore world-generation benchmark and the VideoScore video quality benchmark, comparing against leading static world creators, dynamic scene models, and video outpainting baselines. Experiments were executed on 8 NVIDIA A40 GPUs using HunyuanVideo (Hunyuan1.5_480p_i2v_distilled) at \(512 \times 512\) resolution with \(F=37\) frames per view.

Table 1: Quantitative world generation results on the WorldScore benchmark (from paper Table 1)

Methods Dynamics Only Overall Average Motion Acc Motion Mag Motion Smooth Camera Ctrl 3D Consist
LucidDreamer 0.00 49.28 0.00 0.00 0.00 88.93 90.37
WonderJourney 0.00 44.63 0.00 0.00 0.00 84.60 80.60
WonderWorld 0.00 50.88 0.00 0.00 0.00 92.98 86.87
Voyager 0.00 54.53 0.00 0.00 0.00 85.95 81.56
T2V-Turbo 27.48 40.20 34.87 40.09 7.48 27.80 38.72
4D-fy 41.72 32.10 22.22 22.88 80.06 69.92 35.47
DYNWORLD (Ours) 48.90 66.11 45.41 24.65 76.64 93.27 85.36

Table 2: Video quality and multi-view consistency evaluation (from paper Table 2)

Methods CLIP (↑) Visual Quality VQ (↑) Temporal Consist TC (↑) Text Align TA (↑) Factual Consist FC (↑)
MOTIA 21.84 1.922 1.553 1.956 1.929
VACE 23.80 2.287 1.852 2.654 1.858
DynamicVoyager 23.78 2.381 1.952 2.656 2.003
DYNWORLD (Ours) 24.55 2.781 2.447 2.997 2.235

Ablation Study

Ablations systematically evaluate the impact of Spatial Token Memory, 3D token projection, depth alignment, background completion, and foreground Gaussian decomposition.

Table 3: Ablation study on core components (from paper Table 3)

Config CLIP (↑) Visual Quality VQ (↑) Temporal Consist TC (↑) Text Align TA (↑) Factual Consist FC (↑) Note
w/o memory 23.12 2.331 2.112 2.658 2.012 Eliminating token memory drops temporal consistency
w/o 3D geo 19.98 2.322 1.884 2.672 1.915 Direct 2D token caching causes severe geometric mismatch
w/o depth align 22.04 2.128 1.667 2.542 2.145 Unaligned scale/shift damages multi-view depth consistency
w/o bg completion 22.22 2.134 1.855 2.340 2.029 Occlusions cause holes in background geometry
w/o fg GS 21.07 2.021 1.636 2.242 1.627 Blending fg into single GS leads to severe motion blur
DYNWORLD (Full) 24.55 2.783 2.442 2.997 2.233 Full model achieves optimal performance across metrics

Key Findings

  1. Dramatic Speedup Over Autoregressive Pipelines: In an 8-view world configuration, DynamicVoyager scales linearly and requires 2 hours and 56 minutes (for 16-frame videos). DYNWORLD completes parallel generation in 18–22 minutes, yielding an approximate \(10\times\) speedup.
  2. 3D Projection is Essential for Cross-View Coherence: Directly caching and injecting 2D tokens without 3D geometric projection (w/o 3D geo) causes the CLIP score to drop from 24.55 to 19.98 and temporal consistency TC to plunge from 2.442 to 1.884, leading to ghosting and spatial artifacts.
  3. Layer-Selective Memory Injection Balances Speed and Fidelity: Injecting the Spatial Token Memory only into the last #40–53 attention layers introduces only a negligible FID increase (from 173.2 to 182.2) while cutting generation time by 10 minutes (a 50% runtime reduction).

Highlights & Insights

  • Latent-Space Geometric Communication: Rather than passing high-overhead explicit 3D geometry between iterations, DYNWORLD anchors transformer key-value tokens in 3D world space and projects them across views via RoPE, establishing direct latent geometric synchronization.
  • Posterior-Mean Depth Resolves Parallel Dependency: By analytically estimating the clean latent at intermediate flow-matching steps and sampling only 3 anchor steps across the denoising schedule, the model acquires accurate depth maps without disrupting parallel generation.
  • Disentangled 4D Gaussian Formulation: Treating the background with static geometry and time-varying color alongside fully dynamic foreground Gaussians provides an elegant formulation that respects physical reality and stabilizes reconstruction.

Limitations & Future Work

  • Reliance on External Monocular Depth Estimators: Token 3D positions depend on external video depth estimation models. Systematic errors in depth predictions on reflective, transparent, or textureless surfaces propagate into cross-view projection errors.
  • Memory Scaling with View Count: As the number of camera viewpoints and frames grows, the accumulation of key-value tokens in Spatial Token Memory increases memory footprint. Future work could introduce token pruning or visibility-based eviction.
  • Long-Horizon Tracking Across Widely Dispersed Views: When dynamic objects exit and re-enter views over long trajectories, maintaining global object identity and trajectory continuity remains challenging.
  • vs DynamicVoyager [50]: DynamicVoyager relies on sequential autoregressive video generation and iterative 3D Gaussian updating, requiring over 3 hours and struggling with drift across non-adjacent cameras; DYNWORLD enables parallel generation across all views, achieving a \(10\times\) acceleration while preserving global motion consistency.
  • vs Score-Stitching Methods (e.g., SyncSDE [29]): SyncSDE synchronizes diffusion across views via score blending in overlapping pixel regions, which perturbs denoising trajectories and produces boundary seams; DYNWORLD leverages attention-based token sharing, preventing artifacts and supporting non-overlapping camera arrangements.
  • vs Static World Generators (WonderWorld / WonderJourney [67, 68]): Prior world models only support static 3D scene traversal; DYNWORLD extends capabilities to dynamic 4D worlds featuring diverse moving objects and living environmental dynamics.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Elegant integration of 3D geometry projection into DiT key-value attention and RoPE, unlocking parallel dynamic world generation.
  • Experimental Thoroughness: ⭐⭐⭐⭐☆ Comprehensive quantitative evaluation across WorldScore and VideoScore benchmarks with informative ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Rigorous narrative flow, clean mathematical derivations, and crystal-clear architecture explanations.
  • Value: ⭐⭐⭐⭐⭐ Highly practical milestone for accelerating 4D dynamic environment synthesis in games, robotics simulation, and spatial computing.