OctWorld: Long-Range World-Consistent Video Generation with Octree-based 3D Mapping¶

Conference: ECCV 2026
Paper: ECCV Official
Area: Video Generation
Keywords: Video Generation, 3D World Model, Octree Mapping, TSDF Fusion, Long-Range Geometric Consistency

TL;DR¶

Building upon a chunk-wise autoregressive video diffusion model coupled with a dynamic octree-based TSDF spatial memory (OctMap), OctWorld synthesizes high-fidelity, world-consistent RGB-D videos across long camera trajectories and wide viewing angles, maintaining rigorous geometric and photometric coherence when revisiting explored regions.

Background & Motivation¶

World generation aims to synthesize explorable, high-fidelity, and spatially coherent visual environments directly from generative video foundation models driven by single images and camera trajectories. Recent advances in video diffusion models have incorporated explicit 3D predictions, such as depth or point cloud coordinates, into RGB synthesis to instill geometric awareness and improve inter-frame spatial consistency. However, these geometry-aware approaches primarily focus on short-term geometric coherence within a single generation chunk. Synthesizing expansive scenes across extended camera paths and wide viewing angles remains an open and formidable challenge.

In long-horizon exploration, the fundamental tension lies in the absence of an extensible, globally fused 3D memory that remains geometrically stable over time. Standard chunk-wise autoregressive models suffer from catastrophic forgetting of previously synthesized areas; when the camera revisits earlier viewpoints after a long trajectory loop, severe structural distortion, visual flickering, and scene drift inevitably emerge. Existing heuristic memory mechanisms—such as accumulating raw point cloud caches—fail to mitigate depth estimation noise and occlusions, leading to unbounded memory footprint and fragmented geometry. Conversely, classical SLAM-based volumetric integration (e.g., Truncated Signed Distance Function, TSDF) assumes bounded spaces and fixed metric scales; when applied naively to open-world video generation, a fine uniform grid triggers GPU out-of-memory errors, whereas a coarse grid blurs away essential foreground details.

To resolve this conflict, this work couples chunk-wise autoregressive video diffusion with an extensible, multi-resolution volumetric fusion system. Core idea: construct an extensible, dynamically subdivided sparse octree TSDF spatial memory (OctMap) that adaptively allocates voxel resolutions based on observation depth, balancing fine geometric fidelity and memory efficiency while rendering explicit mesh-based spatial priors to guide subsequent chunk generation.

Method¶

Overall Architecture¶

OctWorld takes an initial RGB image $I_0$ and a user-specified long camera trajectory $\mathcal{T}$ as input, splitting $\mathcal{T}$ into sequential chunks of 36 frames each. The system iterates through geometric fusion and autoregressive video generation chunk by chunk. The core framework consists of two complementary components: the dynamic octree-based spatial memory (OctMap) that progressively integrates RGB-D predictions into a unified 3D volume, and an RGB-D video diffusion backbone that incorporates rendered spatial memory conditions, history context packing, and explicit camera control.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Initial Image I0 and Camera Trajectory T"] --> B["Dynamic Octree Spatial Memory Construction<br/>Adaptive Node Allocation & TSDF Fusion"]
    B --> C["Mesh Extraction & Target View Projection<br/>Render RGB-D Priors & Visibility Masks"]
    C --> D["Chunk-wise RGB-D Autoregressive Diffusion<br/>Multimodal Latent Denoising & Frame Packing"]
    D --> E["Cross-Chunk Depth Normalization & Noise Augmentation<br/>Scale Consistency & Robust Error Correction"]
    E --> F["Output: Long-Range World-Consistent Video"]

Key Designs¶

1. Dynamic Octree Spatial Memory Construction: Adaptive Resolution Breaking the Voxel Memory Bottleneck

To overcome the unbounded growth and noisy fragmentation of point cloud caches, as well as the resolution-scalability dilemma of uniform TSDF grids in open-world scenes, the paper introduces OctMap, an adaptive multi-resolution sparse octree. Starting from a canonical bounding box of size $L$, an octree cell at depth $d$ has edge length $\ell_d = L / 2^d$. Each leaf node stores TSDF values, RGB colors, and integration weights at its eight corner vertices. For each incoming RGB-D frame $(I, D)$, the algorithm determines the required metric voxel size from pixel depth: $$s(u, v, z) = \frac{z}{\sqrt{f_x \cdot f_y}}$$ The target octree depth $d$ is assigned as the finest resolution satisfying $\ell_d \le s < \ell_{d-1}$. Nearby geometry receives fine, sub-divided leaf cells to preserve intricate boundaries, whereas distant regions remain represented by coarse cells to prevent memory bloat. Along each back-projected ray, sample points within the truncation margin $[z-\tau_d, z+\tau_d]$ allocate and split nodes on demand, followed by standard weighted running-average TSDF and color updates.

2. Mesh Extraction & Target View Projection: Grounding Generative Diffusion with Explicit 3D Geometry

Before generating each video chunk, the system runs parallel Marching Cubes over the octree leaf cubes to extract an isosurface triangle mesh, merging duplicate vertices at shared cell corners. This global mesh is then rendered onto the target viewpoints of the upcoming chunk, yielding projected RGB images, projected depth maps, and binary visibility masks that delineate previously reconstructed surfaces from newly exposed regions. Borrowing from the VideoPainter control architecture, these spatial memory projections are injected into the video diffusion backbone via duplicated DiT control blocks. This mechanism explicitly anchors generation to historical observations in visible regions while directing the generative model to inpaint occluded or unobserved areas plausibly.

3. Chunk-wise RGB-D Autoregressive Diffusion: Multimodal Latent Space with History Packing

The generative backbone builds upon the pretrained FramePack architecture (based on the HunyuanVideo DiT backbone). To endow the model with geometric synthesis capabilities, the patch embedding and output projection layers are widened to handle concatenated RGB and depth latents. Because future scene extents are unknown at inference time, depth is dynamically normalized using the maximum depth of current history frames, linearly scaled into $[0, 1]$ grayscale representations compatible with the VAE. To retain long-range temporal context without exceeding computational limits, previous RGB-D frames are grouped and progressively compressed along the temporal dimension into fixed-length latent sequences, while Plücker ray embeddings pass through a convolutional adapter to enforce strict camera trajectory adherence.

4. Cross-Chunk Noise Augmentation: Bridging the Train-Inference Drift Gap

In prolonged autoregressive rollouts, minor generation artifacts and geometric inaccuracies from earlier chunks inevitably accumulate and degrade future synthesis. To build error tolerance, the training pipeline applies a noise-conditioning strategy to the historical context and memory projections. A randomly sampled Gaussian noise level $\alpha$ is injected into the condition latents and simultaneously mapped through a dedicated projection layer to modulate the Transformer timestep embeddings. This training scheme penalizes over-reliance on pristine conditioning and encourages the diffusion network to correct and smooth imperfect historical representations during multi-chunk autoregressive rollouts.

Loss & Training¶

The framework is optimized end-to-end using the Flow-Matching objective, encompassing 15B total parameters. Training is executed in two stages on 32 NVIDIA H100 GPUs with an effective batch size of 64: first at a reduced temporal resolution of $16 \times 480 \times 832$ for 80K steps, followed by full-resolution fine-tuning at $36 \times 480 \times 832$ for 30K steps. The model is optimized using AdamW with a constant learning rate of $2 \times 10^{-5}$.

Key Experimental Results¶

Main Results¶

Evaluation on the photorealistic split of the WorldScore benchmark (200 scenes with camera motion magnitude $\ge 3$, generating 108 frames across 3 chunks per scene):

Paper Table 1: WorldScore evaluation of different models

Method	WorldScore Static↑	3D Consist.↑	Photo Consist.↑	Style Consist.↑	Camera Ctrl.↑
AETHER	46.59	65.65	50.29	83.49	37.65
Voyager	49.93	66.35	43.38	67.89	46.60
VMem	42.40	61.83	41.21	77.72	31.56
Gen3C	45.48	70.52	68.46	78.06	12.57
OctWorld (Ours)	50.31	71.67	77.01	78.17	39.12

Evaluation of spatial memory maintenance on the RealEstate10K test set under reversed-trajectory loops (120 frames forward + 120 frames retraced backward):

Paper Table 2: Evaluation of spatial memory maintenance on Re10K test set

Method	LPIPS↓	Chamfer Distance (CD)↓
VMem	0.4010	0.0620
Gen3C	0.1933	0.0457
OctWorld (Ours)	0.1633	0.0345

Ablation Study¶

Evaluation of reprojection rendering quality across 3D memory representations on 100 scenes from SpatialVID:

Paper Table 5: Reprojection rendering quality experiment

3D Memory	Voxel Size	PSNR↑	SSIM↑	LPIPS↓	Peak Memory (GB)↓	Time (s)↓
PointCloud	-	19.89	0.57	0.38	6.11	18.58
TSDF	0.007	20.47	0.61	0.44	1.37	1.42
TSDF	0.004	21.32	0.65	0.39	1.97	3.15
TSDF	0.002	22.08	0.69	0.33	9.66	16.97
TSDF	0.001	-	-	-	OOM	-
OctMap (Ours)	Adaptive	23.47	0.74	0.27	7.26	17.94

Key Findings¶

OctMap achieves a PSNR of 23.47 dB, exceeding the finest viable uniform TSDF (22.08 dB at voxel size 0.002) by 1.39 dB and PointCloud fusion (19.89 dB) by 3.58 dB, while maintaining a peak memory footprint of only 7.26 GB and avoiding the OOM breakdown observed with dense grids.
In trajectory retracing experiments on Re10K, OctWorld lowers Chamfer Distance to 0.0345 (a 24.5% reduction relative to Gen3C's 0.0457) and achieves an LPIPS of 0.1633, confirming that explicit surface fusion prevents hallucinated geometry when revisiting previous views.
The noise augmentation ablation (Table 4b) demonstrates that setting noise level to 3 provides the optimal balance between high-frequency detail and long-term consistency, improving 3D Consistency on WorldScore from 67.18 (at noise level 0) to 71.67.

Highlights & Insights¶

Adaptive Voxel Subdivision by Depth: By tying octree node subdivision to pixel projection footprint, OctMap concentrates representational capacity on high-resolution foreground details while allocating coarse voxels to expansive backgrounds.
Continuous Surface Prior for Video Diffusion: Extracting an explicit mesh via Marching Cubes and reprojecting RGB-D observations with visibility masks gives the diffusion backbone rigid physical anchors, eliminating the geometric distortion common in warp-only or point-based frameworks.
Robustness to Long-Horizon Error Accumulation: Combining dynamic max-depth normalization and noise augmentation ensures smooth cross-chunk transitions, providing a blueprint for scalable, interactive 3D world models.

Limitations & Future Work¶

Restricted to Static Environments: The underlying TSDF integration relies on static multi-view consistency assumptions; dynamic objects, moving shadows, and non-rigid deformations are currently unmodeled and filtered during preprocessing.
Computational Latency of Explicit Mesh Pipelines: Processing each chunk requires octree updates, Marching Cubes isosurface extraction, and multi-view rendering, requiring approximately 18 seconds per chunk and precluding real-time interactive exploration.
Future Directions: Integrating differentiable scene representations such as 3D Gaussian Splatting with octree hierarchies to eliminate polygon extraction overhead, and introducing decoupled dynamic-static representations.

vs Gen3C: Gen3C aggregates multi-view observations as a point cloud cache, which accumulates monocular depth estimation noise and boundary artifacts; OctWorld utilizes continuous TSDF zero-crossings to fuse surfaces smoothly and resolve noisy disparities.
vs Voyager: Voyager applies warp-conditioned RGB-D diffusion primarily within a single chunk (49 frames) without a persistent global 3D memory; OctWorld scales to arbitrary multi-chunk trajectories with loop closure support.
vs VMem: VMem relies on view memory retrieval in latent space, which struggles to maintain rigid geometry across dramatic camera turns; OctWorld anchors generation in true 3D spatial coordinates.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Seamlessly integrates dynamic octree TSDF spatial mapping with chunk-wise RGB-D video diffusion for persistent world generation.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extensive evaluations on WorldScore, Re10K retraced trajectories, 433-frame long sequences, and isolated 3D mapping ablations.
Writing Quality: ⭐⭐⭐⭐⭐ Clear methodology, coherent mathematical definitions, and detailed architectural descriptions.
Value: ⭐⭐⭐⭐⭐ Establishes a practical, highly scalable paradigm for preventing geometric drift and hallucination in long-range video world models.