GeoWorld: Providing Full-frame Geometry Features to Facilitate 3D Scene Generation

Conference: ECCV2026
Paper: ECCV Paper
Project: https://peaes.github.io/GeoWorld
Area: 3D Vision
Keywords: single-image scene generation, full-frame geometry features, video diffusion models, geometric alignment, Gaussian Splatting

TL;DR

GeoWorld completes a coarse video along a camera trajectory, then uses full-frame VGGT geometry as conditioning and supervision for a second video stage, reaching 17.28 dB novel-view PSNR on RealEstate10K and 16.64 dB after reconstruction into a 3DGS scene.

Background & Motivation

Generating a scene that can be viewed from new camera positions requires more than visually plausible inpainting. As the camera moves, unseen regions must be invented while the shape and position of shared objects remain compatible across frames. Methods such as ViewCrafter and FlexWorld use video priors to synthesize these views before reconstructing a 3D scene, but temporal coherence alone does not eliminate geometric conflicts.

Single-image depth and camera conditions constrain the visible region without supplying complete structure for every future view. Back-projecting the input into new cameras produces only partial images, leaving the video model to infer disoccluded content. GeoWorld observes that a directly fine-tuned video model can already produce content-complete drafts, although blur and structural distortions remain. Such a draft can therefore serve as material for multi-view geometry extraction instead of being treated as the final output.

Core Idea: first generate a geometric draft covering all target frames, then read it with a multi-view geometry model, select useful features, and supervise a second generation stage against real-video geometry, separating content completion from structural correction.

Method

Overall Architecture

The inputs are a scene image and a specified camera trajectory; the final output is a renderable 3D Gaussian Splatting (3DGS) scene. In Figure 3, geometric condition generation first produces a content-complete condition video. VGGT extracts geometry across the video, and a geometry adaptation module converts these features into conditions for a second, geometry-constrained diffusion model.

The second stage also receives the condition video itself, not just geometry tokens. During training, predicted and ground-truth videos pass through the VGGT aggregator to compute geometric alignment loss; the ground-truth branch is absent at inference. The generated views are subsequently reconstructed into 3DGS, so video output and final scene renderings are distinct evaluation targets.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Image and camera trajectory"] --> Draft["Geometric Condition Generation"]
    Draft --> Geometry["Full-frame VGGT features"]
    Geometry --> Adapter["Geometry Adaptation Module"]
    Adapter --> Diffusion["Geometry-constrained diffusion"]
    Draft -->|Condition video| Diffusion
    Diffusion --> Output["Predicted video and 3DGS"]
    Diffusion --> Loss["Geometric Alignment Loss"]
    Truth["Ground-truth VGGT features"] --> Loss
    Loss -.->|Training supervision only| Diffusion

Key Designs

1. Geometric Condition Generation: turn partial projections into a draft with full-frame geometry

The pipeline first renders the regions explained by the input image along the camera trajectory. During training, the complete video is available: the authors reconstruct 3DGS from all available dataset frames, extract depth at a random starting frame, and apply the back-projection and pairing procedure used by FlexWorld. At inference, these real multi-view observations are unavailable, so MASt3R estimates a point cloud from the single image for back-projection. This distinction concerns training-pair construction, not an extra multi-image requirement at inference.

A Wan2.1-1.3B video model is then fine-tuned to complete the rendered sequence into a condition video. Only single-frame geometry is available at this stage, and it enters through cross-attention as an additional condition. After training, the first model runs over the entire training set to generate conditioning data for the second stage. The two stages are therefore not a single generation pass jointly optimized from scratch.

VGGT jointly reads the condition video to produce geometry features for all frames. Here, full-frame coverage refers to the video sequence, not guaranteed recovery of every unseen surface in the scene. The draft can still be wrong, but unlike the original image alone, it supplies a complete structural hypothesis for the target views.

2. Geometry Adaptation Module: align spatiotemporal grids and suppress unreliable geometry tokens

VGGT features and video latents differ in frame count and spatial resolution. The module first pools temporally and interpolates spatially, then uses an MLP adapter to align feature representations. Compared with feeding all geometry tokens directly into cross-attention, this explicitly addresses the mismatch between the two models' spatiotemporal grids.

Matching dimensions does not make the features reliable. Blurred objects in the condition video can yield ambiguous geometry, while sky or plain walls may provide little structural information. An MLP predictor inspired by Squeeze-and-Excitation integrates global information and predicts a weight for each token. These weights multiply the adapted features, after which the lowest-weight 50% of tokens are discarded.

The remaining tokens enter each video-model layer through single-frame cross-attention: video latents supply queries, geometry tokens supply keys and values, and aligned relative position embeddings preserve positional information. The purpose is not to assert that every low-weight location lacks geometry, but to limit the influence of weak or unreliable conditioning.

3. Geometric Alignment Loss: correct draft geometry using real videos instead of copying it

Geometry conditioning alone can propagate first-stage errors into the second stage. GeoWorld therefore also compares VGGT features of predicted and ground-truth videos, using aggregator outputs instead of decoded depth or point clouds to retain a fuller geometric representation. Conditioning supplies a structural hypothesis; alignment supervision specifies what the generated structure should approach. These are different roles.

Following the mean-squared-error definition in Section 3.2, the geometric term and total objective can be expressed below. Formula extraction in the cache loses symbols, so this is an equivalent reconstruction from the prose; it does not infer the corrupted diffusion prediction target:

\[ \mathcal{L}_{\mathrm{geo}}=\operatorname{MSE}\!\left(G(I),G(I_{\mathrm{pred}})\right),\qquad \mathcal{L}=\mathcal{L}_{\mathrm{diff}}+\lambda\mathcal{L}_{\mathrm{geo}},\quad \lambda=0.2. \]

Here, \(G\) is the VGGT aggregator, while \(I\) and \(I_{\mathrm{pred}}\) denote ground-truth and predicted videos. Real-video geometry provides supervision only during training. Inference uses the learned constraints and conditioning features without access to a target video or replacement of the single-image input with real scene geometry.

A Worked Example

Consider an illustrative indoor image and a camera moving rightward. Projecting its point cloud leaves holes where furniture previously occluded the scene. The first stage completes walls and hidden content; even if some edges are distorted, the video now covers the target views. VGGT extracts cross-frame structure, the adapter suppresses low-weight tokens such as ambiguous edges, and the second stage generates refined views using both the draft video and selected geometry.

During training, structural changes in a generated cabinet across views create a discrepancy between predicted and real-video geometry features, supplying an additional corrective signal. The generated frames are finally reconstructed into 3DGS for scene rendering. This example explains the process and is not a separately reported experimental case.

Loss & Training

Both video stages use Wan2.1-1.3B. Training uses a batch size of 16, learning rate of \(5\times10^{-5}\), input resolution of \(192\times336\), and 17-frame clips. The first stage runs for 7000 iterations and the second for 2000 iterations. The geometry adaptation module is trained together with the second-stage diffusion model on 8 NVIDIA A100 GPUs.

Training pairs come from DL3DV using FlexWorld's construction procedure. After sampling two epochs, the authors retain the top 25% of cases with the smallest average camera translation and rotation, yielding approximately 5000 video pairs. This improves partial-view quality but also favors relatively mild camera motion in the training distribution.

The 17-frame, low-resolution training setup must be distinguished from Table 3 inference at \(576\times1024\) with 49 frames. The text does not detail every implementation step connecting these settings; it does not justify inventing a sampler, inference step count, or resolution-scaling procedure.

Key Experimental Results

Main Results

Testing uses RealEstate10K (RE10K) and Tanks and Temples (Tanks), with 100 randomly selected clips per dataset and 49 frames per clip. Table 1 evaluates generated novel-view videos. Table 2 reconstructs the generated frames into 3DGS and evaluates images rendered at the corresponding camera poses. The selection below combines representative metrics only: higher PSNR and lower LPIPS are better.

Task and source table	Dataset	Method	PSNR / dB	LPIPS
Novel view synthesis, Table 1	RE10K	Hunyuan-Voyager	14.85	0.4357
Novel view synthesis, Table 1	RE10K	GeoWorld	17.28	0.3297
Novel view synthesis, Table 1	Tanks	See3D	13.00	0.5400
Novel view synthesis, Table 1	Tanks	GeoWorld	14.99	0.4556
3DGS scene rendering, Table 2	RE10K	See3D	14.67	0.4413
3DGS scene rendering, Table 2	RE10K	GeoWorld	16.64	0.4284
3DGS scene rendering, Table 2	Tanks	See3D	13.14	0.5420
3DGS scene rendering, Table 2	Tanks	GeoWorld	15.00	0.5058

RE10K video PSNR improves by 2.43 dB over Hunyuan-Voyager, while RE10K 3DGS rendering PSNR improves by 1.97 dB over See3D. These gains should not be merged into a single metric because they concern different output levels.

Ablation Study

Table 4 incrementally adds geometric alignment and geometry adaptation on RE10K. Parenthesized gains in the original table are relative to direct embedding, not the preceding row.

Config (original Table 4)	PSNR / dB	LPIPS	PSNR gain over direct embedding
Direct geometry embedding through cross-attention	16.77	0.3381	0.00
Add geometric alignment loss	16.96	0.3284	0.19
Further add resizing in geometry adaptation	17.17	0.3286	0.40
Further add global weighting	17.28	0.3292	0.51

Two details matter: the final LPIPS in Table 4 is 0.3292, whereas Table 1 reports 0.3297 for the full model, without explaining the discrepancy. Also, adaptation improves PSNR further but does not improve LPIPS at every step; the lowest LPIPS occurs in the alignment-loss row.

Table 5 compares discard ratios of 50%, 30%, and 70%, with PSNR values of 17.28, 17.17, and 17.22 dB and SSIM values of 0.6193, 0.6173, and 0.6188, respectively. Discarding 50% performs best in this setting, but the small margins do not establish a universally optimal ratio.

Key Findings

Table 3 measures inference on one A100 80 GB GPU. GeoWorld's reported training and inference times include both stages.

Method (original Table 3)	Parameters	Inference size (height x width x frames)	Inference time	Inference memory
ViewCrafter	2.6B	576 x 1024 x 25	120 s	24 GB
FlexWorld	5.0B	576 x 1024 x 49	201 s	28 GB
Hunyuan-Voyager	12.8B	512 x 768 x 49	1110 s	48 GB
GeoWorld	3.9B	576 x 1024 x 49	148 s	31 GB

The reported 7.5-fold speedup over Hunyuan-Voyager follows from 1110/148, but resolutions differ, so this is not a strictly resolution-matched throughput comparison. GeoWorld is not the smallest or lowest-memory method in the table: ViewCrafter has fewer parameters, and FlexWorld uses less memory.

In Figure 11, first-stage training at 3000, 7000, and 11000 steps gives FID values of 32.17, 29.55, and 29.66. This supports the claim that longer first-stage training does not replace geometric refinement, but it is not a complete compute-matched ablation. In Table 1, FlexWorld's RE10K FID/FVD of 30.56/270.4 outperform GeoWorld's 31.00/311.7, so GeoWorld does not lead on every video metric.

Highlights & Insights

• Treating a coarse generation as a geometry source lets a single-image task exploit a multi-view geometry model. The important step is acknowledging draft errors and addressing them through feature selection and real-video supervision.
• Geometry injection and geometry supervision serve distinct purposes. The former exposes a cross-frame structural hypothesis; the latter discourages simple replication of draft defects.
• Feature selection addresses both unreliable regions and low-information regions. Figure 13 discusses blur as well as sky and plain walls, so the weights should not be interpreted as a geometry-correctness classifier.

Limitations & Future Work

• Section 4.5 acknowledges that an undertrained condition generator can supply insufficient or incorrect geometry. Global weighting mitigates this problem without guaranteeing reliable VGGT features for all generated content.
• The paper has no dedicated limitations section. The following are reader assessments: results on two datasets with 49-frame clips do not establish consistency under long-range exploration, dynamic scenes, or severe occlusion.
• Training retains only the lowest-motion 25% of samples, and geometric initialization differs between training and inference. Robustness to large camera motion and single-image estimation errors requires stratified evaluation.
• Geometry-feature distances and learned metrics such as MEt3R do not replace error against measured 3D geometry. Useful extensions include ground-truth geometry benchmarks, long-trajectory loop tests, and cross-validation with different geometry encoders.
• The reported 31 GB inference memory and 148 s runtime remain deployment constraints. Methods use different resolutions and frame counts, so efficiency claims must retain the conditions of Table 3.

Related Work & Insights

• Compared with ViewCrafter and FlexWorld: all generate views before reconstructing scenes. GeoWorld expands geometric guidance from single-frame or partial-render information to full-frame features extracted from a draft video, rather than merely increasing video-model capacity.
• Relationship to VGGT and MASt3R: VGGT supplies multi-view geometry features and the training supervision space. MASt3R supports inference initialization and the image-matching evaluation in Figure 7; their roles are not interchangeable.
• Compared with Hunyuan-Voyager: GeoWorld achieves higher fidelity with a smaller two-stage system, but this short-video evaluation does not establish a complete replacement for the other method's long-range exploration capabilities.
• Transferable direction: camera-controlled video generation could combine draft videos, multi-view features, reliability filtering, and real-video feature supervision, while separately testing whether gains transfer to the final 3D representation rather than video appearance alone.

Rating

• Novelty: 4/5. Combining coarse generation with full-frame geometric constraints is more systematic than depth conditioning alone.
• Experimental Thoroughness: 4/5. Evaluations cover video, 3DGS, components, and efficiency, but lack long trajectories and strict budget matching.
• Writing Quality: 4/5. The design is clear, although the cross-table LPIPS discrepancy and some implementation details remain unexplained.
• Value: 4/5. Provides a concrete, reusable route for smaller video models to exploit geometry foundation models.