Skip to content

OneWorld: Taming Scene Generation with 3D Unified Representation Autoencoder

Conference: ECCV 2026
Paper: ECCV Official Link
Code: https://github.com/Onishi-Lab/OneWorld ⚠️ Refer to the original paper
Area: 3D Vision
Keywords: 3D Scene Generation, 3D Unified Representation, Representation Autoencoder, Cross-View Correspondence, Manifold-Drift Forcing

TL;DR

OneWorld discards conventional 2D/video latent generation paradigms by constructing a 3D Unified Representation Autoencoder (3D-URAE) that injects appearance and distills semantics, enabling direct diffusion within a geometry-aware 3D latent space while enforcing cross-view correspondence and mitigating exposure bias via manifold-drift forcing.

Background & Motivation

Diffusion-based 3D scene generation aims to synthesize photorealistic and interactive 3D digital worlds conditioned on a single reference image or text prompt, serving as a critical infrastructure for robotics simulation, virtual reality, and gaming asset production. Early approaches predominantly rely on Score Distillation Sampling (SDS) to optimize neural radiance fields (NeRF) or 3D Gaussian Splatting (3DGS) per scene, which incurs prohibitive computational overhead and frequently suffers from multi-face artifacts and semantic disintegration. To improve generation throughput, recent methods either synthesize multi-view images/videos followed by feed-forward 3D reconstruction, or diffuse within frozen 2D image/video VAE latent spaces equipped with latent-to-3DGS decoders (such as FlashWorld and Prometheus). However, 2D latent spaces lack explicit cross-view 3D geometric coupling, rendering them vulnerable to multi-view identity drift and geometric tearing across large baseline shifts.

Performing generative modeling directly inside explicit 3D representation spaces introduces severe representational dilemmas. Although feed-forward 3D foundation models such as Ο€3 and Dust3R have demonstrated impressive geometric reconstruction capabilities, their latent tokens are heavily geometry-dominated and discard fine-grained appearance details, making them incapable of rendering high-fidelity, visually consistent textures. Concurrent attempts such as Gen3R compress 3D representations to align with video VAE latents, which inevitably forces separate decoupled generation of point cloud geometry and video appearance, thereby compromising 3D representation capacity and failing under large camera viewpoint variations. Furthermore, multi-step denoising in high-dimensional 3D latent manifolds suffers acutely from train–inference exposure bias: small prediction discrepancies accumulate across sampling timesteps, and these errors are cascade-amplified by tightly coupled multi-view 3D constraints, ultimately causing the 3D decoder to collapse at inference time.

To overcome these barriers, this paper explores performing generative diffusion directly within an augmented 3D foundation representation space where geometry and appearance are jointly modeled. Core idea: build a 3D Unified Representation Autoencoder (3D-URAE) that injects fine-grained appearance tokens and distills foundation semantic structure into unified 3D latents, deploy conditional diffusion within this geometry-aware space under a token-level Cross-View Correspondence (CVC) loss, and employ Manifold-Drift Forcing (MDF) to train the 3DGS decoder on drifted off-manifold latents, achieving robust and geometrically coherent single-view 3D scene generation.

Method

Overall Architecture

The OneWorld framework operates across three tightly integrated stages: first, a feed-forward 3D foundation model (Ο€3) is transformed into a 3D Unified Representation Autoencoder (3D-URAE) by incorporating an appearance injection branch for fine-grained textures and a semantic distillation branch for latent manifold regularization, producing unified 3D tokens \(V\) encoding geometry, appearance, and semantics; second, a conditional Diffusion Transformer (DiT) is trained to denoise target-view 3D tokens conditioned on an input reference view, regularized by a token-level Cross-View Correspondence (CVC) loss that preserves structural correspondence patterns; finally, to resolve sampling drift from train–inference exposure bias, Manifold-Drift Forcing (MDF) fine-tunes the 3DGS prediction heads on linear interpolations of diffusion-drifted and ground-truth 3D tokens, establishing a resilient decoding manifold.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    A["Single Input Image + Camera Poses"] --> B["3D Unified Representation Autoencoder<br/>Appearance Injection + Semantic Distillation"]
    B --> C["Cross-View Correspondence Diffusion Denoising<br/>DiT Denoising in Unified 3D Latent Space"]
    C --> D["Manifold-Drift Forcing Decoding<br/>Mixed Drifted Latents Fine-Tuning 3DGS Heads"]
    D --> E["High-Fidelity 3DGS Scene & Novel View Renderings"]

Key Designs

1. 3D Unified Representation Autoencoder: Decoupled appearance injection and semantic manifold distillation Pretrained 3D foundation reconstructors (such as Ο€3) typically employ vision foundation models like DINOv2 as tokenizers to extract patchified semantic tokens \(Z \in \mathbb{R}^{N \times h \times w \times C}\), whose excessive semantic abstraction discards high-frequency visual textures and yields blurry 3DGS reconstructions. To compensate for this information loss, 3D-URAE introduces a lightweight convolutional appearance encoder \(E_{\text{app}}\) to extract appearance-preserving tokens \(Z_{\text{app}}\) aligned in spatial resolution and channel dimensions with \(Z\). Concatenating them along the channel dimension as \([Z \parallel Z_{\text{app}}]\) and feeding them alongside camera parameters \(T\) into the geometry Transformer \(E_V\) produces unified geometry-appearance tokens \(V \in \mathbb{R}^{N \times h_v \times w_v \times C_v}\). In parallel, pure rendering supervision yields geometry tokens that lack structured semantic organization, complicating generative diffusion. To address this, a semantic distillation branch maps \(Z\) to geometry space via adapter \(A_{\text{sem}}\) and aligns them using a marginal cosine similarity loss \(\mathcal{L}_{\text{mcos}}\) and a marginal distance matrix similarity loss \(\mathcal{L}_{\text{mdms}}\): $\(\mathcal{L}_{\text{sem}} = \mathcal{L}_{\text{mcos}} + \lambda_{\text{mdms}}\mathcal{L}_{\text{mdms}}\)$ Together with a differentiable 3DGS rendering loss \(\mathcal{L}_{\text{render}}\), 3D-URAE ensures that individual tokens unify explicit 3D geometry, fine-grained appearance, and compact semantic layout.

2. Cross-view correspondence diffusion denoising: Explicit multi-view structural alignment Within the unified 3D token space derived from 3D-URAE, a conditional diffusion model denoises target-view tokens \(x_0\) conditioned on single-view reference tokens. Standard Mean Squared Error objectives (such as the \(v\)-prediction loss \(\mathcal{L}_v = \mathbb{E}[\|\hat{v}_\theta - v\|_2^2]\)) measure token-wise distance in isolation but fail to enforce structural correspondence across camera viewpoints, frequently causing floating artifacts or geometric discontinuities in synthesized novel views. To explicitly maintain geometric correspondence, OneWorld introduces the Cross-View Correspondence (CVC) loss. For each target-view token \(p\), cosine similarity is evaluated against all conditioning tokens \(c_{0,q}^{(\text{cond})}\), selecting the most confident nearest neighbor \(q_p^\star\) only if the maximum similarity exceeds threshold \(\tau = 0.9\). Then, a cross-entropy loss enforces the predicted tokens \(\hat{x}_{0,p}\) to focus its induced softmax probability distribution on the ground-truth target match \(q_p^\star\): $\(\mathcal{L}_{\text{cvc}} = \frac{1}{\sum_p \mathbf{1}_p} \sum_{p=1}^{N_p} \mathbf{1}_p \cdot \left( -\log \frac{\exp(\cos(\hat{x}_{0,p}, c_{0,q_p^\star}^{(\text{cond})}) / \mathcal{T})}{\sum_{q=1}^{N_p} \exp(\cos(\hat{x}_{0,p}, c_{0,q}^{(\text{cond})}) / \mathcal{T})} \right)\)$ where \(\mathcal{T}\) represents temperature. The total diffusion loss \(\mathcal{L}_{\text{diff}} = \mathcal{L}_v + \lambda_{\text{cvc}} \mathcal{L}_{\text{cvc}}\) compels the denoiser to preserve cross-view topological coherence throughout the generative trajectory.

3. Manifold-drift forcing decoding: Interpolated latent training to counter exposure bias During multi-step sampling at inference time, diffusion models inevitably feed their own intermediate predictions back into conditioning, deviating from the ground-truth perturbed distribution seen in training. This sampling drift pulls intermediate tokens off the 3D-URAE manifold, and the resulting distortion is aggravated in 3D scene generation because cross-view geometric constraints couple all viewpoints simultaneously. As a consequence, 3DGS heads trained strictly on pristine ground-truth latents produce geometric tearing and visual ghosting. Manifold-Drift Forcing (MDF) addresses this exposure bias by proactively exposing the decoder to drifted latents. Specifically, freezing \(E_V\), the diffusion model generates predicted clean latents \(\hat{V}_0^{(t)}\) at random intermediate steps \(t \sim \mathcal{U}([T_1, T_2])\), which are blended with ground-truth latents \(V\) using random weights: $\(\tilde{V} = \alpha \hat{V}_0^{(t)} + (1 - \alpha) V, \quad t \sim \mathcal{U}([T_1, T_2]), \ \alpha \sim \mathcal{U}([0, 1])\)$ The perturbed latents \(\tilde{V}\) are fed directly into 3D prediction heads \(D_V\) to predict 3DGS parameters \((\tilde{G}, \tilde{D})\) and supervised via multi-view differentiable rendering \(\tilde{\mathcal{L}}_{\text{render}}\). By training over this continuous off-manifold envelope, the decoder develops substantial tolerance against sampling drift, stabilizing inference-time rendering.

Loss & Training

The training pipeline is structured across three consecutive stages: 1. 3D-URAE Pre-training: Trained on a 1:1 mixture of RealEstate10K and DL3DV-10K initialized from Ο€3 checkpoints. For each scene, 8 input views and 4 novel supervision views are sampled at \(224 \times 448\) resolution. The overall objective \(\mathcal{L}_{\text{URAE}} = \mathcal{L}_{\text{render}} + \lambda_{\text{sem}} \mathcal{L}_{\text{sem}}\) is optimized with \(\lambda_{\text{lpips}} = 0.05, \lambda_{\text{sem}} = 0.1, \lambda_{\text{mdms}} = 1.0\), and margins \(m_1 = m_2 = 0.05\). Optimized using AdamW with learning rate decaying from \(2 \times 10^{-4}\) to \(2 \times 10^{-5}\) across 30K steps with global batch size 64 on 32 NVIDIA A100 GPUs. 2. Diffusion Training: Conditional DiT initialized from Wan-2.1-T2V-1.3B trained with \(v\)-prediction objective plus CVC loss (\(\tau = 0.9, \lambda_{\text{cvc}} = 0.2\)). Classifier-free guidance drops text prompts with a 0.5 rate. Trained for 100K steps with batch size 256, learning rate decaying from \(1 \times 10^{-4}\) to \(1 \times 10^{-5}\), and EMA decay 0.9995. 3. MDF Fine-tuning: Timestep window set to \(T_1 = 10, T_2 = 20\). With \(E_V\) frozen, only the decoder heads \(D_V\) are updated using drifted latents from DiT, trained with learning rate \(2 \times 10^{-5}\) and batch size 256 for 10K steps.

Key Experimental Results

Main Results

Under the calibrated 1-view Novel View Synthesis (NVS) protocol, models are evaluated on RealEstate10K and DL3DV-10K test sets (500 scenes each) across standard image metrics (PSNR, SSIM, LPIPS) and VBench video generative metrics (Subject Consistency I2V Subj., Background Consistency I2V BG, and Imaging Quality I.Q.).

Dataset Metric OneWorld (Ours) Second Best Runner-up Baseline Gain
RealEstate10K PSNR ↑ 21.57 20.18 FlashWorld +1.39 dB
RealEstate10K SSIM ↑ 0.735 0.724 FlashWorld +0.011
RealEstate10K LPIPS ↓ 0.231 0.256 FlashWorld -0.025 (lower is better)
RealEstate10K I2V Subj. ↑ 0.993 0.994 Gen3R -0.001 (comparable)
RealEstate10K I2V BG ↑ 0.995 0.994 FlashWorld / Gen3R +0.001
RealEstate10K I.Q. ↑ 0.604 0.593 Gen3R +0.011
DL3DV-10K PSNR ↑ 17.19 16.02 FlashWorld +1.17 dB
DL3DV-10K SSIM ↑ 0.589 0.566 FlashWorld +0.023
DL3DV-10K LPIPS ↓ 0.418 0.451 FlashWorld -0.033 (lower is better)
DL3DV-10K I2V Subj. ↑ 0.966 0.964 FlashWorld / Gen3R +0.002
DL3DV-10K I2V BG ↑ 0.973 0.970 Gen3R +0.003
DL3DV-10K I.Q. ↑ 0.556 0.543 Gen3R +0.013

Under the reference-free WorldScore benchmark: - WorldScore-Indoor: OneWorld achieves the highest 3D Consistency score of 84.98 (vs. FlashWorld 83.57) and the best Photometric Consistency of 81.67 (vs. FlashWorld 80.19), with Style Consistency at 76.74. - DL3DV (Outdoor): OneWorld ranks first in 3D Consistency at 78.21 (vs. FlashWorld 76.74) and Photometric Consistency at 74.09 (vs. FlashWorld 72.76), with Style Consistency at 70.62 and Subjective Quality at 45.98.

Ablation Study

Ablations systematically evaluate the 3D-URAE architectural components and the generation modules (CVC and MDF).

Component Category Configuration / Setting PSNR ↑ SSIM ↑ LPIPS ↓ Note
3D-URAE Reconstruction Full 3D-URAE (with App. Inject) 28.19 0.932 0.102 Full 3D representation reconstruction
w/o Appearance Injection Pure DINOv2 semantic features 21.14 0.669 0.293 Lacks fine textures, PSNR drops by 7.05 dB
OneWorld Generative Model Full generative model (with Sem. Distill) 21.57 0.735 0.231 Full generation performance
w/o Semantic Distillation Geometry tokens without DINOv2 distillation 17.45 0.644 0.352 Unstructured manifold, PSNR drops by 4.12 dB
OneWorld (Full) RealEstate10K 1-view NVS 21.57 0.735 0.231 Full model (I.Q. 0.604)
w/o Cross-View Correspondence Omitting CVC matching loss 19.10 0.682 0.284 Unconstrained cross-view tokens, PSNR drops 2.47 dB
w/o Manifold-Drift Forcing Decoder trained only on GT latents 20.59 0.714 0.256 Vulnerable to sampling drift, PSNR drops 0.98 dB

Key Findings

  • Appearance injection is the foundation of photorealism: Removing appearance injection causes reconstruction PSNR to plummet from 28.19 to 21.14 dB (-7.05 dB), confirming that high-level foundation features like DINOv2 alone cannot recover fine-grained scene textures for 3DGS rendering.
  • Semantic distillation stabilizes diffusion convergence: Omitting semantic distillation causes a drastic 4.12 dB drop in generative PSNR, demonstrating that regularizing geometric tokens toward a semantically organized manifold is essential for DiT to effectively model complex 3D distributions.
  • CVC preserves multi-view structural consistency: Disabling CVC causes PSNR to degrade by 2.47 dB and LPIPS to worsen to 0.284, showing that point-wise MSE alone is insufficient to prevent multi-view structural drift during novel view extrapolation.
  • MDF eliminates inference-time decoding collapse: Training decoder heads on interpolated drifted tokens improves PSNR by 0.98 dB and reduces perceptual distance, affirming that manifold drift caused by exposure bias can be successfully alleviated without architectural bloat.

Highlights & Insights

  • Native 3D foundation latent generative paradigm: Moving away from compressing scenes into 2D/video VAE spaces, OneWorld demonstrates that diffusion directly in an augmented 3D foundation feature space yields superior multi-view consistency and physical fidelity.
  • Confidence-filtered cross-view correspondence alignment: Exploiting cosine affinity thresholds to establish pseudo-ground-truth token correspondences and regularizing them via cross-entropy provides an elegant, parameter-free mechanism to bake multi-view geometry into DiT.
  • Cost-effective manifold drift remedy: Formulating train–inference exposure bias as off-manifold latent deviation and resolving it via linear feature interpolation offers an actionable and generalizable blueprint for latent diffusion decoders.

Limitations & Future Work

  • Computational resource requirements: Training the 3D-URAE foundation autoencoder and DiT backbone requires substantial GPU resources (e.g., 32 A100s across 100K+ total steps), limiting lightweight fine-tuning.
  • Complex outdoor unbounded topologies: While showing strong results on DL3DV, extreme outdoor scenarios with severe occlusions and non-static lighting can still experience peripheral boundary artifacts.
  • Future directions: Extending the framework to dynamic 4D world modeling and integrating adaptive step-size ODE solvers to accelerate generation speed for interactive virtual navigation.
  • vs FlashWorld / Prometheus: These methods diffuse in 2D image or video VAE latents before decoding to 3DGS, which frequently introduces geometric tearing under large camera displacements; OneWorld synthesizes directly within a unified 3D space, ensuring global multi-view structural integrity.
  • vs Gen3R: Gen3R separates geometry (point clouds) and appearance (video) into decoupled generations due to video latent compression; OneWorld achieves joint geometry-appearance synthesis within a unified 3D token space.
  • vs LVSM / Aether: Pure feed-forward view synthesis models lack the generative diversity and inpainting power of diffusion models, whereas OneWorld marries the structural priors of feed-forward 3D models with the open-ended synthesis capability of DiT.

Rating

  • Novelty: ⭐⭐⭐⭐⭐ Pioneering diffusion generation directly inside a unified 3D foundation representation space.
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated across RealEstate10K, DL3DV, and WorldScore benchmarks with comprehensive ablations.
  • Writing Quality: ⭐⭐⭐⭐⭐ Coherent technical exposition, clear architectural formulation, and well-designed experimental validation.
  • Value: ⭐⭐⭐⭐⭐ Sets an impactful foundation for transitioning world models from 2D video approximations to native 3D representations.