SynCity 3000: Bootstrapping Scene-Scale 3D Diffusion¶

Conference: ECCV 2026
Paper: ECCV Official
Project: https://research.paulengstler.com/syncity-3k
Area: 3D Vision
Keywords: 3D Scene Generation, Diffusion Models, Convolutional Inference, 3D Gaussian Splatting, Synthetic Data Bootstrapping

TL;DR¶

SynCity 3000 introduces a two-stage framework for large-scale 3D scene generation that first produces an ultra-high-resolution dimetric scene template via overlapping latent window denoising, and then converts it into seamless, arbitrary-scale 3D Gaussian Splats through a convolutional 3D diffusion model fine-tuned on procedurally bootstrapped synthetic scenes.

Background & Motivation¶

Generating high-fidelity, large-scale 3D scenes is of critical importance across film production, interactive gaming, and autonomous robotics simulations. While recent image-to-3D diffusion architectures have demonstrated impressive capabilities in synthesizing detailed 3D assets from single images, they remain predominantly object-centric, restricted to isolated items centered within an isotropic bounding sphere. Existing attempts to generate scene-level environments typically pursue either asset-retrieval composition based on predicted layouts or sequential image outpainting coupled with monocular depth projection. The former frequently causes unnatural object collisions and monotonous repetition, while the latter suffers from severe geometric tears, distorted depth boundaries, and gaping occlusion holes whenever the virtual camera deviates from the input trajectory.

To circumvent the fundamental scarcity of diverse, large-scale 3D scene datasets for training end-to-end models from scratch, pioneering works like SynCity discretize the target world into an orthogonal grid of tiles and repurpose off-the-shelf single-object generators (such as TRELLIS) to synthesize each tile semi-independently before fusing them. Nevertheless, this rigid tile-by-tile reconstruction inevitably induces glaring seam boundaries, severs structures that naturally span across neighboring tiles, and imposes an artificial, visually jarring checkerboard topology across the generated world.

The core tension lies in achieving seamless visual and geometric continuity across arbitrarily expansive environments while reusing object-centric 3D diffusion priors, all without relying on prohibitively expensive, non-existent real-world 3D scene annotations. Core idea: decouple scene synthesis into globally coherent dimetric 2D template generation and convolutional sliding-window 3D Gaussian diffusion, supported by a procedural synthetic scene engine that bootstraps training via core-versus-context masked fine-tuning.

Method¶

Overall Architecture¶

SynCity 3000 operates via a strictly decoupled two-stage pipeline: Stage 1 utilizes a 2D latent diffusion model across a shared canvas with overlapping sliding windows to synthesize a globally coherent, ultra-high-resolution dimetric template image conditioned on global style and local layout prompts; Stage 2 feeds this template into a fine-tuned convolutional 3D generator to first sample a coarse sparse voxel grid, subsequently infer high-dimensional structured appearance latents, and finally decode the resulting representations into a globally seamless 3D Gaussian Splatting world.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Text Prompts & Optional Layout Constraints"] --> B["Stage 1: Dimetric Template Generation<br/>Overlapping MultiDiffusion Latent Denoising"]
    B --> C["Stage 2 - Coarse Structure: Convolutional Sparse Grid Generation<br/>Core + Surrounding Context Window Denoising"]
    C --> D["Stage 2 - Fine Features: Convolutional Structured Latents<br/>Sparse Feature Extraction & Conditioned Denoising"]
    D --> E["3DGS Decoding & Color Correction<br/>Seamless Large-Scale 3D Gaussian Scene Output"]

Key Designs¶

1. Overlapping Dimetric Template Denoising: Unbounded Global Canvas Synthesis

Off-the-shelf 2D text-to-image models are constrained by fixed maximum training resolutions, and naive tiled outpainting creates harsh photometric and semantic seams. SynCity 3000 adopts a dimetric projection initialized with an inpainting base slab and mask. The global latent canvas \(z \in \mathbb{R}^{C \times (H/\epsilon) \times (W/\epsilon)}\) is partitioned into partially overlapping windows \(w \in \mathcal{W}\) sliding with stride \(s < S\). At each denoising step, each window predicts noise estimates conditioned on the global style prompt \(P_{\text{style}}\) and optional local layout prompts \(P_w\): \(\omega_w = \omega_{\text{2D}}(z_w \mid I_{\text{base}}, M_{\text{base}}, P_w, P_{\text{style}})\). By averaging these noise estimates across all overlapping windows over the entire canvas (in the spirit of MultiDiffusion), semantic elements flow smoothly across window borders, completely eliminating the visible grid seams characteristic of SynCity.

2. Core-Context Convolutional 3D Diffusion Inference: Scaling Object-Centric Priors to Continuous Space

Off-the-shelf 3D generators like TRELLIS expect isolated objects centered in a canonical frame. When applied directly to cropped scene tiles, they generate artificial boundaries and collapse along the margins. SynCity 3000 establishes a tight spatial correspondence between 2D template patches and 3D voxel coordinates, reformulating the DiT architecture into a sliding-window convolutional operator. For each spatial window \(w\), the model denoises an expanded latent grid covering both the target "core" \(o_w \in \mathbb{R}^{M \times M \times M \times C}\) and its surrounding 3D context \(o_w^c\), spanning a wider volume \(M_{\text{context}} = M + 2V\). The conditioning image crop \(I_w\) is strictly centered on the core with surrounding regions masked out (\(I_w \odot M\)). By extending sinusoidal positional encodings to negative and boundary coordinates outside the core, the core tokens retain their original canonical coordinates while attending to neighboring 3D structures. Denoised predictions are averaged across overlapping sliding steps, enabling continuous convolutional generation across arbitrary scene spans.

3. Core-Isolated Supervision with Dual-Objective Anti-Forgetting Fine-Tuning

Applying an off-the-shelf object DiT directly to scene patches fails because the model erroneously attempts to infer an arbitrary canonical front-facing direction. Conversely, aggressive unconstrained fine-tuning risks catastrophic forgetting of the rich 3D object prior. SynCity 3000 introduces a dual-objective training strategy sampled with probability \(p=0.5\): one task provides both core and context latents to train cross-boundary spatial awareness, while the other provides only core latents to mirror TRELLIS's original single-object task. Crucially, the mean squared error (MSE) loss is evaluated exclusively within the core region:

\[ \mathcal{L}_{\text{MSE}} = \left\| \omega_w(\text{core}) - \hat{\omega}_w(\text{core}) \right\|_2^2 \]

This design enforces that while the network must actively consume surrounding contextual geometry to maintain topological continuity, it is held responsible only for synthesizing the core volume, preventing boundary divergence and artifact accumulation.

4. Procedural Terrain & Asset Bootstrapping Engine: Overcoming the 3D Scene Data Deficit

The absence of expansive, non-indoor 3D scene training datasets is overcome through a dedicated procedural synthetic dataset engine built in Blender, inspired by LRM-Zero. The engine synthesizes randomized terrain surfaces with procedural heightmaps and stochastic hard-transition color bands, upon which random subsets of 3D assets from Objaverse-XL are scattered with randomized scales, rotations, and collision-free bounds under overhead lighting and cast shadows. Dimetric orthographic renders are captured to match Stage 1's projection, alongside extracted \(M_{\text{context}} \times M_{\text{context}} \times M\) voxel crops around the origin. By rendering 37 auxiliary perspective viewpoints (32 top views, 4 side views, 1 bottom view) to extract and project DINOv2 visual features, the engine procedurally generated 320,000 paired training samples of sparse voxel grids and structured latents, bootstrapping scene-scale diffusion without relying on human annotations.

Loss & Training¶

Fine-tuning was conducted on 2 NVIDIA RTX A6000 GPUs using AdamW with a learning rate of \(5 \times 10^{-6}\), batch size of 1, and no frozen layers. The sparse structure transformer was fine-tuned for 260k steps at \((16 + 2 \times 8)^2 \times 16\) resolution; the structured latent transformer was trained for 660k steps at \((64 + 2 \times 32)^2 \times 64\) resolution. Inference operates on \(896 \times 448\) px template windows with an overlapping stride of 0.5 patch, followed by global color balance correction on the final 3D Gaussian Splats.

Key Experimental Results¶

Main Results¶

To evaluate reconstruction faithfulness to the guiding template, the method was evaluated on 35 LLM-generated templates against TRELLIS, TripoSG, and Hunyuan3D-2.1 using LPIPS, SSIM, and PSNR. Furthermore, geometric reconstruction accuracy was benchmarked against TRELLIS on synthetic proxy scenes across standard (\(1344 \times 672\)) and large (\(2240 \times 1120\)) template resolutions over 32 evaluation views (original paper Tables 1 & 3).

Method	Template LPIPS ↓	Template SSIM ↑	Template PSNR ↑	Geometric Chamfer (Std/Lrg) ↓	Geometric F-score (Std/Lrg) ↑
TRELLIS (Baseline)	0.4094	0.4966	13.5911	0.0137 / 0.0392	0.6685 / 0.6967
TripoSG	0.4182	0.4953	12.2768	-	-
Hunyuan3D-2.1	0.4689	0.4490	11.5394	-	-
SynCity 3000 (Ours)	0.3993	0.5247	14.4137	0.0166 / 0.0302	0.7029 / 0.7536

In a blind user preference study across 27 participants (original paper Table 2), SynCity 3000 achieved a 100.0% win rate over SynCity on layout control flexibility. For 3D reconstruction faithfulness to the template, it scored 71.6% against TRELLIS, and 100.0% against TripoSG and Hunyuan3D-2.1. In overall scene quality, it was preferred over SynCity (63.0%), NuiScene (74.1%), 3DTown (59.3%), and TRELLIS with our templates (78.6%).

Ablation Study¶

The necessity of convolutional fine-tuning, context padding, and overlapping window stride was systematically ablated on template faithfulness (original paper Table 1):

Config	LPIPS ↓	SSIM ↑	PSNR ↑	Note
Full model (Ours)	0.3993	0.5247	14.4137	Complete framework with fine-tuning and context
w/o fine-tuning	0.4726	0.4657	11.7622	Severe degradation due to undefined canonical orientation
w/o context	0.4121	0.5142	14.2038	Lacks neighboring 3D awareness; degraded boundary alignment
Smaller context (\(V=4/16\))	0.4000	0.5047	14.1266	Reduced spatial receptive field impairs long-range structure
Large stride (1.0 patch)	0.4143	0.5192	14.0624	No overlap averaging; sharp increase in duplicated ghost structures

Key Findings¶

Fine-tuning is essential for convolutional application: Using off-the-shelf object models without fine-tuning catastrophically reduces PSNR from 14.41 to 11.76, as unadapted models attempt to force a canonical front-facing orientation onto directionless terrain and scene fragments.
Superior scalability on large scenes: As scene size increases to \(2240 \times 1120\), TRELLIS's Chamfer distance deteriorates sharply from 0.0137 to 0.0392, whereas SynCity 3000 remains robust at 0.0302 and achieves an improved F-score of 0.7536 (+0.057 over TRELLIS), highlighting the power of translation-invariant sliding-window inference.
Overlapping averaging suppresses occlusion ambiguity: Non-overlapping inference (stride 1.0) creates ghosting artifacts where background structures visible in multiple view angles are repeatedly synthesized; a 0.5-patch stride with prediction averaging cleanly resolves these spatial ambiguities.

Highlights & Insights¶

Repurposing object diffusion into spatial convolution: Exploiting dimetric geometric projection allows local 3D denoising windows to tile seamlessly across an arbitrary canvas, transforming an isolated DiT into an unbounded convolutional generator.
Core-context supervision paradigm: Providing wide context while computing gradients solely on the core patch resolves boundary discontinuity while strictly isolating error accumulation.
Procedural synthetic bootstrapping: Bypasses the critical lack of large-scale 3D scene scans by scattering Objaverse-XL assets onto procedural Blender terrain, proving that high-capacity diffusion models can bootstrap scene understanding from stochastic synthetic assemblies.

Limitations & Future Work¶

Dimetric projection dependency: The approach relies heavily on orthographic dimetric projection to preserve scale uniformity across sliding windows; extending the formulation to ground-level perspective frustums requires handling perspective scale deformation.
Vertical height extrusion bounds: The 60 px vertical mask extrusion limits the maximum height of structures at boundary regions; towering spires or steep cliffs may still suffer height clipping.
Future directions: Integrating octree hierarchical latents or streaming radiance fields for truly infinite open-world interactive generation.

vs SynCity: SynCity relies on rigid tile-by-tile generation and hard concatenation, resulting in disjointed seams and checkerboard grids; SynCity 3000 achieves organic, continuous spatial flow via overlapping template diffusion and convolutional 3D inference.
vs 3DTown: 3DTown relies on error-prone monocular depth estimation heuristics and inpainting masks that cause geometric hollows; SynCity 3000 denoises directly in 3D structured latent space, ensuring watertight geometry.
vs NuiScene: NuiScene trains on a tiny 43-scene dataset with visible VecSet chunk discontinuities and secondary texture generation; SynCity 3000 trains on 320k procedurally bootstrapped scenes and directly outputs rich 3D Gaussian Splats.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Elegant reformulation of object DiTs into convolutional operators with core-context supervision.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous benchmark covering template fidelity, dual-scale geometric accuracy, ablations, and a 27-user study.
Writing Quality: ⭐⭐⭐⭐⭐ Lucid explanation of the mathematical and architectural mechanisms bridging 2D canvas and 3D voxel space.
Value: ⭐⭐⭐⭐⭐ Sets a compelling blueprint for bootstrapping large-scale 3D world synthesis without specialized scene datasets.