content_hash: 2c54061abf9c0b8c
Scale3D: Autoregressive Modeling for Large Outdoor Scene Generation¶
Conference: ECCV 2026
Paper: ECCV Official
Area: 3D Vision
Keywords: Unbounded 3D Scene Generation, Hierarchical Autoregressive Models, 3D Chunk VAE, Direct Preference Optimization, Spatial Text Conditioning
TL;DR¶
Scale3D decouples global layout planning from local geometric synthesis via a hierarchical autoregressive framework, combining an enhanced high-fidelity 3D chunk VAE and Direct Preference Optimization (DPO) to synthesize seamless, large-scale outdoor 3D meshes with natural spatial text control.
Background & Motivation¶
Generating large-scale outdoor 3D environments, such as expansive city blocks, presents an enduring challenge: models must maintain long-range structural coherence across vast layouts without degrading high-frequency local geometric details. Existing paradigms inevitably falter on at least one of these objectives. Approaches that lift 2D diffusion priors into 3D via multi-view reconstruction produce visually appealing textures but accumulate depth projection errors and cross-view inconsistencies, resulting in warped and unreliable underlying geometry. Conversely, direct 3D generation frameworks that denoise explicit volumetric representations typically rely on local window-based diffusion models. When extended to unbounded scenes, patch outpainting or resampling-based inpainting introduces visible seams, discontinuities, and ground-plane misalignments due to inherently restricted context windows.
Shifting toward autoregressive (AR) models offers a principled mechanism to capture long-range spatial dependencies across extended scenes. However, treating continuous 3D geometry as a naive monolithic causal sequence fails in practice. Forcing a single causal Transformer to simultaneously forecast macro-scale layout distribution and high-frequency local surface features overburdens the model, precipitating severe mode collapse into overly smooth, featureless meshes. Compounding these structural limitations, existing 3D chunk VAEs suffer from upper-bound resolution constraints imposed by whole-scene SDF tools, while prior controllable generation frameworks depend on difficult-to-acquire structural priors such as scene graphs or HD maps.
This paper tackles the challenge by explicitly decoupling macro-level structural planning from micro-level geometric synthesis. Core idea: establish a hierarchical autoregressive paradigm where a causal AR Transformer first predicts macro-chunk layout embeddings along a 2D Z-order trajectory, followed by a bidirectional masked autoregressive generator with a continuous diffusion head that synthesizes high-fidelity local geometry conditioned on adjacent chunk boundaries, supported by an enhanced salient-guided chunk VAE and DPO alignment.
Method¶
Overall Architecture¶
Scale3D represents an expansive outdoor scene as an \(H \times W\) spatial grid of latent chunks, where each chunk is encoded by an enhanced 3D chunk VAE into a set of \(l\) continuous latent tokens. Macro-level generation proceeds via a Z-order space-filling traversal, utilizing a causal Transformer to sequentially plan global chunk embeddings while preserving 2D spatial locality. Micro-level generation then operates on each chunk using a bidirectional masked autoregressive (MAR) model conditioned on hierarchical context—integrating global chunk guidance alongside top and left spatial neighbors—and synthesizes continuous geometric tokens via a diffusion head. Finally, the VAE decoder reconstructs seamless, detailed 3D meshes across the entire unbounded layout.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Mesh & Spatial Text Prompt"] --> B["Enhanced 3D Chunk VAE<br/>Direct mesh partitioning & salient edge guidance"]
B --> C["Z-Order Causal AR Global Planning<br/>2D RoPE spatial layout planning for chunk embeddings"]
C --> D["Bidirectional Masked Local Detail Generation<br/>Continuous diffusion denoising with neighbor conditioning"]
D --> E["Collaborative Optimization Strategy<br/>Coherency regularization & Direct Preference Optimization"]
E --> F["Output Unbounded Large-Scale 3D Mesh"]
Key Designs¶
1. Enhanced 3D Chunk VAE: direct mesh partitioning and salient geometry guidance Prior chunk VAEs convert the entire scene into an SDF before sampling occupancy grids, but toolchain 32-bit representations limit the overall scene volume to roughly \(1000^3\) voxels, constraining individual chunk occupancy to a coarse resolution (e.g., \(50 \times h_{\text{vox}} \times 50\)). Scale3D introduces Flexible Resolution Control (FRC), which directly partitions the raw scene mesh into individual chunk meshes before computing localized chunk SDFs, boosting chunk occupancy resolution up to \(256 \times h_{\text{vox}} \times 256\) (and up to 1000 voxels per axis). To further preserve sharp building contours, Salient Geometry Guidance (SGG) augments uniform surface points \(P_u\) and normals \(N_u\) with sharp-edge sampled points \(P_s\) and normals \(N_s\), encoding both via cross-attention with learnable queries \(L\) and aggregating them through element-wise summation: $\(Z = Z_u + Z_s = \text{CrossAttn}(L, [P_u, N_u]) + \text{CrossAttn}(L, [P_s, N_s])\)$ This dual enhancement delivers significantly sharper geometric boundaries and surface reconstruction fidelity.
2. Z-Order Causal AR Global Planning: 2D RoPE spatial layout planning for chunk embeddings To enforce long-range structural coherence across expansive layouts, the global stage treats each chunk as a macro unit. The generation follows a 2D Z-order (Morton) spatial traversal, which maintains strict causal sequence ordering while preserving two-dimensional spatial locality. A causal Transformer predicts the global embedding \(\mathbf{z}_i^{\text{global}}\) for chunk \(i\) conditioned on all prior chunk representations, an optional text condition \(c\), and 2D Rotary Positional Embeddings (\(\text{RoPE}(x,y)\)): $\(p(\mathbf{z}_i^{\text{global}} \mid \mathbf{z}_{<i}^{\text{global}}, \texttt{[BEGIN]}, c, \text{RoPE}(x,y))\)$ By encoding continuous relative spatial coordinates via 2D RoPE, the global planner reliably captures macro-scale terrain undulations and architectural layouts across arbitrarily expanding scene scales.
3. Bidirectional Masked Local Detail Generation: continuous diffusion denoising with neighbor conditioning Because 3D geometric tokens lack natural causal ordering, imposing strict causal prediction within chunks induces error accumulation. Scale3D instead deploys a bidirectional Masked Autoregressive (MAR) model. For a randomly selected subset of masked tokens \(\mathcal{M}_i\), the model predicts missing continuous tokens conditioned on unmasked tokens \(\mathbf{z}_i^{\mathcal{U}_i}\) and a hierarchical context vector \(\mathbf{c}_i^{\text{hierarchical}}\) that concatenates the global layout token with boundary tokens from the top and left adjacent chunks: $\(\mathbf{c}_i^{\text{hierarchical}} = \text{Concat}([\mathbf{z}_{\text{top}}, \mathbf{z}_{\text{left}}, \mathbf{z}_i^{\text{global}}])\)$ Conditioned on representations from the bidirectional Transformer, a lightweight 12-layer continuous diffusion head iteratively denoises tokens via DDPM. Injecting explicit top and left neighbor boundaries completely eliminates chunk boundary seams and elevation steps.
4. Collaborative Optimization Strategy: coherency regularization and Direct Preference Optimization Unconditional large-scale 3D generation frequently collapses into empty plains or repetitive trivial geometry. To combat this failure mode, Scale3D combines neighborhood coherency regularization (\(\mathcal{L}_{\text{coherency}}\)) and Direct Preference Optimization (\(\mathcal{L}_{\text{DPO}}\)). Coherency loss enforces cosine-based feature diversity between adjacent chunk embeddings to prevent spatial lock-in. Furthermore, using a curated dataset of roughly 400 human-ranked \(8 \times 8\) chunk scene generation pairs \(\mathcal{D}\), DPO fine-tunes the policy model: $\(\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(\mathbf{Z}_w, \mathbf{Z}_l) \sim \mathcal{D}} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(\mathbf{Z}_w \mid c)}{\pi_{\text{ref}}(\mathbf{Z}_w \mid c)} - \beta \log \frac{\pi_\theta(\mathbf{Z}_l \mid c)}{\pi_{\text{ref}}(\mathbf{Z}_l \mid c)} \right) \right]\)$ DPO steers the distribution away from desolate, monotonous layouts and heavily aligns generation toward physically coherent, topographically diverse scenes.
Loss & Training¶
The 3D chunk VAE is optimized with binary cross-entropy on high-resolution occupancy \(\mathcal{L}_{bce}\), latent KL divergence \(\mathcal{L}_{kl}\), embedding reconstruction loss \(\mathcal{L}_{emb}\), and chunk surface height regression \(\mathcal{L}_{height}\): $\(\mathcal{L}_{\text{VAE}} = \lambda_{bce}\mathcal{L}_{bce} + \lambda_{kl}\mathcal{L}_{kl} + \lambda_{emb}\mathcal{L}_{emb} + \lambda_{height}\mathcal{L}_{height}\)$ The hierarchical autoregressive generator is trained using the joint objective: $\(\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{diffusion}} + \lambda_{\text{coh}}\mathcal{L}_{\text{coherency}} + \lambda_{\text{dpo}}\mathcal{L}_{\text{DPO}}\)$ Hyperparameters are set to \(\lambda_{\text{coh}} = 0.1\), \(\lambda_{\text{dpo}} = 0.05\), and DPO temperature \(\beta = 0.1\). For spatial text conditioning, text embeddings are randomly dropped with \(p_{\text{drop}} = 0.1\) during training to support Classifier-Free Guidance (CFG).
Key Experimental Results¶
Main Results¶
On single-scene and 13-scene benchmarks across multi-scale grids (\(2 \times 2\), \(8 \times 8\), and \(32 \times 32\)), unconditional scene generation is evaluated using Fréchet PointNet++ Distance (FPD) and Kernel PointNet++ Distance (\(\text{KPD}^* = \text{KPD} \times 10^3\), lower is better):
| Dataset | Method | 2×2 FPD (↓) | 2×2 KPD* (↓) | 8×8 FPD (↓) | 8×8 KPD* (↓) | 32×32 FPD (↓) | 32×32 KPD* (↓) |
|---|---|---|---|---|---|---|---|
| 1 scene | NuiScene (2×2) | 0.07 | 0.21 | 0.32 | 0.57 | 0.87 | 1.92 |
| 1 scene | NuiScene (16×16) | 0.06 | 0.22 | 0.29 | 0.54 | 0.76 | 1.38 |
| 1 scene | Scale3D (Ours) | 0.07 | 0.24 | 0.24 | 0.52 | 0.71 | 0.91 |
| 13 scenes | NuiScene (2×2) | 0.11 | 0.34 | 0.43 | 0.67 | 0.97 | 2.65 |
| 13 scenes | NuiScene (16×16) | 0.12 | 0.34 | 0.34 | 0.63 | 0.82 | 2.36 |
| 13 scenes | Scale3D (Ours) | 0.12 | 0.33 | 0.27 | 0.59 | 0.75 | 1.97 |
In text-controlled generation, Scale3D demonstrates high multi-modal alignment from \(2 \times 2\) to \(16 \times 16\) scales (CLIP scores: 0.98 to 0.91; Uni3D similarity: 0.36 to 0.29). On 50 spatial control test prompts at \(8 \times 8\), Spatial Relation Accuracy reaches 0.78 for directional constraints and 0.71 for height rank (against random baselines of 0.25 and 0.17).
Ablation Study¶
Quantitative evaluations of the Scale3D VAE modules and autoregressive design variations on the single-scene benchmark:
Table 1: Ablations of FRC and SGG in Scale3D VAE (Single-Scene)
| Config | FRC (High-Res) | SGG (Salient Normal) | Volumetric IoU (↑) | Chamfer-p (↓) | Chamfer-n (↓) | F-Score (↑) | Note |
|---|---|---|---|---|---|---|---|
| Baseline (NuiScene VAE) | ✗ | ✗ | 0.4690 | 0.0062 | 0.0212 | 0.8314 | Original low-resolution occupancy |
| FRC Only | ✓ | ✗ | 0.6631 | 0.0009 | 0.0090 | 0.9657 | Resolution increased from 50 to 256 |
| SGG Only | ✗ | ✓ | 0.4692 | 0.0059 | 0.0225 | 0.8362 | Sharp-edge sampling added |
| Full Model | ✓ | ✓ | 0.6686 | 0.0009 | 0.0087 | 0.9696 | Substantial gains across all metrics |
Table 2: Ablation Study of Unconditional AR Framework (Single-Scene)
| Variant / Config | 2×2 FPD (↓) | 2×2 KPD (↓) | 8×8 FPD (↓) | 8×8 KPD (↓) | 32×32 FPD (↓) | 32×32 KPD (↓) | Note | |---|---|---|---|---|---|---| | Block-wise Causal AR | 0.13743 | 12.34 | 0.17264 | 14.57 | 0.21043 | 17.42 | Single causal head generates full chunk; coarse | | Pure Bidirectional MAR | 0.00022 | 0.05 | 0.00038 | 0.13 | 0.00265 | 0.68 | Lacks global planning; collapses at large scale | | w/o DPO | 0.00031 | 0.07 | 0.00072 | 0.16 | 0.00132 | 0.47 | Defaults to empty or repetitive flat terrain | | w/o CA Loss | 0.00026 | 0.05 | 0.00043 | 0.13 | 0.00098 | 0.38 | Seam artifacts and duplicate bridge structures | | Scale3D (Full) | 0.00025 | 0.05 | 0.00039 | 0.13 | 0.00095 | 0.36 | Maintains structural coherence at scale |
Key Findings¶
- Scale3D's performance advantage expands monotonically as scene scale scales up from \(8 \times 8\) to \(32 \times 32\). At \(32 \times 32\), single-scene \(\text{KPD}^*\) decreases from 1.92/1.38 (NuiScene) to 0.91, and 13-scene \(\text{KPD}^*\) from 2.65/2.36 to 1.97, proving that hierarchical macro-micro AR planning prevents the accumulated drift characteristic of local window diffusion models.
- Increasing occupancy supervision resolution via FRC contributes the primary performance leap (IoU jumps from 0.4690 to 0.6631), while salient edge guidance (SGG) sharpens surface boundaries, yielding the lowest normal Chamfer distance (0.0087).
- In architectural ablations, a monolithic block-wise causal model fails catastrophically (\(\text{KPD} > 12\)), while pure bidirectional MAR collapses at \(32 \times 32\) (FPD degrades nearly tenfold). Decoupled hierarchical modeling is strictly necessary to balance macro continuity and micro detail.
Highlights & Insights¶
- Hierarchical Decoupling of AR Planning and Diffusion Denoising: Separating causal global layout planning from local bidirectional masked diffusion circumvents capacity overload in monolithic 3D generators, ensuring macroscopic coherence alongside intricate geometric contours.
- Mesh-First Partitioning Data Pipeline: By identifying the 32-bit floating-point limitations in whole-scene SDF tools, partitioning raw meshes prior to localized SDF computation elevates occupancy supervision resolution by an order of magnitude.
- DPO for 3D Diversity and Coherence: Introducing preference alignment with approximately 400 human-ranked scene comparisons effectively mitigates the pervasive tendency of unconditional 3D generators to collapse into flat, barren terrains.
Limitations & Future Work¶
- Inference Latency Scalability: Sequential chunk autoregression combined with iterative multi-step diffusion sampling leads to cumulative inference latency across very large scenes (e.g., \(32 \times 32 = 1024\) chunks). Future iterations could explore non-autoregressive chunk dispatching or consistency distillation.
- Spatial Control Coarseness on Complex Overhangs: Capturing spatial relationships via 2D heightmap watershed segmentation limits descriptive fidelity for complex vertical overhangs (e.g., tunnels, bridges, cavernous terrain). Incorporating 3D volumetric semantic occupancy fields would improve geometric prompting.
- Absence of Photometric Texture: Scale3D focuses exclusively on untextured geometric meshes; integrating PBR materials or view-consistent appearance models represents an essential next step.
Related Work & Insights¶
- vs NuiScene: NuiScene employs local context diffusion outpainting, which remains prone to boundary seams and drift across unbounded expanses; Scale3D leverages causal global planning with explicit neighbor conditioning, eliminating boundary seams at \(32 \times 32\) scale.
- vs XCube / LT3SD: XCube and LT3SD rely on global voxel hierarchies and rigid scale bindings, encountering severe voxel blurring and discontinuous ground planes at \(8 \times 8\) scale; Scale3D preserves crisp structural integrity across expanding boundaries.
- vs GraphDreamer / InstructLayout / UrbanWorld: Existing controllable generation relies heavily on structural scene graphs or HD maps; Scale3D enables direct spatial control using natural language instructions coupled with 2D RoPE coordinate encoding.
Rating¶
- Novelty: ⭐⭐⭐⭐☆ Effective hierarchical decoupling of causal layout planning and masked continuous diffusion, paired with high-resolution chunk VAE and DPO.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive cross-scale benchmarks (\(2 \times 2\) to \(32 \times 32\)) across 1-scene and 13-scene datasets, in-depth architectural and VAE ablations, and rigorous spatial reasoning evaluation.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear exposition, thorough baseline comparisons, and rigorous mathematical formulations.
- Value: ⭐⭐⭐⭐☆ Highly impactful for procedural world generation, autonomous driving simulation, and digital twin synthesis.