GaussianGPT: Towards Autoregressive 3D Gaussian Scene Generation

Conference: ECCV 2026
arXiv: 2603.26661
Code: None
Area: 3D Vision
Keywords: 3D Gaussian Splatting, Autoregressive Generation, Scene Generation, Vector Quantization, Transformer

TL;DR

GaussianGPT compresses 3D Gaussian scenes into discrete token sequences and employs a causal Transformer with 3D rotary position embeddings for step-by-step autoregressive token prediction. This introduces the first purely autoregressive paradigm for 3D Gaussian scene generation, achieving or outperforming state-of-the-art diffusion models in shape synthesis and scene generation while naturally supporting scene completion and outpainting.

Background & Motivation

The mainstream paradigm for current 3D generative modeling relies on diffusion models and flow matching, which formulate generation as a global denoising or continuous-time transformation process, delivering high visual fidelity in shape and scene synthesis. However, these methods suffer from a fundamental contradiction: real-world scenes are built incrementally—designers progressively lay out, expand, and edit them—whereas diffusion models refine the entire scene all at once, making incremental editing, local completion, and controllable generation unnatural. In contrast, autoregressive Transformers have demonstrated powerful capabilities in structured sequence modeling and controllable generation in both language and vision domains, yet remain rarely explored in structured 3D scene synthesis.

The core obstacle lies in the lack of a natural 1D sequential order in 3D data. 3D Gaussian primitives are unstructured point sets containing continuous attributes such as position, opacity, scale, rotation, and color, which lack a canonical ordering and are difficult to directly feed into discrete token sequence models. Furthermore, indoor scenes require both global structural consistency (room layout) and local compositional flexibility (object placement), adding to the modeling difficulty.

This work's Key Insight is to: compress 3D Gaussian scenes into discrete latent grids using vector quantization, serialize them into a 1D token stream using xyz traversal, and finally employ a causal Transformer with 3D spatial position biases for autoregressive generation. The Core Idea is to transform 3D scene generation into a sequence prediction problem: "predicting the next occupied voxel location and its corresponding Gaussian features step-by-step based on the generated spatial context," enabling an autoregressive pathway alternative to the diffusion paradigm.

Method

Overall Architecture

The goal of GaussianGPT is to autoregressively generate a complete scene representation token-by-token given a 3D Gaussian scene (or an empty scene). The entire pipeline is divided into two major stages: (1) Scene Compression—compressing continuous Gaussian scenes into discrete latent grid tokens via a sparse 3D convolutional autoencoder; (2) Autoregressive Modeling—serializing the latent grid into a token stream and modeling the joint distribution using a causal Transformer. These two stages are trained independently: the autoencoder is trained first to obtain high-quality discrete representations, followed by GPT training on its latent space.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["3D Gaussian Scene<br/>Position/Opacity/Scale/Rotation/Color"] --> B["Sparse 3D Feature Grid<br/>Voxel Assignment + Encoding Heads"]
    B --> C["Sparse 3D CNN Encoder<br/>3x Downsampling"]
    C --> D["LFQ Discretization<br/>Binarization by Sign → Codebook Indices"]
    D --> E["Discrete Latent Grid<br/>Voxel Size 20cm"]
    E --> F["xyz Serialization<br/>Alternating Position and Feature Tokens"]
    F --> G["Causal Transformer<br/>3D RoPE + Split Vocabularies"]
    G -->|Training: Teacher Forcing| H["Token-by-Token Prediction<br/>Cross-Entropy Loss"]
    G -->|Inference: Autoregressive Sampling| I["Generated Token Sequence"]
    I --> J["LFQ Decoding → Sparse 3D CNN Decoder"]
    J --> K["Reconstructed 3D Gaussian Scene"]
    G -->|Partial Scene Prefix| L["Scene Completion/Outpainting<br/>Same Model, No Modification Required"]

Key Designs

1. Sparse 3D Latent Encoding and LFQ Discretization: Bridging Gaussian Scenes to Token Sequences

Feeding continuous, unstructured 3D Gaussian scenes into an autoregressive Transformer requires converting them into discrete token sequences first. This work adopts a three-tier compression pipeline: First, Gaussian primitives are voxelized into a coordinate grid based on their positions, where absolute positions are replaced with voxel center offsets, and multiple Gaussians within the same voxel are randomly downsampled. Each Gaussian attribute is encoded by an independent encoding head (linear layer + residual MLP) and concatenated into a unified feature vector, forming a sparse input feature grid. Next, a sparse 3D convolutional encoder applies three downsampling steps (compressing voxel size from 2.5cm to 20cm) to compress the scene into a compact latent representation, preserving spatial locality and translation equivariance. Finally, Lookup-Free Quantization (LFQ) is used to directly discretize the encoder output \(z\) based on its sign into 0/1, meaning the codebook index is determined by \(\text{sign}(z)\), dispensing with explicit codebook lookups.

The key reason for selecting LFQ over traditional VQ is that traditional VQ requires nearest-neighbor search, which increases training time by approximately one-third and often suffers from codebook collapse. LFQ encourages uniform codebook utilization through an entropy maximization loss, with experiments demonstrating a 99.2% utilization rate on a 4096-size codebook. The autoencoder training loss consists of three components:

\[\mathcal{L} = \underbrace{\lambda_{RGB}\mathcal{L}_{RGB} + \lambda_{perc}\mathcal{L}_{perc}}_{\text{Re-rendering}} + \underbrace{\lambda_{occ}\mathcal{L}_{occ}}_{\text{Occupancy}} + \underbrace{\lambda_{LFQ}\,\text{softplus}(\mathcal{L}_{LFQ}+5)}_{\text{Codebook Entropy}}\]

where the re-rendering loss calculates RGB L1 and VGG-19 perceptual losses on sampled views, the occupancy loss supervises the occupancy predictions in the decoder's upsampling layers using binary cross-entropy (BCE), and the codebook entropy loss is kept positive via a softplus offset.

2. Split Position-Feature Vocabularies and Alternating Sequences: Decoupling Geometry and Appearance Modeling

When serializing the latent grid into a token stream, this work utilizes a fixed xyz traversal order (minimum z first, i.e., column-by-column scanning), where each voxel corresponds to a pair of tokens: a position token (the relative index of the voxel within the current chunk) and a feature token (the quantized latent feature of the voxel), arranged alternately. A key design is that position and feature tokens use two independent vocabularies and output heads: they share the same Transformer backbone, but during position steps, a position head predicts the next occupied voxel index, and during feature steps, a feature head predicts the quantized feature of the preceding position.

The motivation for this is straightforward: geometry (where is occupied) and appearance (what the occupied space looks like) are two entirely different prediction tasks. Sharing a single vocabulary forces them to compete for limited codebook indices, confusing the model. Splitting the vocabularies eliminates this competition and allows independent scaling of codebook sizes. Crucially, this design naturally supports flexible control: the index range of the position head is bound to the chunk size, while the codebook size of the feature head is independent of chunk dimensions. During training, the cross-entropy loss is computed only on the corresponding vocabulary for alternating steps, with invalid vocabulary entries masked out.

3. 3D Rotary Position Embedding (RoPE) and 4th-Dimensional Token-Type: Enabling Spatial Distance-Aware Attention

Standard 1D position embeddings in Transformers (whether learnable absolute embeddings or RoPE) primarily encode sequence order information. However, under xyz serialization, tokens that are adjacent in the 1D sequence may be far apart in 3D space (e.g., jumping to the next column after finishing one), whereas spatially adjacent voxels may be separated by the length of an entire column in the sequence. Directly applying 1D RoPE forces the model to learn a flawed "sequential proximity = spatial proximity" prior.

The proposed solution extends RoPE from 1D to 3D: in attention calculations, the rotary encoding is applied to the token's actual voxel coordinates \((x, y, z)\) rather than its sequence index \(i\). Consequently, the attention score becomes a function of relative 3D spatial offsets instead of sequence offsets—allowing the model to bypass long sequence distances and directly attend to spatially adjacent, already-generated voxels. Furthermore, because position and feature tokens alternate and share the same 3D coordinates, this work introduces a fourth token-type dimension: type=0 for position tokens and type=1 for feature tokens. This is encoded using an additional rotary frequency, helping the Transformer further decouple geometric and appearance signals under a unified attention formulation.

4. Autoregressive Scene Completion and Outpainting: Same Model, Same Mechanism, No Architectural Modification

A natural advantage of the autoregressive paradigm is that generation is formulated as predicting subsequent tokens based on a prefix. This means that given a partial scene, we only need to serialize it into prefix tokens, allowing the model to naturally "continue writing" the remaining scene. Thus, completion and unconditional generation are completely unified in mechanism, without requiring specialized inpainting processes like RePaint used in diffusion models. The same principle extends to large-scale scene outpainting: while the model is trained on fixed-size chunks (constrained by the context window), sliding-window inference can be applied during deployment. By predicting one new column at a time and shifting the window forward using the already generated regions as context, the model can synthesize large 12m x 12m scenes, bypassing the training-time chunk size limit (where chunks are only 4m x 4m).

Two practical tricks are introduced during inference: (1) Occupied position masking: Since position tokens map directly to voxel indices in the chunk, already occupied positions can be masked out during generation to ensure that sampling always adheres to ordering constraints; (2) Backtracking resampling: During large-scale scene generation, if an empty column void of any geometry is predicted, the model backtracks to the previous token and resamples (up to 5 retries by default), effectively curbing generative degeneration and preserving scene continuity.

Loss & Training

Training is split into two independent stages. Autoencoder Stage: Optimized using Adam with a learning rate of \(10^{-4}\) and cosine decay to 10% of the initial value. Trained on 4x RTX A6000 GPUs with an effective batch size of 8 for scenes (approx. 4 days) and 24 for PhotoShape objects (approx. 2 days). Chunks satisfying a minimum occupancy threshold (0.2) are randomly sampled during training, and camera sampling prioritizes views that capture more of the chunk area. GPT Stage: Built on nanochat (GPT-2 backbone), combining AdamW and Muon optimizers with per-module learning rates (0.004 for output heads, 0.2 for embeddings, 0.005 for residual weights, etc.) and cosine decay to 10%. Trained on 4x GH200 GPUs with an effective batch size of 64 for approx. 1 day (scenes) and 4.5 hours (objects). The loss is standard autoregressive cross-entropy. During inference, a temperature of 0.9 and Nucleus Sampling (\(p=0.9\)) are used.

Key Experimental Results

Main Results

Shape Synthesis (PhotoShape Chairs): On the unconditional chair generation task, GaussianGPT achieves an FID of 5.68, KID of 0.00184, and COV of 67.40%, comprehensively outperforming the previous state-of-the-art method L3DG (FID 8.49, KID 0.00315, COV 63.80%) and the diffusion method DiffRF (FID 15.95). Its MMD is on par with L3DG (4.278 vs 4.241), demonstrating that autoregressive models can equal or exceed diffusion methods in both visual quality and geometric diversity.

Method	FID ↓	KID ↓	COV ↑	MMD ↓
π-GAN	52.71	0.01364	39.92	7.387
EG3D	16.54	0.00841	47.55	5.619
DiffRF	15.95	0.00794	58.93	4.416
L3DG	8.49	0.00315	63.80	4.241
GaussianGPT	5.68	0.00184	67.40	4.278

Scene Synthesis (3D-FRONT chunk): For unconditional generation, GaussianGPT significantly outperforms L3DG in appearance (FID 94.85 vs. 100.98) and layout (FID 84.14 vs. 93.84), while showing slightly lower geometric diversity and distribution matching (COV 0.548 vs. 0.692, MMD 0.118 vs. 0.096). This reflects the "clean yet conservative" tendency of autoregressive methods: they generate tidier geometry with fewer anomalous primitives, but their sample diversity is lower than that of diffusion models. However, when transitioning from unconditional generation to completion, the advantages of the autoregressive approach become evident: under 25% prefix completion, GaussianGPT outperforms L3DG + RePaint across all dimensions (appearance, geometry, layout, CLIP similarity), with the margin widening further as the context increases to 50%.

Task	Method	Appearance FID ↓	Geometry COV ↑	Layout FID ↓	CLIP-Sim ↑
Unconditional	L3DG	100.98	0.692	93.84	—
Unconditional	Ours	94.85	0.548	84.14	—
25% Completion	L3DG	109.35	0.925	112.57	0.834
25% Completion	Ours	100.06	0.879	95.60	0.841
50% Completion	L3DG	106.54	0.940	111.71	0.843
50% Completion	Ours	98.13	0.940	90.77	0.851

Ablation Study

Serialization Strategy (Table 3): Comparing xyz, Z-order, Hilbert curves, and transposed variants, the xyz traversal yields the lowest training cross-entropy (CE) of 2.151, and transposed Z-order is close on validation CE (2.423 vs. 2.421). However, orderings like Hilbert curves that offer stronger spatial locality guarantees do not bring additional gains. This implies that 3D RoPE already encodes sufficient spatial information, thereby reducing the necessity of hardcoded sequence-level locality.

Serialization Strategy	Training CE ↓	Validation CE ↓
Z-order	2.203	2.432
Transposed Z-order	2.204	2.423
Hilbert	2.347	2.439
Transposed Hilbert	2.334	2.434
xyz (Ours)	2.151	2.421

Core Component Ablation (Table 4): Sharing vocabularies increases validation CE from 2.421 to 2.449. Learnable absolute position embedding performs worse (validation CE 2.461), and 1D RoPE is similarly inferior to 3D RoPE (validation CE 2.446). All three designs make independent contributions to the final performance.

Configuration	Training CE ↓	Validation CE ↓
Full Model	2.151	2.421
w/ Shared Vocabulary	2.157	2.449
w/ Learnable PE	2.210	2.461
w/ 1D RoPE	2.227	2.446

Key Findings

• Splitting vocabularies is the most cost-effective design: Although sharing vocabularies seems simpler, it forces geometry and appearance to compete within a limited codebook, resulting in a 0.028 increase in validation CE and hindering independent control over chunk size and codebook dimensions in inference.
• 3D RoPE makes simple xyz traversal sufficient: Even though the 3D locality of xyz traversal is subpar (adjacent sequence tokens can be far apart in space), 3D RoPE enables the model to directly attend based on spatial coordinates, bypassing the need for complex space-filling curves.
• Autoregressive inference efficiency excels in completion scenarios: While autoregressive sampling is slower than diffusion during unconditional generation (78s vs. 24s/chunk), it drops to 39s under 50% prefix completion (whereas diffusion remains at 24s) because fewer tokens need to be generated.
• Large-scale scene generation quality degrades more severely along the x-direction: xyz serialization causes x-extension to require larger sequence leaps, leading to more pronounced cumulative errors, whereas y-extension is relatively local and shows slower degradation.

Highlights & Insights

• Decoupling geometry and appearance via "alternating sequences + split vocabularies": This is an extremely elegant design choice. Position and feature tokens alternate in the sequence but are predicted using different heads and vocabularies, essentially factorizing the joint "what is where" problem into two conditionally independent sub-problems. This scheme can be transferred to any spatial generation task requiring simultaneous modeling of "existence" and "attributes" (e.g., point clouds, voxels, graph structures).
• 3D RoPE + 4th-dimensional token-type: Extending RoPE from 1D sequence dimensions to 3D spatial coordinates, alongside an extra token-type dimension, retains RoPE's relative position encoding benefits (length extrapolation) while liberating the model from the spatial distortions of serialization. This design is highly relevant for any context requiring the serialization of high-dimensional structures (e.g., spatio-temporal position embeddings in tokenized videos, node position embeddings in serialized graphs).
• The unification of "generation as completion" is an underappreciated contribution: Doing completion on diffusion models requires external patch-up strategies like RePaint, whereas autoregressive models handle unconditional generation and completion via the exact same mechanism. This is a structural advantage of the paradigm, not a trick. Experiments confirm this: the more context provided, the more dominant the autoregressive approach becomes.
• The practical utility of replacing VQ with LFQ: Traditional VQ suffers severely from codebook collapse and nearest-neighbor search overheads when dealing with sparse high-dimensional data like 3D scenes. LFQ's simple and efficient binarization scheme achieves a highly robust codebook utilization rate (99.2%), offering directly applicable engineering value to other 3D tasks requiring vector quantization.

Limitations & Future Work

• Slow training-free/unconditional generation speed: Unconditional generation takes 78s per chunk (compared to L3DG's 24s) due to sequential token-by-token sampling. The authors suggest several acceleration paths (Flash Decoding, speculative decoding, multi-token prediction heads, sparse attention), which remain unimplemented.
• Quality degradation in large-scale outpainting: As the number of sampling steps increases, cumulative error causes farther regions to degrade in quality (normalized KID rises from 1.0 to 1.3–1.6). Backtracking resampling mitigates but does not fully eliminate this degradation.
• Trade-off between geometric diversity and diffusion methods: During unconditional scene generation, the COV is lower than L3DG (0.548 vs. 0.692), indicating that autoregressive models yield narrow structural variations because they tend to output the "most probable" rather than "highly diverse" layouts. This is an inherent limitation of autoregressive maximum likelihood modeling, which might necessitate multimodal modeling strategies beyond mere temperature tuning.
• Insufficient autoencoder fidelity on real-world datasets: ScanNet++ experiments expose reconstruction bottlenecks, where high-frequency details are sub-optimal, and missing/unobserved regions inherent in real scans are not faithfully reconstructed. The authors note that autoregressive models are naturally suited to handle uncertain regions (via masking), pointing to future work in uncertainty-aware modeling.
• Anisotropic degradation patterns caused by xyz traversal: Degradation along the x-direction is significantly more severe than along the y-direction, showing that the fixed serialization order introduces directional biases. Future work could explore adaptive or multi-path serialization strategies.

Related Work & Insights

• vs L3DG: L3DG serves as the foundation for the autoencoder in this work (sharing the sparse 3D CNN architecture), but while L3DG directly generates in the latent space using a diffusion model, this work adopts an autoregressive Transformer + LFQ. The core difference lies in the generative paradigm: diffusion relies on global iterative refinement, while autoregressive sampling operates on local sequential decisions. Experiments show complementary strengths: diffusion excels in unconditional diversity, while the autoregressive approach dominates in completion and controllability.
• vs MeshGPT / MeshAnything: These works tackle autoregressive mesh generation, sharing the broad "autoregressive 3D" direction but operating on mesh face sequences. Such sequences possess inherent topological structures. GaussianGPT operates on 3D Gaussian primitives, which are more general but completely unstructured, presenting a steeper challenge that requires first "constructing" a sequential structure through compression and quantization.
• vs DiffRF: A representative diffusion model for 3D generation that performs denoising over voxelized radiance fields. GaussianGPT comprehensively outperforms DiffRF in shape synthesis (FID 5.68 vs. 15.95), proving that the autoregressive paradigm is fully capable of catching up to and exceeding diffusion-based methods in 3D generation.
• vs G3PT / VAR-3D: These are also autoregressive 3D generation methods but operate on structured geometric tokens (such as point clouds or mesh faces). GaussianGPT is the first to extend autoregression to scene-level generation using unstructured 3D Gaussian primitives.

Rating

• Novelty: ⭐⭐⭐⭐ Formulates and validates the first purely autoregressive paradigm for 3D Gaussian scene generation, utilizing sensible design choices (split vocabularies, 3D RoPE, LFQ) rather than a naive application of GPT. However, autoregressive 3D generation is not an entirely new concept, having been explored in works like MeshGPT.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Covers four distinct tasks: shape synthesis, scene generation, completion, and large-scale outpainting. Comprehensively evaluates against multiple baselines, ablates serialization strategies and core components, and provides extensive analysis on inference efficiency, real-world data, and autoencoder designs in the appendix.
• Writing Quality: ⭐⭐⭐⭐ Well-structured with clear motivations, detailed methodology, and rich information in the appendix. However, some acronyms are not defined clearly on their first usage, and some tables recovered from LaTeX are slightly cluttered.
• Value: ⭐⭐⭐⭐ Pioneers an alternative generation pathway complementary to diffusion for 3D scene generation. Its inherent strengths in completion and controllable generation are highly attractive for downstream applications, and its engineering recipes (LFQ, split vocabularies, 3D RoPE) offer solid benchmarks for other 3D tokenization efforts.

Related Papers

• [CVPR 2026] Repurposing 3D Generative Model for Autoregressive Layout Generation
• [ICLR 2026] AssetFormer: Modular 3D Assets Generation with Autoregressive Transformer
• [CVPR 2026] ARMFlow: AutoRegressive MeanFlow for Online 3D Human Reaction Generation
• [ICLR 2026] QuadGPT: Native Quadrilateral Mesh Generation with Autoregressive Models
• [CVPR 2026] AutoRegressive Generation with B-rep Holistic Token Sequence Representation