GaussianGPT: Towards Autoregressive 3D Gaussian Scene Generation¶

Conference: ECCV 2026
arXiv: 2603.26661
Code: None
Area: 3D Vision
Keywords: 3D Gaussian Splatting, autoregressive generation, scene generation, vector quantization, Transformer

TL;DR¶

GaussianGPT compresses 3D Gaussian scenes into discrete token sequences and performs token-by-token autoregressive prediction using a causal Transformer with 3D rotary position embeddings. It realizes the pure autoregressive paradigm for 3D Gaussian scene generation for the first time, achieving or surpassing state-of-the-art (SOTA) diffusion models in shape synthesis and scene generation, and inherently supporting scene completion and outpainting.

Background & Motivation¶

The dominant paradigm in current 3D generative modeling relies on diffusion models and flow matching, which model generation as a process of global denoising or continuous-time transformation, achieving strong visual fidelity in shape and scene synthesis. However, these methods suffer from a fundamental challenge: real-world scenes are constructed incrementally—designers sequentially layout, expand, and edit—whereas diffusion models refine the entire scene at once, making incremental editing, partial completion, and controllable generation unnatural. In contrast, autoregressive Transformers have demonstrated powerful capabilities in structured sequence modeling and controllable generation in language and vision domains, yet remain rarely explored in structured 3D scene synthesis.

The core obstacle lies in the lack of a natural 1D sequential order for 3D data. 3D Gaussian primitives are unstructured point sets containing continuous attributes such as position, opacity, scale, rotation, and color. They lack a canonical ordering and are difficult to directly input into discrete token sequence models. Indoor scenes also simultaneously require global structural consistency (room layout) and local compositional flexibility (object placement), which further increases modeling difficulty.

The paper's key insight is: to compress the 3D Gaussian scene into a discrete latent grid using vector quantization, serialize it into a 1D token stream using xyz traversal, and finally perform autoregressive generation using a causal Transformer with 3D spatial positional bias. Core Idea: To formulate 3D scene generation as a sequence prediction problem of "predicting the next occupied voxel position and its Gaussian features step-by-step based on the generated spatial context," enabling an autoregressive pathway beyond the diffusion paradigm.

Method¶

Overall Architecture¶

The goal of GaussianGPT is: given a 3D Gaussian scene (or an empty scene), to autoregressively generate the complete scene representation token-by-token. The entire pipeline is divided into two major stages: (1) Scene Compression—utilizing a sparse 3D convolutional autoencoder to map the continuous Gaussian scene to discrete latent grid tokens; (2) Autoregressive Modeling—serializing the latent grid into a token stream and modeling the joint distribution with a causal Transformer. The two stages are trained independently: the autoencoder is trained first to obtain high-quality discrete representations, followed by training the GPT on its latent space.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["3D Gaussian Scene<br/>Position/Opacity/Scale/Rotation/Color"] --> B["Sparse 3D Feature Grid<br/>Voxel Assignment + Encoding Head"]
    B --> C["Sparse 3D CNN Encoder<br/>Three Downsampling Stages"]
    C --> D["LFQ Discretization<br/>Binarization by Sign -> Codebook Index"]
    D --> E["Discrete Latent Grid<br/>Voxel Size 20cm"]
    E --> F["xyz Serialization<br/>Alternating Position and Feature Tokens"]
    F --> G["Causal Transformer<br/>3D RoPE + Separate Vocabulary"]
    G -->|Training: Teacher Forcing| H["Token-by-Token Prediction<br/>Cross-Entropy Loss"]
    G -->|Inference: Autoregressive Sampling| I["Generate Token Sequence"]
    I --> J["LFQ Decoding -> Sparse 3D CNN Decoder"]
    J --> K["Reconstructed 3D Gaussian Scene"]
    G -->|Partial Scene Prefix| L["Scene Completion / Outpainting<br/>Same Model without Modification"]

Key Designs¶

1. Sparse 3D Latent Encoding and LFQ Discretization: Bridging the Gap from Gaussian Scenes to Token Sequences

The prerequisite for feeding a continuous, unstructured 3D Gaussian scene into an autoregressive Transformer is to convert it into a discrete token sequence. This work proposes a three-stage compression pipeline: First, Gaussian primitives are assigned to voxel grids in world coordinates based on their positions. Absolute positions are replaced with offsets relative to voxel centers, and multiple Gaussians within the same voxel are randomly downsampled. Each Gaussian attribute is encoded by an independent encoding head (linear layer + residual MLP) and then concatenated into a unified feature vector, forming a sparse input feature grid. Next, a sparse 3D convolutional encoder is used with three downsampling stages (compressing voxel size from 2.5cm to 20cm) to compress the scene into a compact latent representation, preserving spatial locality and translation equivariance. Finally, Lookup-Free Quantization (LFQ) is employed to directly discretize the encoder output z into 0/1 based on sign, meaning the codebook index is determined by sign(z), avoiding explicit codebook searches.

The key reason for choosing LFQ over traditional VQ: traditional VQ requires nearest-neighbor search, which increases training time by approximately one-third and frequently suffers from codebook collapse; LFQ encourages uniform codebook utilization through an entropy maximization loss, with experiments demonstrating a 99.2% utilization rate for a 4096-entry codebook. The autoencoder training loss consists of three parts:

\[\mathcal{L} = \underbrace{\lambda_{RGB}\mathcal{L}_{RGB} + \lambda_{perc}\mathcal{L}_{perc}}_{\text{重渲染}} + \underbrace{\lambda_{occ}\mathcal{L}_{occ}}_{\text{占用}} + \underbrace{\lambda_{LFQ}\,\text{softplus}(\mathcal{L}_{LFQ}+5)}_{\text{码本熵}}\]

Here, the rendering loss computes RGB L1 and VGG-19 perceptual loss on sampled views, the occupancy loss supervises the occupancy prediction of decoder upsampling layers using BCE, and the codebook entropy loss ensures positive values through a softplus offset.

2. Separate Position-Feature Vocabularies and Alternating Sequences: Decoupling Geometric Structure and Appearance Modeling

When serializing the latent grid into a token stream, this work adopts a fixed xyz traversal order (minimum z first, i.e., column-by-column scanning). Each voxel corresponds to a pair of tokens: a position token (the relative index of the voxel within the current chunk) and a feature token (the quantized latent feature of the voxel), arranged in an alternating manner. The key design is that position and feature tokens utilize two independent vocabularies and output heads: the backbone Transformer is shared, but the position head predicts the next occupied voxel index during position steps, and the feature head predicts the quantized feature of the preceding position during feature steps.

The motivation for this is clear: geometric structure (where is occupied) and appearance (what the occupied space looks like) are two entirely different prediction tasks. Sharing a single vocabulary would force them to compete over finite codebook indices, confusing the model. Utilizing separate vocabularies eliminates this competition, allowing each to scale its codebook size independently. More importantly, this design naturally supports flexible control: the index range of the position head is bound to the chunk size, while the codebook size of the feature head is unaffected by chunk dimensions, enabling them to be tuned independently. During training, cross-entropy loss is computed only on the corresponding vocabulary at alternating steps, with invalid vocabulary entries masked out.

3. 3D Rotary Position Embeddings and a Fourth Token-Type Dimension: Enabling Attention to Perceive True Spatial Distances over Sequential Distances

Standard 1D positional encodings in Transformers (whether learnable absolute positions or RoPE) embed sequential ordering information. However, in xyz serialization, adjacent tokens in the sequence may be very far apart in 3D space (e.g., jumping to the next column after scanning one column), whereas adjacent voxels in 3D space can be separated by the length of an entire column in the sequence. Directly applying 1D RoPE forces the model to learn an incorrect "sequence proximity = spatial proximity" prior.

The solution in this work is to extend RoPE from 1D to 3D: in the attention computation, the rotation encodes the actual voxel coordinates (x, y, z) corresponding to the token, rather than the sequence position i. Consequently, the attention score becomes a function of relative 3D spatial offsets rather than sequential offsets—allowing the model to bypass long sequential distances and focus directly on spatially adjacent, already generated voxels. Furthermore, because position tokens and feature tokens occur alternately and share the same 3D coordinates, this work introduces a fourth token-type dimension: type=0 for position tokens and type=1 for feature tokens. This is encoded using an additional rotation frequency, helping the Transformer further decouple and distinguish geometric and appearance signals under a unified attention formulation.

4. Autoregressive Scene Completion and Outpainting: Same Model, Same Mechanism, Without Architectural Modifications

A natural advantage of the autoregressive paradigm is: generation equals predicting subsequent tokens based on a prefix. This means that given a partial scene, the model can autoregressively "continue writing" the remaining scene simply by serializing it as prefix tokens—unifying completion and unconditional generation within the same mechanism, rather than introducing specialized inpainting pipelines like RePaint as in diffusion models. The same principle extends to large-scale scene outpainting: the model is trained with a fixed chunk size (limited by the context window). During inference, a sliding window strategy is applied—generating one new column at a time and moving the window forward while using the already generated area as context. Repeating this process at inference time overcomes the chunk size limit used during training, enabling the synthesis of large 12m x 12m scenes (while the training chunk is only 4m x 4m).

Two practical inference heuristics are also introduced: (1) Occupied Position Masking: since position tokens directly correspond to voxel indices inside the chunk, already occupied positions can be masked during generation to ensure that sampling strictly follows order constraints; (2) Backtracking Resampling: in large-scale scene generation, if an empty column without any geometry is predicted, the model backtracks to the previous token and resamples (default up to 5 retries), which effectively suppresses generative degradation and maintains scene continuity.

Loss & Training¶

Training is conducted in two independent stages. Autoencoder Stage: Adam optimizer is used with a learning rate of 10^{-4} and cosine decay to 10% of the initial value. It is trained on 4 RTX A6000 GPUs, with an effective batch size of 8 for scenes (approx. 4 days) and 24 for PhotoShape objects (approx. 2 days). During training, chunks satisfying a minimum occupancy threshold (0.2) are randomly sampled, and camera sampling prioritizes viewpoints with greater visibility of the chunk. GPT Stage: Based on nanochat (GPT-2 backbone), combining AdamW and Muon optimizers with a per-module learning rate (0.004 for output heads, 0.2 for embeddings, 0.005 for residual weights, etc.) and cosine decay to 10%. It is trained on 4 GH200 GPUs with an effective batch size of 64, taking approx. 1 day for scenes and 4.5 hours for objects. The loss is standard autoregressive cross-entropy. During inference, temperature=0.9 and Nucleus Sampling with p=0.9 are utilized.

Key Experimental Results¶

Main Results¶

Shape Synthesis (PhotoShape Chairs): On the unconditional chair generation task, GaussianGPT achieves FID of 5.68, KID of 0.00184, and COV of 67.40%, comprehensively outperforming the previous state-of-the-art method L3DG (FID 8.49, KID 0.00315, COV 63.80%) and the diffusion method DiffRF (FID 15.95). Its MMD is on par with L3DG (4.278 vs 4.241), demonstrating that the autoregressive model matches or exceeds diffusion methods in both visual quality and geometric diversity.

Method	FID ↓	KID ↓	COV ↑	MMD ↓
π-GAN	52.71	0.01364	39.92	7.387
EG3D	16.54	0.00841	47.55	5.619
DiffRF	15.95	0.00794	58.93	4.416
L3DG	8.49	0.00315	63.80	4.241
GaussianGPT	5.68	0.00184	67.40	4.278

Scene Synthesis (3D-FRONT chunk): For unconditional generation, GaussianGPT performs significantly better than L3DG in appearance (FID 94.85 vs 100.98) and layout (FID 84.14 vs 93.84). However, it falls slightly short in geometric diversity and distribution matching (COV 0.548 vs 0.692, MMD 0.118 vs 0.096)—reflecting the "clean but conservative" tendency of autoregressive methods: the generated geometry is tidier with fewer anomalous primitives, but sample diversity is lower than that of diffusion models. Nevertheless, when shifting the task from unconditional generation to completion, the advantages of the autoregressive approach become evident: under 25% prefix completion, GaussianGPT comprehensively outperforms L3DG + RePaint across all dimensions (appearance, geometry, layout, and CLIP similarity). As the context increases (50%), the performance gap widens further.

Task	Method	Appearance FID ↓	Geometry COV ↑	Layout FID ↓	CLIP-Sim ↑
Unconditional Generation	L3DG	100.98	0.692	93.84	—
Unconditional Generation	Ours	94.85	0.548	84.14	—
25% Completion	L3DG	109.35	0.925	112.57	0.834
25% Completion	Ours	100.06	0.879	95.60	0.841
50% Completion	L3DG	106.54	0.940	111.71	0.843
50% Completion	Ours	98.13	0.940	90.77	0.851

Ablation Study¶

Serialization Strategies (Table 3): Comparing xyz, Z-order, Hilbert curves, and transposed variants, xyz traversal achieves the lowest training CE (2.151). Transposed Z-order yields a close validation CE (2.423 vs. 2.421), but orderings with stronger spatial locality guarantees (such as Hilbert curves) do not yield performance gains. This indicates that 3D RoPE already provides sufficient spatial information, reducing the reliance on sequential locality.

Serialization Strategy	Training CE ↓	Validation CE ↓
Z-order	2.203	2.432
Transposed Z-order	2.204	2.423
Hilbert	2.347	2.439
Transposed Hilbert	2.334	2.434
xyz (Ours)	2.151	2.421

Ablation of Core Components (Table 4): Utilizing a shared vocabulary increases the validation CE from 2.421 to 2.449. Learnable absolute positional encoding performs worse (validation CE of 2.461). Similarly, 1D RoPE is inferior to 3D RoPE (validation CE of 2.446). All three designs independently contribute to the final performance.

Config	Training CE ↓	Validation CE ↓
full model	2.151	2.421
w/ Shared Vocabulary	2.157	2.449
w/ Learnable PE	2.210	2.461
w/ 1D RoPE	2.227	2.446

Key Findings¶

Separate vocabularies are the most cost-effective design: A shared vocabulary seems simplified but forces geometry and appearance to compete over finite codebook indices, increasing the validation CE by 0.028 and making it impossible to independently control chunk size and codebook size during inference.
3D RoPE makes simplified xyz sorting sufficient: Even though the 3D locality of xyz traversal is poor (adjacent tokens in the sequence can be far apart in space), 3D RoPE enables the model to attend directly based on spatial coordinates, bypassing the need for more complex space-filling curves.
The inference efficiency of the autoregressive approach is advantageous in completion scenarios: During unconditional generation, autoregressive sampling is slower than diffusion (78s vs. 24s/chunk), but under 50% prefix completion, the inference time drops to 39s (while diffusion remains unchanged at 24s)—owing to the reduced number of tokens to generate.
The quality of large-scale scene generation degrades more severely along the x-direction: xyz serialization causes expansions along x to require larger sequence jumps, rendering the accumulated error more significant. Conversely, expansion along y is relatively local, showing slower quality degradation.

Highlights & Insights¶

Decoupling geometry and appearance with "alternating sequences + separate vocabularies": This is an elegant design decision—position tokens and feature tokens appear alternately in the sequence but are predicted using different heads and vocabularies. This essentially decomposes the joint problem of "what is where" into two conditionally independent sub-problems. This scheme can be transferred to any spatial generation task that requires joint modeling of "existence" and "attributes" (such as point clouds, voxels, and graph structures).
3D RoPE + a fourth token-type dimension: Extending RoPE from the 1D sequence dimension to 3D spatial coordinates, while adding a type dimension to distinguish token types, preserves the relative positional encoding benefits of RoPE (such as length extrapolation) while freeing the model from the spatial distortions introduced by serialization. This design can be transferred to any scenario requiring the serialization of high-dimensional structures (e.g., spatiotemporal positional encodings in video tokenization, node positional encodings in graph serialization).
The unification of "autoregression as completion" is a highly underrated contribution: While diffusion models require external inpainting workflows like RePaint to perform completion, autoregressive models inherently handle both unconditional generation and completion under the same mechanism—which is a structural advantage of the paradigm rather than a trick. Experiments also confirm this: the more context provided, the more pronounced the advantages of the autoregressive paradigm.
The practical value of LFQ replacing VQ: The codebook collapse and nearest-neighbor search overhead of traditional VQ are particularly severe in sparse, high-dimensional data like 3D scenes. The binarization mechanism of LFQ is simple, highly efficient, and achieves an exceptionally high codebook utilization rate (99.2%), offering direct engineering value for 3D tasks that require vector quantization.

Limitations & Future Work¶

Slow unconditional generation speed of autoregressive sampling: Unconditional generation takes 78s per chunk (whereas L3DG only requires 24s) because it relies on sequential token-by-token sampling. The authors outline several acceleration directions (Flash Decoding, speculative decoding, multi-token prediction heads, sparse attention), but these have yet to be implemented.
Quality degradation in large-scale scene outpainting: As the number of sampling steps increases, accumulated errors cause quality degradation in distant regions (with normalized KID values rising from 1.0 to 1.3-1.6). Backtracking resampling can only mitigate but not completely eliminate this degradation.
A trade-off between geometric diversity and diffusion methods: During unconditional scene generation, the COV is lower than that of L3DG (0.548 vs. 0.692), indicating that the structural variation of scenes generated by the autoregressive model is narrower, as it tends to predict the "most likely" layouts rather than "adequately diverse" layouts. This is an inherent limitation of autoregressive modeling (likelihood maximization), which may require incorporating multimodal modeling strategies beyond temperature adjustments.
Insufficient autoencoder fidelity on real-world data: ScanNet++ experiments reveal a bottleneck in which high-frequency details are poorly reconstructed and the missing/unobserved regions inherent in real scans cannot be faithfully modeled by the current pipeline. The authors point out that autoregressive models are naturally suited for managing uncertain regions (which can be masked) and suggest that uncertainty-aware modeling remains a promising avenue for future exploration.
Anisotropic degradation patterns caused by xyz traversal order: Outpainting along the x-direction degrades much more severely than along the y-direction, demonstrating that a fixed serialization order introduces directional bias. Future work can investigate adaptive or multi-path serialization strategies.

vs L3DG: L3DG serves as the foundation of the autoencoder in this work (the sparse 3D CNN architecture). However, while L3DG directly employs a diffusion model on the latent space for generation, this work replaces it with an autoregressive Transformer + LFQ. The core difference between the two lies in the generative paradigm: diffusion relies on global iterative refinement, whereas autoregression utilizes local sequential decision-making. Experiments show that both have their respective strengths—diffusion is stronger in unconditional diversity, while autoregression excels in completion and controllability.
vs MeshGPT / MeshAnything: These works conduct autoregressive mesh generation and fall under the same broad direction of "autoregressive 3D" as this work. However, their operating targets are mesh face sequences, which possess inherently regular topological structures. GaussianGPT operates on 3D Gaussian primitives, which are more general but also more unstructured, presenting a greater challenge—it must "artificially" construct a sequential structure through compression and quantization first.
vs DiffRF: A representative work of 3D generation using diffusion models, performing denoising on voxel radiance fields. GaussianGPT comprehensively outperforms DiffRF in shape synthesis (FID 5.68 vs. 15.95), showing that even with a later start, the autoregressive paradigm can catch up with and surpass diffusion models in 3D generation.
vs G3PT / VAR-3D: Also performing autoregressive 3D generation, but operating on structured geometric tokens (such as point clouds and mesh faces), GaussianGPT is the first to extend autoregression to scene-level generation of unstructured 3D Gaussian primitives.

Rating¶

Novelty: ⭐⭐⭐⭐ Demonstrates the feasibility of a pure autoregressive paradigm in 3D Gaussian scene generation for the first time, with deliberate design efforts (separate vocabularies, 3D RoPE, LFQ) rather than a simple application of GPT; however, autoregressive 3D generation is not an entirely new concept, as works like MeshGPT have already explored it.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Covers four tasks: shape synthesis, scene generation, completion, and large-scale outpainting. It compares against multiple baselines, ablates serialization strategies and core components, and provides solid auxiliary details in the appendix regarding inference efficiency, real-world data, and autoencoder ablations.
Writing Quality: ⭐⭐⭐⭐ The structure is clear, the motivation is sufficient, the methodology is described in detail, and the appendix is highly informative. However, the first definitions of some abbreviations are not prominent enough, and the table layouts are somewhat cluttered due to LaTeX-to-plaintext conversion.
Value: ⭐⭐⭐⭐ Opens up a technical pathway for 3D scene generation that is complementary to diffusion. The intrinsic advantages of completion and controllable generation may be more appealing in practical applications. The engineering designs (LFQ, separate vocabularies, 3D RoPE) offer valuable references for other 3D tokenization efforts.