LaGen: Towards Autoregressive LiDAR Scene Generation¶

Conference: ECCV 2026
arXiv: 2511.21256
Code: https://github.com/szzhou88/LaGen
Area: Autonomous Driving
Keywords: LiDAR scene generation, autoregressive generation, latent diffusion model, scene decoupled estimation, noise modulation

TL;DR¶

LaGen introduces the autoregressive framework to LiDAR scene generation for the first time. Starting from a single-frame point cloud and conditioning on 3D bounding boxes and ego-vehicle states, it generates high-fidelity, long-sequence LiDAR scenes frame-by-frame, supporting interactive closed-loop simulation.

Background & Motivation¶

LiDAR is a critical sensor for providing precise geometric information in autonomous driving systems. World models capable of generating LiDAR content hold immense value for applications such as data augmentation, closed-loop simulation, and safety-critical scenario evaluation. However, existing LiDAR generation efforts (e.g., LiDARGen, RangeLDM) primarily focus on learning dataset distributions, only synthesizing non-sequential single-frame point clouds. Recent 4D scene generation methods (e.g., UniScene, LiDARCrafter), though capable of generating continuous frames, mostly generate the entire sequence in one go, lacking interactivity. In closed-loop simulations, the ego-vehicle's decision at each step dynamically affects the scene content of the next frame in real time, and the one-shot generation paradigm cannot integrate such immediate decision feedback into the generation process. In contrast, although LiDAR prediction methods (e.g., 4D-Occ, ViDAR) can predict future frames, they require multi-frame historical inputs, whereas only the initial frame is available when closed-loop simulation starts. More importantly, these methods predict along predetermined trajectories, and errors accumulate rapidly over time, leading to severe performance degradation in long-term prediction tasks.

The Key Challenge between these two paths lies in: the generation paradigm lacks the temporal dimension and interactive feedback, while the prediction paradigm relies on multi-frame inputs and cannot handle decision-driven scene changes. Neither can support the "long-duration + interactive" LiDAR scene generation truly needed for autonomous driving simulation. Autoregressive generation—treating the output of the previous frame as the input for the next and iterating frame-by-frame—naturally fits this requirement but faces obvious challenges: generation errors accumulate frame-by-frame during the autoregressive process, and object-level detail generation is difficult to maintain with spatial consistency.

The Key Insight of this paper is: utilizing a latent diffusion model as the base, fusing the previous frame's LiDAR, 3D bounding boxes, and ego-vehicle states into each generation step through meticulously designed multi-modal conditional encoding. Concurrently, a Scene Decoupled Estimation (SDE) module is introduced to provide a dense foreground/background estimation of the current frame prior to generation, enhancing object details. A Noise Modulation (NM) module is also introduced to inject noise into the previous frame's features during training, forcing the model to learn smooth mappings and theoretically ensuring error convergence. Core Idea: Formulate LiDAR scene generation as an autoregressive diffusion process starting from a single frame. By leveraging SDE to provide object-level spatial priors and Noise Modulation to mathematically guarantee long-term error convergence, the method achieves high-fidelity, interactive, and closed-loop simulation-ready long-sequence LiDAR generation for the first time.

Method¶

The overall architecture of LaGen is based on the Latent Diffusion Model (LDM), which converts LiDAR point clouds into compact 2-channel range images (depth + intensity) via spherical projection to perform diffusion generation within the VAE latent space. The generation process proceeds autoregressively frame-by-frame: given the previous LiDAR point cloud \(P^{s-1}\), the current 3D bounding boxes \(B^s\), and the ego-vehicle state \(E^s\), the model first predicts the current frame's foreground/background point clouds using the Scene Decoupled Estimation (SDE) module. All these 3D inputs are then projected into range images, encoded into the latent space via VAE, processed by the Noise Modulation (NM) module, and finally denoised by a UNet guided by multiple conditions to reconstruct the current frame's LiDAR. The output then becomes the input for the next frame.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Previous LiDAR<br/>P^{s-1}"] --> B["Scene Decoupled Estimation<br/>SDE"]
    C["Bounding Boxes B^{s-1}, B^s"] --> B
    B --> D["Foreground Point Cloud Estimation<br/>+ Background Point Cloud Estimation"]

    A --> E["Spherical Projection →<br/>Range Image"]
    D --> E
    E --> F["VAE Encode →<br/>Latent Space"]

    F --> G["Noise Modulation NM<br/>(Noise Injection to Prev Latent)"]
    G --> H["UNet Conditional Denoising<br/>cross-attention injection"]

    C --> I["Bounding Box →<br/>Range Mask → Encode"]
    I --> H
    J["Ego State Encode"] --> H

    H --> K["VAE Decode"]
    K --> L["Current LiDAR<br/>P^s"]
    L --> A

Overall Architecture¶

The generation loop of LaGen consists of several phases. First, the previous frame's LiDAR point cloud \(P^{s-1}\) and its corresponding 3D bounding boxes \(B^{s-1}\) are fed into the Scene Decoupled Estimation (SDE) module. SDE decouples the point cloud into foreground (within bounding boxes) and background (outside bounding boxes), and combines them with the current frame's bounding boxes \(B^s\) to estimate the expected positions of foreground objects and the rotated background coordinates. It outputs the estimated foreground and background point clouds for the current frame. Meanwhile, the previous frame's LiDAR and the SDE-estimated foreground/background point clouds are independently projected into 2-channel range images (depth + intensity). A weight-shared VAE encoder maps them into the latent space.

In the latent space, the Noise Modulation (NM) module injects random Gaussian noise of controllable magnitude into the latent features of the previous frame's LiDAR—this operation is executed on the conditional features during both training and inference without changing the sampling method of generative noise. The perturbed features are concatenated with the latent features of the SDE's foreground/background estimations as UNet input. Concurrently, the current 3D bounding boxes are projected as binary category maps on range images (different semantic classes represented by different channels) and encoded into mask latent features \(H_B\) via a lightweight VAE. The ego-vehicle states (velocity, acceleration, steering angle, coordinate transformation matrix) are encoded into compact token embeddings. Both are injected into various layers of the UNet via cross-attention to provide multi-modal guiding conditions. The UNet outputs denoised latent representations, which are decoded by the VAE into range images and reconstructed into 3D point clouds \(P^s\) via inverse spherical projection. During inference, \(P^s\) directly serves as the "previous LiDAR" for the next frame, and the SDE re-estimates based on \(P^s\) and the next frame's bounding boxes, forming a complete autoregressive closed-loop.

Key Designs¶

1. Multi-modal Conditional Encoding and Injection: Unifying Heterogeneous Control Signals into the Diffusion Process

Autoregressive LiDAR generation needs to utilize multiple control conditions simultaneously: the previous frame's LiDAR point cloud provides scene geometric continuity, 3D bounding boxes depict the precise spatial positions and semantic categories of individual objects, and the ego-vehicle state determines the sensor's perspective shift. How to effectively encode and inject this heterogeneous, high-dimensional information into the diffusion UNet is the first key challenge. For the previous frame's LiDAR, LaGen projects it into a range image, encodes it into the latent feature \(z^{s-1}\) using a VAE, and then utilizes a FiLM (Feature-wise Linear Modulation) layer to perform an affine transformation on \(z^{s-1}\) based on the sensor coordinate transformation matrix between the two frames, aligning its perspective to the current frame's coordinate system. For 3D bounding boxes, instead of simply using the 8 vertex coordinates as conditions—which would omit the spatial structure within the boxes—LaGen extracts the 4 bottom corners of each box, interpolates them to obtain a dense bounding point cloud, projects these point clouds of different semantic categories into multi-channel binary masks, and encodes them into mask latent features \(H_B\) via a small VAE to be injected into the UNet via cross-attention. Notably, the cross-attention keys are derived from the previous box encoding, while the values are from the current box encoding, forcing the UNet to reference past object positions while predicting current targets during denoising. The ego-vehicle states are processed via an MLP and timestep embeddings into a single token, which is similarly injected using cross-attention. This stratified encoding scheme allows each condition to enter the diffusion process in its most natural representation (geometric continuity \(\rightarrow\) FiLM alignment, spatial category \(\rightarrow\) mask encoding, motion state \(\rightarrow\) token embedding), avoiding information interference from raw concatenation.

2. Scene Decoupled Estimation (SDE): Providing Object-level Spatial Priors via Foreground/Background Decoupling

Relying solely on bounding box masks only provides coarse locations, causing the UNet to lose dense point cloud distributions and geometric details inside the boxes. The core idea of SDE is: since the previous frame already contains the precise point distributions of each object, and object motion between adjacent frames is relatively small, one can project the expected positions of these objects in the current frame based on center offsets. This dense "object-level estimate" serves as an additional condition for the diffusion process. Specifically, SDE splits \(P^{s-1}\) into foreground \(P_{obj}\) and background \(P_{bg}\) based on \(P^{s-1}\) and \(B^{s-1}\). For each foreground point cloud \(p_{ij}\), it first applies the relative sensor transformation matrix \(E_{rel}\) to align the perspective to obtain \(\hat{p}_{ij}\). Then, it searches for the closest bounding box of the same class in the current frame and estimates displacement using the difference between the centers (\(C^s - C^{s-1}\)), yielding \(\tilde{p}_{ij}^s = \hat{p}_{ij}^{s-1} + (C_{ij}^s - C_{ij}'^{s-1})\). These foreground estimates are aggregated into \(\tilde{P}_{obj}^s\). The background estimation is straightforward: since the background only rotates relative to the sensor frame due to ego-motion, it is transformed using only the rotation matrix. Finally, the foreground and background estimates are projected into range images, VAE-encoded, and used as additional inputs to the UNet. Compared to using only bounding box masks, SDE-guided estimation provides fine-grained geometric priors, preventing the diffusion model from "imagining" object internal structures from scratch and significantly reducing generation difficulty.

3. Noise Modulation (NM): Autoregressive Regularization for Theoretically Guaranteed Error Convergence

The greatest challenge in autoregressive generation is train-inference mismatch: during training, the model always conditioned on ground-truth previous frames, but during inference, the previous frames are self-generated and inherently contain errors. These errors accumulate and amplify over time, causing long-term degradation. The Noise Modulation module addresses this issue from a concise mathematical perspective: it perturbs the previous frame's conditional latent features \(z^{s-1}\) with a random intensity of Gaussian noise \(\hat{z}_\epsilon^{s-1} = \sqrt{\bar{\alpha}_n}\hat{z}^{s-1} + \sqrt{1-\bar{\alpha}_n}\epsilon\), where the noise level \(n\) is uniformly sampled from \([0, N]\). Through polynomial Taylor expansion, it is proved that this perturbation is equivalent to implicitly introducing a regularization term on the Frobenius norm of the UNet's Jacobian matrix with respect to the input conditions. In other words, training with noise forces the UNet to learn a smoother mapping—ensuring that slight perturbations in the input conditions (i.e., self-generated errors) do not cause drastic output changes. From the perspective of error propagation, this constrains the spectral radius of the Jacobian matrix to be less than 1, thereby preventing exponential error growth and guaranteeing error convergence. During inference, the NM module similarly injects noise into the conditional features (including the previous LiDAR's latent representation and SDE outputs) to align the distribution and maintain stability. This module has minimal impact on short-term generation (where historical errors are low) but yields increasingly pronounced improvements for long-term generation (exceeding 20 frames).

Loss & Training¶

The core loss function is the standard LDM noise prediction loss, defined as \(L_{LDM} = E_{z, \epsilon, t, c} [||\epsilon_t - \epsilon_\theta(z_t; t, c)||^2]\). During training, the previous frame's data is sampled from the ground-truth dataset. The model is trained on the nuScenes dataset for 200 epochs (176,000 steps total) with a batch size of 32, a learning rate of 1e-4 under a cosine annealing scheduler, taking approximately 23 hours on 4 NVIDIA H200 GPUs. The UNet consists of 6 downsampling/upsampling blocks (some incorporating Transformer layers). The VAE utilizes pre-trained weights from RangeLDM, with a range image resolution of 1024×32 and a VAE downsampling factor of 4. Since range images have a circular structure (0 and 360 degrees azimuth are spatially adjacent), all standard convolutions in the VAE and UNet are replaced with circular convolutions to handle boundary periodicity. Inference utilizes a DDIM sampler with 50 denoising steps, generating a single frame in approximately 0.43 seconds.

Key Experimental Results¶

Main Results¶

Metric	Baseline Methods	SOTA Results	LaGen	Gain
nuScenes MMD (\(10^{-4}\))	LiDARCrafter (AAAI'26)	1.52	0.11	92.8%
nuScenes JSD (\(10^{-2}\))	LiDARCrafter	5.43	2.77	49.0%
KITTI-360 MMD (\(10^{-4}\))	LiDARGen (ECCV'22)	13.36	1.41	89.4%
KITTI-360 JSD (\(10^{-2}\))	OpenDWM (CVPR'25)	12.93	5.75	55.5%
Downstream Detection mAP	UniScene (CVPR'25)	0.51	0.52	+0.01
Downstream Detection NDS	UniScene	0.47	0.49	+0.02
Object CD (All Categories)	UniScene	0.939	0.855	8.9%
Object EMD (All Categories)	UniScene	1.245	0.822	34.0%
CTC (4 frames)	LiDARCrafter	4.81	2.22	53.8%

LaGen outperforms all existing methods in single-frame generation quality by a large margin—on nuScenes, the MMD is reduced to 0.11 (previous SOTA was 1.52) and the JSD is reduced to 2.77 (previous SOTA was 5.43). The absolute values of these metrics indicate that the generated BEV point cloud distributions are almost indistinguishable from real data. It also maintains a significant advantage on cross-dataset evaluation (KITTI-360, 64-channel LiDAR), demonstrating robust generalization. Regarding temporal consistency, the CTC metric reaches 2.22 over 4 frames, significantly outperforming LiDARCrafter's 4.81, which validates the advantage of frame-by-frame autoregressive generation over one-shot generation.

Ablation Study¶

Configuration	CTC (2 frames)	CTC (4 frames)	[email protected] (\(\text{m}^2\))	[email protected] (\(\text{m}^2\))	Description
Full LaGen	0.60	2.22	1.36	2.16	Full model
w/o SDE	0.71	2.49	1.42	2.23	CTC and CD degrade across the board without SDE
w/o NM	0.73	2.77	1.38	2.82	Long-term errors escalate significantly without NM

Inference Steps	[email protected]	[email protected]	Time/Frame
10 steps	0.64	2.60	0.21s
50 steps	0.50	2.34	0.43s
200 steps	0.50	2.20	1.44s

Key Findings¶

The NM module contributes the most, especially in long-term generation: At 4 frames, the CTC without NM is 2.77 vs. 2.22 with NM; at 8.0s and 9.5s, the gap doubles (CD from 2.82 down to 2.16), proving that NM's suppression of error accumulation becomes increasingly significant over time.
SDE has a greater impact on shorter frame intervals: The improvement in CTC for frames 1-2 (0.71 \(\rightarrow\) 0.60) is larger than that for subsequent frames, as SDE-derived object-level priors are highly effective for fine-grained alignment between adjacent frames.
Diminishing returns in inference denoising steps: Increasing steps from 10 to 50 yields significant gains ([email protected]: 2.60 \(\rightarrow\) 2.34), but moving from 50 to 200 offers minimal improvement, confirming that 50 steps achieves an optimal quality-speed trade-off.
Interactive editing capability: LaGen supports editing bounding boxes in intermediate frames (moving or deleting objects). Point clouds on the modified objects are reconstructed correctly, and formerly occluded regions are automatically filled—a capability missing in one-shot generation methods like LiDARCrafter.

Highlights & Insights¶

First to adopt frame-by-frame autoregressive paradigms in LiDAR scene generation: In contrast to rendering the entire sequence at once, the autoregressive approach seamlessly incorporates closed-loop feedback at each simulation step, which is a key milestone towards interactive world models.
SDE's innovation lies in using previous frames' dense point clouds to describe the current frame: Traditional methods only feed bounding box coordinates or sparse masks into the model. SDE directly shifts thousands of point clouds from the previous frame and inputs them to the UNet, providing a detailed "sketch" rather than a simple description, significantly reducing the burden of imagination.
NM formulates error accumulation as Jacobian regularization: Underneath training with noise lies a robust mathematical proof—implicitly constraining the smooth mapping of the network to ensure that recursive errors do not explode. This approach is reminiscent of adversarial training interpretations, blending simplicity with theoretical depth.
Natural support for interactive editing: Since each frame is generated conditioned on bounding boxes, users can insert, delete, or move bounding boxes at any step to alter the scene dynamically. This capability holds high practical value for safety-critical corner case simulation.

Limitations & Future Work¶

Sensor limitations: LaGen is primarily designed for single-return spinning mechanical LiDARs (like nuScenes 32-channel). It does not explicitly model multi-return or solid-state (non-spinning) LiDARs, which require exploring alternative LiDAR representations.
SDE matching issues in crowded scenes: The nearest-neighbor object matching strategy in SDE can fail in highly dense scenes. Incorporating tracklet information from simulator feedback for inter-frame tracking could mitigate this in the future.
Computational overhead: Generating a single frame takes approximately 0.43 seconds (50-step DDIM). While near real-time, there remains a gap to the 10Hz or 20Hz required for real-time closed-loop simulation. Acceleration methods (e.g., distillation or fewer-step solvers) are crucial for deployment.
Completeness of evaluation metrics: Although temporal consistency relies heavily on geometric metrics like CD and CTC, there is a lack of semantic evaluation for dynamic objects (e.g., whether vehicles follow traffic rules). Future work could introduce detection- or trajectory-based semantic metrics. Nevertheless, its performance on downstream detection tasks offers a preliminary validation of practical feasibility.

vs LiDARCrafter (AAAI'26): LiDARCrafter also explores autoregressive generation, but it pre-computes trajectory control signals (bounding box sequences) to render the 3D sequence in one shot, lacking frame-by-frame interaction. LaGen accepts bounding box inputs step-by-step from the simulator, naturally supporting closed-loop closed-loop simulation. In terms of generation quality, LaGen reduces MMD to 0.11 (vs. LiDARCrafter's 1.52), outperforming it by more than order of magnitude.
vs UniScene (CVPR'25): UniScene utilizes occupancies as intermediate representations for sequence generation but does not support step-by-step interaction. LaGen consistently outperforms UniScene in temporal consistency (CTC).
vs 4D-Occ (CVPR'23): As a representative LiDAR prediction baseline, 4D-Occ requires 2-6 historical frames and suffers from rapid trajectory drift; at 9.5s, its CD degrades to 55.47 (2-frame input) and 21.08 (6-frame input), whereas LaGen relies on only 1 frame of input and maintains a CD of 2.34. This underscores the limitations of purely predictive models with fixed dynamics assumptions in long-duration tasks.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ First to introduce a frame-by-frame autoregressive framework for LiDAR generation; SDE and NM modules exhibit both theoretical and engineering elegance.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Extensively validated across generation quality, temporal consistency, long-term prediction, downstream detection, cross-dataset generalization, and interactive editing, paired with detailed ablation and runtime analyses.
Writing Quality: ⭐⭐⭐⭐ Method details are clearly described, theories are backed by solid proofs in the appendix, and diagrams are intuitive; however, some formulas are dense, and there is minor repetition between the main text and appendix.
Value: ⭐⭐⭐⭐⭐ Directly addresses the critical demand for closed-loop autonomous driving simulation, and the autoregressive interactive paradigm points towards next-generation LiDAR world models. Code is open-sourced.