U4D: Uncertainty-Aware 4D World Modeling from LiDAR Sequences¶

Conference: CVPR 2026 arXiv: 2512.02982 Code: N/A Area: Autonomous Driving Keywords: LiDAR generation, uncertainty modeling, diffusion models, 4D world model, spatio-temporal consistency

TL;DR¶

This paper proposes U4D, the first uncertainty-aware 4D LiDAR world modeling framework. It adopts a "hard-first, easy-second" two-stage diffusion generation strategy that first reconstructs high-uncertainty regions and then conditionally completes the entire scene. A MoST module is designed to adaptively fuse spatio-temporal features for temporal consistency.

Background & Motivation¶

LiDAR data acquisition bottleneck: Collecting large-scale, diverse, and annotated LiDAR data is extremely costly and labor-intensive, making generative LiDAR modeling an important avenue for data augmentation and pre-training.

Uniform assumption of existing methods: Methods such as LiDARGen, LiDM, and R2DM treat all spatial regions equally during generation, ignoring the non-uniform distribution of semantic difficulty in real-world scenes.

Existence of high-uncertainty regions: Sparse distant regions, occlusion boundaries, small-scale structures, and semantically ambiguous areas inherently exhibit high uncertainty in LiDAR observations; uniform generation leads to geometric artifacts and temporal instability in these regions.

Inspiration from human cognition: Humans first parse ambiguous regions before understanding global context. U4D draws on this insight by generating high-uncertainty regions as structural anchors before completing the remaining regions.

Insufficient temporal consistency modeling: Existing methods focus primarily on spatial reconstruction and insufficiently model inter-frame temporal coherence, resulting in unnatural object motion in generated sequences.

Downstream task requirements: Safety-critical applications such as autonomous driving require generated data that genuinely improves the robustness and calibration reliability of perception models, rather than merely pursuing visual fidelity.

Method¶

Overall Architecture¶

U4D adopts a two-stage "hard-first, easy-second" generation paradigm:

Stage 1: Uncertainty region modeling — A pre-trained LiDAR segmentation model (RangeNet++) estimates per-point uncertainty maps (Shannon entropy). The top-K high-entropy points form a sparse uncertainty point cloud, which is converted to range-view representation and reconstructed by an unconditional diffusion model.
Stage 2: Uncertainty-conditioned completion — The uncertainty regions generated in Stage 1 serve as conditional inputs to a conditional diffusion model that completes the full LiDAR frame, ensuring global structural consistency.
Both stages share a unified latent scene representation, allowing global context to back-refine local uncertainty regions.

Key Designs¶

1. Uncertainty measurement and representation

Per-point Shannon entropy is computed as: \(H(\mathbf{p}) = -\sum_{c=1}^{C} D_c(\mathbf{p}) \log D_c(\mathbf{p})\)
The top 20% high-entropy points are retained on nuScenes; the top 5% on SemanticKITTI.
The sparse uncertainty point cloud is projected to a range image \(\mathbf{x}_0^u \in \mathbb{R}^{H \times W \times 2}\) (depth + reflectance), accompanied by a binary mask \(\mathbf{m}^u\).

2. Uncertainty region diffusion model

An unconditional diffusion model \(\epsilon_\theta^u\) learns the generative distribution of uncertainty regions in range-view.
The standard DDPM forward process is adopted; the reverse denoising simultaneously reconstructs the spatial validity mask.

3. Uncertainty-conditioned completion

A conditional diffusion model \(\epsilon_\theta^c\) learns \(p(\mathbf{x}_0 | \mathbf{x}_0^u)\).
The noisy input \(\mathbf{x}_t\) and the uncertainty prior \(\mathbf{x}_0^u\) are concatenated along the feature dimension, enabling the network to leverage both global and local cues for denoising.

4. MoST (Mixture of Spatio-Temporal) module

Intermediate features \(\mathbf{F}_i \in \mathbb{R}^{C_i \times L \times H_i \times W_i}\) are decomposed into a spatial branch (\(1 \times 3 \times 3\) convolution for intra-frame geometry) and a temporal branch (\(3 \times 1 \times 1\) convolution for inter-frame dynamics).
The outputs of both branches are concatenated and passed through a shared MLP embedding, then adaptively fused via a mixture-of-experts-style gating mechanism: \((α_i^s, α_i^t) = \text{Softmax}(\mathbf{F}_i^{\text{share}} \cdot \mathbf{W}_i^g + \mathbb{I}(\chi \cdot \sigma(\mathbf{F}_i^{\text{share}} \cdot \mathbf{W}_i^z)))\)
Gaussian noise perturbation is applied to the gate during training to prevent deterministic overfitting.
Spatial branches dominate in the input/output layers (geometric detail), while temporal branches dominate in the intermediate layers (motion dynamics).

Loss & Training¶

Uncertainty stage loss: \(\mathcal{L}_u = \mathbb{E}[\|\epsilon^u - \epsilon_\theta^u(\mathbf{x}_t^u, t)\|_2^2] + \lambda \mathcal{L}_{\text{mask}}(\mathbf{m}^u, \mathbf{m}^p)\), where \(\mathcal{L}_{\text{mask}}\) is binary cross-entropy.
Conditional completion loss: \(\mathcal{L}_c = \mathbb{E}[\|\epsilon^c - \epsilon_\theta^c(\mathbf{x}_t, t, \mathbf{x}_0^u)\|_2^2]\)
Gate regularization: \(\mathcal{L}_{\text{reg},i} = \frac{\text{Var}(\alpha_i^s)}{(\mathbb{E}[\alpha_i^s])^2} + \frac{\text{Var}(\alpha_i^t)}{(\mathbb{E}[\alpha_i^t])^2}\), preventing the gate from collapsing to a single modality.
The two stages are trained separately: Stage 1 for 1M steps, Stage 2 for 500K steps; batch size 8, sequence length 6.
AdamW optimizer, learning rate \(1 \times 10^{-4}\), cosine annealing with 10K-step warmup; EMA decay 0.995.
4 × NVIDIA RTX 4090, FP16 mixed-precision training.

Key Experimental Results¶

Main Results¶

Table 1: nuScenes scene-level generation fidelity

Method	FRD ↓	FPD ↓	JSD ↓	MMD(×10⁻⁴) ↓
LiDARGen (ECCV'22)	549.18	22.80	0.04	0.76
R2DM (ICRA'24)	253.80	14.35	0.03	0.48
UniScene (CVPR'25)	-	976.47	0.32	13.61
U4D	223.96	12.90	0.03	0.53

Table 3: nuScenes temporal consistency (TTCE/CTC)

Method	TTCE-3 ↓	TTCE-4 ↓	CTC-1 ↓	CTC-3 ↓
UniScene (CVPR'25)	2.74	3.69	0.90	3.64
LiDARCrafter (AAAI'26)	2.65	3.56	1.12	3.02
U4D	2.63	3.51	0.97	2.98

Table 4: Downstream semantic segmentation mIoU (%)

Method	1% labels	10% labels	50% labels
Sup.-only	58.3	71.0	75.1
R2DM	64.1	73.0	75.9
U4D	65.3	73.7	76.4

Ablation Study¶

Uncertainty region selection strategy (Table 6)

Strategy	FRD ↓	FPD ↓	ECE(%) ↓
No uncertainty	235.91	14.03	3.98
Random sampling	235.23	13.21	4.35
Confidence sampling	228.24	13.04	3.02
Entropy sampling	223.96	12.90	2.72

MoST fusion strategy (Table 7)

Fusion method	FRD ↓	FPD ↓	JSD ↓
Cascade (no parallel)	536.23	23.34	0.63
Additive fusion	242.81	13.42	0.28
Concatenation fusion	242.43	12.51	0.03
Adaptive fusion	223.96	12.90	0.03

Key Findings¶

Entropy-based uncertainty selection substantially outperforms random selection and unconditional generation, reducing ECE from 3.98% to 2.72%.
Adaptive fusion yields a significant FRD improvement (~20 points) over simple additive or concatenation fusion, validating the effectiveness of dynamic gating.
U4D's generated data provides the largest downstream segmentation gain under the 1% annotation setting (+7.0 mIoU), indicating that uncertainty-aware generated data is most valuable in data-scarce scenarios.
MoST exhibits distinct spatio-temporal activation patterns across network depths: shallow and deep layers favor spatial branches, while intermediate layers favor temporal branches.

Highlights & Insights¶

First uncertainty-driven LiDAR generation framework: The uncertainty outputs of perception models are fed back into the generation process, forming a closed loop of "perception → uncertainty → generation → enhanced perception."
"Hard-first, easy-second" generation philosophy: Tackling semantically ambiguous difficult regions first and using them as anchors to complete simpler regions — a strategy with broad applicability.
Elegant MoST module design: Gating mechanism + noise regularization + coefficient-of-variation regularization collectively ensure balanced spatio-temporal feature fusion.
Downstream calibration experiments (ECE metric) demonstrate that the generated data not only improves accuracy but also enhances model confidence calibration.

Limitations & Future Work¶

Inference speed (8.9 s/frame) is higher than single-frame methods (R2DM: 3.5 s); efficiency remains to be improved.
Uncertainty estimation relies on the quality of the pre-trained segmentation model; biases in the segmentation model propagate into the generation process.
Validation is limited to nuScenes and SemanticKITTI; additional sensor configurations and indoor scenes have not been tested.
The two-stage training pipeline (1.5M steps total) incurs high training cost; an end-to-end approach may be more efficient.
Information loss of the range-view representation in self-occluded and distant regions is not sufficiently discussed.

LiDAR generation: LiDARGen (ECCV'22, score-based) → R2DM (ICRA'24, diffusion) → LiDARCrafter (AAAI'26, autoregressive temporal) → U4D (uncertainty-aware).
Uncertainty modeling: SalsaNext (Bayesian inference), Calib3D (depth-aware calibration) — U4D is the first to incorporate uncertainty into a generative framework.
Spatio-temporal modeling: ViDAR (image-predicted LiDAR), cascaded spatio-temporal modules in video diffusion — MoST proposes parallel decomposition with adaptive gating as an alternative to cascading.

Rating¶

Novelty: ⭐⭐⭐⭐ (Uncertainty-driven "hard-first" generation is a genuinely novel perspective)
Experimental Thoroughness: ⭐⭐⭐⭐ (Dual datasets, multiple metrics, complete ablations, downstream task validation)
Writing Quality: ⭐⭐⭐⭐ (Clear structure, polished figures, well-motivated)
Value: ⭐⭐⭐⭐ (The uncertainty + generation paradigm has practical significance for autonomous driving simulation and data augmentation)