ReinDriveGen: Reinforcement Post-Training for Out-of-Distribution Driving Scene Generation

Conference: ECCV 2026
Paper: ECCV Official
Project Page: https://drive-sim.github.io/ReinDriveGen/
Area: Autonomous Driving
Keywords: driving scene generation, reinforcement post-training, video diffusion model, point cloud completion, out-of-distribution editing

TL;DR

ReinDriveGen couples dynamic point cloud completion with a condition-guided video diffusion model and introduces an RL-based post-training paradigm with a dedicated pairwise preference model and pairwise win-rate rewards, substantially enhancing the photorealism of out-of-distribution safety-critical driving scenarios without ground-truth supervision.

Background & Motivation

Photorealistic driving scene simulation is indispensable for developing, validating, and stress-testing autonomous driving systems. A high-fidelity simulator must allow users not only to synthesize novel ego viewpoints, but also to freely manipulate actor trajectories to produce safety-critical corner cases such as sudden cut-ins, head-on collisions, high-speed vehicle drifting, and out-of-control spinning. However, prevailing driving simulators—whether based on 3D scene reconstruction (e.g., 3D Gaussian Splatting) or conditional video diffusion models—are trained or optimized strictly on recorded driving logs. When users perform arbitrary, large-scale edits to actor trajectories or render from aggressively shifted ego viewpoints, the conditioning inputs inevitably drift far outside the training distribution.

The core tension of this out-of-distribution (OOD) regime is that safety-critical corner cases are virtually absent from normal driving logs, and no paired ground-truth videos can ever exist for counterfactual or edited maneuvers. Consequently, standard supervised fine-tuning (SFT) is fundamentally incapable of bridging this distribution gap. Among all scene components, dynamic vehicles represent the primary target of trajectory editing and therefore suffer the most severe domain shifts, manifesting severe geometric distortion, texture blur, and implausible lighting that form the critical bottleneck for controllable simulation.

This paper addresses the challenge by connecting data-driven generation with reinforcement learning: when ground-truth supervision is unavailable, an RL post-training framework directly explores and optimizes sample fidelity across the OOD conditioning space. Core idea: condition a video diffusion model on pseudo-images rendered from dynamically aggregated and completed 3D point clouds, and leverage forward flow-matching reinforcement learning guided by an automated pairwise vehicle preference model to eliminate OOD distortions without ground-truth reference.

Method

Overall Architecture

The ReinDriveGen framework comprises two synergistic systems. The first is a Point Cloud-Conditioned Video Diffusion Simulator, which aggregates sparse multi-frame LiDAR sweeps into category-aware dynamic 3D point clouds, reconstructs complete 360° geometry for partially occluded vehicles, and renders structural 2D pseudo-images to condition a video diffusion model. The second is an RL-Based Post-Training framework, which generates multiple candidate rollouts under curated OOD conditions, scores them through a specialized pairwise preference model to calculate win-rate rewards, and optimizes the policy via contrastive flow matching.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    A["Raw multi-frame LiDAR sweeps + 3D bounding boxes"] --> B["Dynamic point cloud aggregation and vehicle completion<br/>AdaPoinTr reconstructs full 360° vehicle geometry"]
    B --> C["Scene trajectory editing and pseudo-image rendering<br/>Projected 2D structural pseudo-image sequences"]
    C --> D["Conditional video diffusion model generation<br/>VACE-1.3B integrates reference frame and pseudo-images"]
    D --> E["OOD condition candidate rollout generation<br/>Sample N=16 video candidates per condition"]
    E --> F["Pairwise preference model<br/>DINOv3 CLS tokens with antisymmetric scoring"]
    F --> G["Pairwise reward mechanism and policy optimization<br/>Area-weighted win rates + DiffusionNFT contrastive updates"]

Key Designs

1. Dynamic point cloud aggregation and vehicle completion: eliminating unobserved holes during trajectory editing

Pure 2D video diffusion models struggle to maintain strict multi-view 3D consistency, while raw single-frame LiDAR points are too sparse to provide robust spatial guidance. ReinDriveGen accumulates all 200 frames of static background into a shared world frame, integrates \(\pm 2\) frames for deformable agents (pedestrians and cyclists), and transforms vehicle LiDAR points into normalized object canonical frames using annotated 3D bounding boxes before colorizing them with reprojected RGB pixels. Nonetheless, captured vehicles are often severely occluded, revealing only single-side surfaces. Repositioning them along new paths or observing them from novel angles exposes unobserved regions as hollow geometric voids. To resolve this, the authors introduce AdaPoinTr, fine-tuned on projected ShapeNet-55 car models, to complete full 360° vehicle geometry. Completed surfaces are left uncolored (assigned black) to delegate photorealistic texture and lighting synthesis to the downstream diffusion model, guaranteeing hole-free geometric consistency across arbitrary edits.

2. Pseudo-view masked conditional video diffusion: synthesizing realistic textures from appearance priors

Rendered pseudo-images inherently exhibit noise and discrete point cloud artifacts, and completed vehicle surfaces lack color. ReinDriveGen adopts VACE-1.3B as its video diffusion backbone, injecting an original reference frame alongside the sequence of rendered pseudo-images through its Video Condition Unit (VCU). The reference frame provides global appearance and environmental priors, while the pseudo-images provide per-frame geometric constraints. To make the model robust against novel viewpoints that expose untextured vehicle surfaces unseen during SFT, the authors implement pseudo-view masking during supervised training: randomly masking regions along depth discontinuities to emulate viewpoint-induced occlusion edges, coupled with random rectangular block masking. This encourages the diffusion network to hallucinate consistent textures, reflections, and shadow interactions on newly exposed surfaces using contextual priors.

3. Pairwise preference model: overcoming the lack of domain-specific vehicle rewards

RL post-training hinges on an accurate reward function, yet general-purpose aesthetic or image-quality models fail to penalize domain-specific driving artifacts such as warped vehicle chassis, distorted wheels, or mismatched reflections. ReinDriveGen observes that systematically subsampling the pseudo-image conditioning frequency (e.g., conditioning every frame, every 2nd frame, or every 4th frame) induces monotonic, controllable degradation in generated vehicle quality. Using this mechanism, the authors automatically extract ordered triplets \((i) \succ (ii) \succ (iii)\) of identical vehicles across identical frames and scenes in Waymo logs via YOLO26, augmented with asymmetric motion blur and elastic distortions. The preference model uses a shared DINOv3 ViT-H+ encoder to extract CLS representations \(z_1, z_2\), concatenates them in both permutations through a shared MLP head to produce logits \(s_{12}\) and \(s_{21}\), and defines the preference probability antisymmetrically:

\[P(I_1 \succ I_2) = \sigma\left(\frac{s_{12} - s_{21}}{2}\right)\]

This formulation enforces logical reciprocity \(P(I_1 \succ I_2) = 1 - P(I_2 \succ I_1)\) and circumvents the reward hacking and calibration drift typical of pointwise scalar scoring.

4. Pairwise reward mechanism and policy optimization: robust relative win rates for OOD training

To post-train on extreme OOD conditions (e.g., in-place spinning, sharp collision maneuvers, and vehicle rollovers), the reference policy generates \(N = 16\) candidate videos per condition. YOLO26 detects vehicle bounding boxes on the first candidate, and these identical spatial windows are applied across all \(N\) candidates to form \(M\) independent evaluation units. Within each unit \(m\), all pairwise combinations among the \(N\) candidate crops are evaluated with a confidence threshold \(\tau = 0.85\):

\[R_m^i = \frac{1}{N-1} \sum_{j \neq i} \mathbb{I}\Big[P(a_m^i \succ a_m^j) > \tau\Big]\]

The overall video reward \(\hat{R}(x_i)\) is computed as the area-weighted average of per-unit win rates across all \(M\) units, prioritizing visual fidelity on larger, closer vehicles. The RL training extends DiffusionNFT to video sequences, updating the policy via the CFG-free forward flow-matching objective:

\[\mathcal{L}(\theta) = \mathbb{E}_{\mathbf{c},\, \mathbf{x}_0 \sim \pi_{\mathrm{old}}(\cdot|\mathbf{c}),\, t} \left[ r \cdot \big\| v_\theta^+(\mathbf{x}_t, \mathbf{c}, t) - \mathbf{v} \big\|^2 + (1 - r) \cdot \big\| v_\theta^-(\mathbf{x}_t, \mathbf{c}, t) - \mathbf{v} \big\|^2 \right]\]

Using the scalar win-rate reward \(\hat{R}(x_i)\) in place of \(r\), the policy reinforces high-performing generation trajectories and suppresses low-quality artifacts without requiring paired ground-truth references.

Loss & Training

The framework follows a two-stage training scheme: 1. Supervised Fine-Tuning (SFT): Trained on 1,000 Waymo scenes using front, front-left, and front-right camera views. Initialized from VACE-1.3B, the DiT backbone and Context Adapter are fine-tuned jointly on paired pseudo-images and ground-truth videos with a constant text prompt ("A realistic autonomous driving scene"). Videos consist of 49 frames at \(480 \times 832\) resolution. Training uses the Adam optimizer with a constant learning rate of \(1 \times 10^{-5}\) and batch size 32 for roughly 500 H100 GPU hours. 2. RL-Based Post-Training: Conducted on 20 representative OOD conditions spanning in-place spinning, sharp turns, vehicle rollovers, and lateral ego viewpoint offsets. LoRA is applied to preserve policy stability. For each condition, groups of 16 candidate rollouts are evaluated via the pairwise reward mechanism. Optimization runs for 500 steps using Adam with a learning rate of \(1 \times 10^{-5}\), requiring approximately 400 H100 GPU hours.

Key Experimental Results

Main Results

In the off-trajectory novel ego-viewpoint benchmark (simulating a 4m cumulative lateral lane change over 40 frames at 0.1m/frame), ReinDriveGen is evaluated on Waymo against state-of-the-art reconstruction and generative approaches using Novel Trajectory Agent IoU (NTA-IoU), Novel Trajectory Lane IoU (NTL-IoU), and FID:

Model	NTA-IoU ↑	NTL-IoU ↑	FID ↓
PVG (IJCV 2026)	0.256	50.70	105.29
DriveDreamer4D w/ PVG	0.438	53.06	71.52
ReconDreamer w/ PVG	0.464	53.21	74.32
S3Gaussian (CVPR 2024)	0.175	49.05	124.90
DriveDreamer4D w/ S3Gauss.	0.495	53.42	66.93
ReconDreamer w/ S3Gauss.	0.413	51.62	123.61
Deformable-GS (2023)	0.240	51.62	92.24
DriveDreamer4D w/ Deform.-GS	0.335	52.93	77.32
ReconDreamer w/ Deform.-GS	0.443	53.78	76.24
ReinDriveGen (Ours)	0.549	56.13	51.99

In the vehicle trajectory editing setting (keeping ego trajectory fixed while applying a 5m lateral shift to a target vehicle across 5 StreetCrafter scenes), performance is evaluated with VBench metrics and FID:

Method	Motion Smoothness ↑	Background Consistency ↑	Image Quality ↑	FID ↓
Street Gaussians (ECCV 2024)	0.9811	0.9310	0.6398	84.7970
StreetCrafter (CVPR 2025)	0.9817	0.9069	0.6509	83.6278
ReinDriveGen (Ours)	0.9774	0.9397	0.6928	80.6267

Ablation Study

To track policy progression across training steps independently of group-relative win rates, the authors fix the random seed to 0 and compute an area-weighted pairwise win rate against the frozen SFT baseline \(\pi_{\mathrm{sft}}\):

\[E(\mathbf{c}) = \frac{\sum_{m=1}^{M} w_m \cdot P(a_m^{\mathrm{rl}} \succ a_m^{\mathrm{sft}})}{\sum_{m=1}^{M} w_m}\]

Configuration / Stage	Evaluation Score \(E(\mathbf{c})\)	Geometric Fidelity	Texture & Photorealism
Step 0 (SFT Baseline)	0.5000	Severe vehicle tearing, shearing, and stretching	Flat unnatural color patches and mismatched lighting
Step 60 (Mid-training)	0.6120	Vehicle contours stabilize; geometric tears diminish	Reflections begin aligning with environmental lighting
Step 300 (Converged RL)	0.7450+	Consistent rigid-body geometry even during 360° spin	Fine wheel hub details and sharp, coherent specularities
Ablation: w/o Vehicle Completion	- (Severe drop)	Catastrophic holes on newly exposed vehicle surfaces	Diffusion model hallucinates distorted color artifacts

Key Findings

• RL post-training drives true out-of-distribution generalization: Across speed doubling, in-place spinning, and vehicle rollovers, generation quality metrics improve monotonically with RL steps. While SFT baselines collapse into blurry, melting blobs on non-physical maneuvers absent from training logs, RL training systematically guides the diffusion model toward rigid-body coherence.
• Geometric completion is a prerequisite for diffusion synthesis: Omitting AdaPoinTr vehicle completion leaves wide point cloud holes that cause severe vehicle shearing and color bleeding, proving that generative models require dense spatial scaffolds to synthesize clean geometry under trajectory edits.
• Pairwise win rates prevent reward gaming: Evaluating full pairwise comparisons within candidate clusters and weighting by bounding-box area prevents the model from optimizing trivial background pixels and focuses optimization gradients squarely on visual vehicle fidelity.

Highlights & Insights

• First extension of forward-process diffusion RL to video generation: By leveraging DiffusionNFT's CFG-free formulation on the forward flow-matching path, the framework sidesteps the prohibitive compute and memory bottlenecks of spatio-temporal video RL rollouts.
• Label-free preference dataset construction via conditioning decay: Subsampling pseudo-image conditioning frames creates perfectly paired, controllable quality degradations across identical vehicle instances, offering an elegant blueprint for training driving-domain reward models without human annotation.
• Hybrid explicit-implicit representation: Combining explicit 3D point cloud scaffolds with implicit video diffusion priors overcomes the view-extrapolation limits of 3DGS while preventing the unconstrained stochastic background drift of pure video diffusion models.

Limitations & Future Work

• Temporal sequence length and inference latency: Due to GPU memory constraints, the current implementation is restricted to 49-frame clips (~few seconds of driving), with each rollout taking roughly one minute—rendering it unsuitable for real-time closed-loop testing.
• Deformable agent completion: The vehicle completion network specifically assumes rigid car models; extending complete geometric synthesis to deformable actors (pedestrians, cyclists, animals) during contact dynamics remains an open challenge.
• Full multi-camera surrounding consistency: Current experiments focus primarily on three front-facing cameras; expanding the framework to synchronized 360° surround-view rigs (6–8 cameras) requires handling wider cross-camera boundary transitions.

Related Work & Insights

• vs Street Gaussians / PVG: Reconstruction-based 3DGS methods fail when vehicles are shifted, exposing hollow background holes and static baked-in reflections; ReinDriveGen uses point completion to fill geometry and video diffusion to dynamically re-synthesize lighting and reflections.
• vs DriveDreamer / UniMLVG: Prior diffusion-based simulators conditioned only on bounding boxes or HD maps lack 3D point cloud scaffolds, causing severe structural degradation when executing maneuvers outside standard driving datasets (e.g., rollovers and spinning); ReinDriveGen's point-conditioned RL post-training ensures robust out-of-distribution generalization.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Pioneering introduction of RL post-training and pairwise preference learning to out-of-distribution driving scene generation.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive quantitative benchmarks on off-trajectory and vehicle-editing tasks, accompanied by rigorous step-wise ablation tracking.
• Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative, disciplined math formulations, and well-structured qualitative comparisons.
• Value: ⭐⭐⭐⭐⭐ Provides a concrete methodological breakthrough for generating safety-critical corner cases to validate autonomous driving foundation models.