ReconDreamer-RL: Enhancing Reinforcement Learning via Diffusion-based Reconstruction¶
Conference: ECCV 2026
Paper: ECCV Official Link
Project Page: https://recondreamer-rl.github.io/
Area: Autonomous Driving
Keywords: Autonomous Driving, Reinforcement Learning, Scene Reconstruction, Video Diffusion Models, Sim-to-Real
TL;DR¶
ReconDreamer-RL integrates video diffusion priors and kinematic physical constraints into 3D Gaussian Splatting scene reconstruction to build ReconSimulator, and couples it with a Dynamic Adversary Agent (DAA) and Cousin Trajectory Generator (CTG) to slash the closed-loop collision ratio of end-to-end autonomous driving to nearly one-third of prior RL baselines (and over 5x lower than imitation learning).
Background & Motivation¶
End-to-end autonomous driving models that map multi-view raw sensory inputs directly to planning actions exhibit superior potential over traditional modular pipelines. However, prevailing imitation learning paradigms (such as VAD and GenAD) rely strictly on expert demonstration datasets, making them prone to compounding errors and distribution drift when confronting unseen long-tail conditions during closed-loop testing. While reinforcement learning (RL) in closed-loop simulations offers an avenue to break this bottleneck, current simulators suffer from an acute trade-off between fidelity and explorable diversity. Game-engine-based simulators (like CARLA) lack sensor-level photorealism, inducing a large Sim-to-Real gap; conversely, 3D Gaussian Splatting (3DGS) neural scene reconstructions achieve high sensor fidelity but remain tightly bounded to recorded expert trajectories—any deviation results in severe background blur, broken lane markings, and artifacts that undermine downstream policy perception.
The deeper contradiction lies in environmental interactivity and data distribution sparsity. In real-world datasets like nuScenes, over 88% of driving sequences consist of benign straight-line cruising, while high-maneuver actions such as sharp turns or U-turns are heavily underrepresented. Moreover, critical corner cases like aggressive cut-ins and emergency braking are rarely captured in normal real-world logs. Existing 3DGS-based RL frameworks (such as RAD) explore within static reconstruction scenes that lack both novel-view perceptual fidelity and adversarial multi-agent traffic dynamics, leaving the learned policy ill-equipped to handle sudden hazardous interactions.
This paper addresses the challenge by leveraging the generative denoising and inpainting priors of video diffusion models to enhance 3DGS scene reconstruction while enforcing kinematic physical consistency, alongside targeted data synthesis engines. Core idea: construct ReconSimulator by harmonizing video diffusion priors for appearance modeling with kinematic bicycle constraints for physical dynamics, and integrate a Dynamic Adversary Agent (DAA) and Cousin Trajectory Generator (CTG) across two-stage imitation pretraining and closed-loop RL fine-tuning.
Method¶
Overall Architecture¶
The training pipeline of ReconDreamer-RL consists of two complementary stages: Stage 1 performs imitation learning (IL) warm-up via behavior cloning using expert nuScenes demonstrations, DAA-synthesized adversarial corner cases, and CTG-augmented non-straight trajectories; Stage 2 conducts closed-loop reinforcement learning (RL) fine-tuning where \(N\) parallel ReconSimulator instances collect interactive rollouts \((s_t, a_t, r_{t+1}, s_{t+1})\) for PPO optimization, periodically interleaved with imitation updates to preserve human-like smoothness.
The overall architecture integrates the appearance and physical simulator with dual data-generation modules:
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Multi-View Sensor Data + Vectorized HDMap"] --> B["ReconSimulator: Appearance & Physical Modeling<br/>3DGS FG/BG Decoupling + Diffusion Restoration + Kinematics"]
B --> C["Dynamic Adversary Agent (DAA)<br/>BEV Target Selection + Trajectory Generation + Feasibility Checks"]
B --> D["Cousin Trajectory Generator (CTG)<br/>Action Space Extension + Rare Trajectory Linear Interpolation"]
C --> E["Two-Stage End-to-End Policy Training<br/>Stage 1: Imitation Warm-Up / Stage 2: Closed-Loop PPO with IL Steps"]
D --> E
E --> F["Robust Closed-Loop Driving Policy"]
Key Designs¶
1. ReconSimulator: Appearance and Physical Modeling
Standard 3DGS driving simulators degrade substantially when an ego vehicle deviates from recorded camera trajectories, introducing severe blur on background lane markings that misleads end-to-end planners. ReconSimulator decomposes dynamic scenes into a static background representation \(\mathcal{G}_{\text{Background},w}\) in world coordinates and rigid Gaussian groups \(\mathcal{G}_{\text{Rigid},l}^v\) for each moving obstacle \(v\) in local coordinates. During rendering at time step \(t\), object Gaussians are placed into the world frame via rotation matrix \(M_t^v\) and translation vector \(S_t^v\): $\(\mathcal{G}_{\text{Rigid},w}^v(t) = M_t^v \cdot \mathcal{G}_{\text{Rigid},l}^v + S_t^v\)$
When the ego vehicle explores novel paths and yields degraded views \(\hat{V}_{\text{novel}}\), ReconSimulator applies a frozen video diffusion restoration model DriveRestorer \(\mathcal{R}\) conditioned on 3D bounding boxes and HDMap vectors \(s\), producing restored video \(V_{\text{novel}} = \mathcal{R}(\hat{V}_{\text{novel}}, s)\) to supervise and refine 3DGS parameters. For physical plausibility, a kinematic bicycle model updates every vehicle's pose \(W_t = [R_t \mid P_t] \in \mathrm{SE}(3)\) according to linear velocity \(v_t\), steering angle \(\delta_t\), and wheelbase length \(L\), computing the incremental yaw change: $\(\Delta \theta_t = \frac{v_t}{L}\tan(\delta_t)\Delta t\)$ Strict kinematic boundary checks on steering angles, angular rates, and acceleration bounds prevent physically impossible motions and ensure valid trajectories across all dynamic agents.
2. Dynamic Adversary Agent (DAA)
Corner cases like aggressive vehicle cut-ins rarely occur naturally in driving datasets, preventing reinforcement learning policies from experiencing near-collision states. DAA generates realistic and diverse corner case interactions during both training stages. In the Bird's-Eye View (BEV) space, DAA first identifies candidate target vehicles based on their Euclidean distance to the ego car and an interactive behavior specification \(\mathcal{B}\). It then crafts an adversarial trajectory \(T_{\text{target}}'\) by perturbing the target vehicle's original path \(T_{\text{target}}\) relative to the ego trajectory \(T_{\text{ego}}\): $\(T_{\text{target}}' = f(T_{\text{ego}}, T_{\text{target}}, \mathcal{B})\)$
The newly generated trajectory undergoes a rigorous three-fold validation: ① it must lie strictly within the drivable road surface; ② it must maintain a minimum safety clearance \(d_{\text{min}}\) against all other surrounding vehicles \(o_j\) (\(\|T_{\text{target}}' - o_j\| \ge d_{\text{min}}\)), while deliberately permitting collision paths with the ego vehicle to construct adversarial pressure; ③ it must satisfy the vehicle's kinematic acceleration and steering limits. During imitation learning, DAA produces cut-in scenarios paired with collision-avoiding ego paths for offline sensor rendering; during online RL, DAA dynamically perturbs surrounding vehicle trajectories (e.g., speed variations) to test policy robustness against active hazards.
3. Cousin Trajectory Generator (CTG)
Behavior cloning pretraining is vulnerable to dataset distribution bias; in nuScenes, non-straight driving maneuvers account for less than 12% of total frames, causing policies to overfit to passive straight cruising. CTG alleviates this limitation by systematically expanding the diversity of ego action patterns, yielding the Cousin-nuScenes dataset.
CTG generates action extension trajectories featuring large-angle turns and lane changes through kinematically bounded trajectory synthesis. Furthermore, to extract richer motion signals from scarce expert maneuvers such as U-turns, CTG performs continuous linear interpolation between consecutive discrete timestamps \(t_i\) and \(t_{i+1}\) on expert paths \(X_{\text{ego}} = \{X_{\text{ego}}^{t_1}, X_{\text{ego}}^{t_2}, \dots, X_{\text{ego}}^{t_n}\}\): $\(X_{\text{ego}}^t = X_{\text{ego}}^{t_i} + \frac{t - t_i}{t_{i+1} - t_i} \left(X_{\text{ego}}^{t_{i+1}} - X_{\text{ego}}^{t_i}\right)\)$ By adding \(m\) intermediate waypoints and synchronously transforming surrounding vehicle poses, CTG preserves coherent spatiotemporal relations while enriching driving granularity. With CTG, non-straight-line maneuvers expand more than four-fold from 3,877 to 18,173 instances, ensuring balanced exposure across all maneuver types.
Loss & Training¶
The framework adopts a two-stage training scheme: 1. Perception Warm-up & Behavior Cloning: The BEV encoder, map head, and agent head are first trained using ground-truth HDMap annotations and dynamic object bounding boxes. Subsequently, these perception heads are frozen, and supervised behavior cloning optimizes the image encoder and planning head across nuScenes expert demonstrations, DAA corner cases, and CTG-augmented Cousin-nuScenes rollouts. 2. Closed-Loop PPO with Alternating Imitation: In closed-loop ReconSimulator instances, rollouts of \((s_t, a_t, r_{t+1}, s_{t+1})\) are collected to update the policy using Proximal Policy Optimization (PPO). To prevent policy degeneration and erratic driving jitter, PPO updates are alternated with supervised imitation updates using the cached augmented datasets.
Key Experimental Results¶
Main Results¶
Policy evaluations in the closed-loop 3DGS environment report Collision Ratio (CR), Dynamic Collision Ratio (DCR), Static Collision Ratio (SCR), Deviation Ratio (DR), Positional Deviation Ratio (PDR), and Heading Deviation Ratio (HDR). As shown below, ReconDreamer-RL substantially outperforms both imitation learning baselines and prior 3DGS-based RL methods:
| Method | CR ↓ | DCR ↓ | SCR ↓ | DR ↓ | PDR ↓ | HDR ↓ |
|---|---|---|---|---|---|---|
| VAD | 0.386 | 0.234 | 0.152 | 0.163 | 0.103 | 0.060 |
| GenAD | 0.333 | 0.190 | 0.143 | 0.146 | 0.093 | 0.053 |
| VADv2 | 0.290 | 0.162 | 0.128 | 0.154 | 0.107 | 0.047 |
| RAD | 0.238 | 0.143 | 0.095 | 0.084 | 0.057 | 0.027 |
| ReconDreamer-RL (Ours) | 0.077 | 0.048 | 0.029 | 0.040 | 0.027 | 0.013 |
In specialized, safety-critical vehicle cut-in corner case scenarios, the performance breakdown is as follows:
| Method | CR ↓ | DCR ↓ | SCR ↓ |
|---|---|---|---|
| VAD | 0.449 | 0.293 | 0.156 |
| GenAD | 0.379 | 0.234 | 0.145 |
| VADv2 | 0.436 | 0.276 | 0.160 |
| RAD | 0.317 | 0.210 | 0.107 |
| ReconDreamer-RL (Ours) | 0.089 | 0.053 | 0.036 |
Ablation Study¶
A comprehensive component ablation evaluates the independent and joint contributions of ReconSimulator, DAA, and CTG against the RAD-3DGS baseline:
| Config | ReconSimulator | DAA | CTG | CR ↓ | DCR ↓ | SCR ↓ | DR ↓ | PDR ↓ | HDR ↓ | Note |
|---|---|---|---|---|---|---|---|---|---|---|
| Baseline (RAD-3DGS) | - | - | - | 0.238 | 0.143 | 0.095 | 0.084 | 0.057 | 0.027 | Vanilla 3DGS RL |
| DAA Only | - | ✓ | - | 0.167 | 0.102 | 0.065 | 0.076 | 0.052 | 0.024 | Corner cases on legacy simulator |
| CTG Only | - | - | ✓ | 0.191 | 0.121 | 0.070 | 0.068 | 0.046 | 0.022 | Action space diversity boost |
| DAA + CTG | - | ✓ | ✓ | 0.142 | 0.082 | 0.060 | 0.063 | 0.043 | 0.020 | Dual data engines combined |
| ReconSimulator Only | ✓ | - | - | 0.172 | 0.103 | 0.069 | 0.073 | 0.053 | 0.020 | Diffusion prior view refinement |
| Simulator + DAA | ✓ | ✓ | - | 0.117 | 0.069 | 0.048 | 0.067 | 0.050 | 0.017 | Adversarial training under photorealism |
| Simulator + CTG | ✓ | - | ✓ | 0.143 | 0.086 | 0.057 | 0.053 | 0.040 | 0.013 | Improved trajectory tracking |
| Full Model (Ours) | ✓ | ✓ | ✓ | 0.077 | 0.048 | 0.029 | 0.040 | 0.027 | 0.013 | 67.6% drop in overall collision ratio |
For novel-view rendering quality under a 6m lane shift, adding the video diffusion prior improves foreground NTA-IoU from 0.148 to 0.325 (+119.6%) and reduces FID from 256.42 to 125.43. Meanwhile, ReconSimulator delivers 125 FPS rendering throughput, matching the high sample rate required by large-scale PPO rollouts (surpassing EmerNeRF's 0.21 FPS by several orders of magnitude).
Key Findings¶
- Adversarial corner cases are the primary driver for dynamic collision avoidance: Adding DAA alone in ReconSimulator drops CR from 0.172 to 0.117 (with DCR falling from 0.103 to 0.069), proving that active generation of cut-in dynamics is vital to conquering long-tail collision modes.
- Cousin trajectories prevent straight-line overfitting and improve trajectory tracking: CTG lowers positional deviation (PDR) from 0.053 to 0.040 and heading deviation (HDR) from 0.020 to 0.013, solidifying ego tracking stability during curved paths.
- Diffusion priors resolve perception-level visual distortion during exploration: Standard 3DGS renders blurred lane dividers during lane changes; restoring foreground and road surface details allows the policy to safely maintain lane boundaries, reducing static collisions (SCR) by nearly 70%.
Highlights & Insights¶
- Decoupled Diffusion Enhancement for High-Throughput RL: ReconDreamer-RL incorporates the heavy video diffusion model as an offline restoration supervisor during scene reconstruction, enabling 3DGS to retain its ultra-fast 125 FPS rendering speed while inheriting diffusion-grade visual fidelity across novel viewpoints.
- Strong Zero-Shot Cross-Dataset Transferability: DriveRestorer trained on the Waymo dataset transfers zero-shot to nuScenes scene reconstruction with noticeable visual improvements; fine-tuning on the target dataset requires only 3 hours (adding merely ~0.3 hours per scene), establishing high practical viability.
- Alternating RL-IL Updates Balance Collision Avoidance and Passenger Comfort: Interleaving imitation learning updates within closed-loop PPO stabilizes policy optimization, yielding a low lateral jerk of 0.189 and avoiding erratic, unhumanlike steering maneuvers during sudden evasions.
Limitations & Future Work¶
- Admitted Limitations: DAA's adversarial scenario generation still depends on geometric proximity heuristics and predefined behavior templates (such as cut-ins and abrupt braking), falling short of fully emergent, multi-agent game-theoretic traffic interactions.
- Weather and Lighting Generalization: Evaluations remain centered on daytime, clear-weather sequences in nuScenes. Performance under low-light night driving, dense precipitation, or non-rigid spray artifacts remains an open research question.
- Future Directions: Integrating multimodal foundation models (VLMs) into DAA could enable prompt-driven generation of diverse social interactions, such as jaywalking pedestrians or road construction reroutes.
Related Work & Insights¶
- vs RAD: While RAD initiated RL for end-to-end driving in 3DGS environments, it suffers from severe off-trajectory rendering artifacts and lacks adversarial corner cases (resulting in a high CR of 0.238). ReconDreamer-RL resolves both defects, cutting CR down to 0.077.
- vs DriveDreamer4D / ReconDreamer: Prior diffusion-based driving works focused primarily on open-loop 4D video generation and reconstruction benchmarking. ReconDreamer-RL is the first to combine diffusion-guided reconstruction with physical kinematics to form an active, closed-loop RL simulator.
- vs VAD / VADv2: Supervised imitation learning models exhibit excessive failure rates under unexpected cut-in events (DCR exceeding 0.27). ReconDreamer-RL equips the policy with predictive and proactive evasive capabilities through closed-loop exploration.
Rating¶
- Novelty: ⭐⭐⭐⭐ [Innovative fusion of diffusion-guided 3DGS reconstruction, kinematic modeling, and adversarial data generation for end-to-end RL]
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Exhaustive evaluations covering standard metrics, cut-in corner cases, rendering speed, driving comfort, and fine-grained ablations]
- Writing Quality: ⭐⭐⭐⭐⭐ [Clear problem formulation, transparent two-stage design, and rigorous logical progression]
- Value: ⭐⭐⭐⭐⭐ [Offers a robust, photorealistic, and computationally feasible paradigm to bridge the Sim-to-Real gap in autonomous driving]