title: >- [Paper Note] ReCamDriving: LiDAR-Free Camera-Controlled Video Synthesis for Novel Trajectories description: >- [ECCV 2026][video_generation][3DGS] ReCamDriving introduces a pure vision-based two-stage diffusion framework with 3DGS geometric guidance and cross-trajectory data curation (ParaDrive), achieving state-of-the-art 3D-consistent novel-trajectory driving video synthesis without LiDAR. tags: - ECCV 2026 - video_generation - autonomous_driving - 3D Gaussian Splatting - camera control date: 2026-09-19 content_hash: 53c41d16b1421fc2

ReCamDriving: LiDAR-Free Camera-Controlled Video Synthesis for Novel Trajectories¶

Conference: ECCV 2026
Paper: ECCV Official
Project Page: https://recamdriving.github.io/
Area: Video Generation
Keywords: Video Generation, Autonomous Driving, Novel Trajectory Synthesis, 3D Gaussian Splatting, Camera Control

TL;DR¶

Addressing out-of-distribution artifact overfitting in reconstruction-repair methods and geometric sparsity in LiDAR-based camera control, ReCamDriving presents a pure vision-based two-stage progressive diffusion framework guided by dense 3DGS renderings, coupled with a cross-trajectory data curation strategy (ParaDrive), delivering state-of-the-art camera controllability and 3D-consistent novel-trajectory video generation.

Background & Motivation¶

Synthesizing photorealistic multi-pass driving videos is crucial for autonomous driving, as it provides dense viewpoints required for 3D scene reconstruction and world-model simulation. However, capturing real-world multi-pass driving data requires multiple synchronized sensor-equipped vehicles operating simultaneously, incurring prohibitive logistical and operational costs. Synthesizing geometrically consistent, high-fidelity novel-trajectory videos directly from readily available single-pass monocular or multi-camera logs has therefore emerged as an essential, scalable alternative.

Existing paradigms for novel-trajectory synthesis fall into two main categories, each plagued by fundamental limitations. The first paradigm is the reconstruction-then-repair pipeline (e.g., Difix3D+, GSFixer), which reconstructs a driving scene using 3DGS or NeRF, renders novel trajectories, and trains diffusion models to remove rendering degradation. Because these models learn local degraded-to-clean mappings, they easily overfit to training-time artifact distributions and fail when facing severe out-of-distribution (OOD) artifacts induced by large lateral viewpoint extrapolations. The second paradigm is camera-controlled video generation (e.g., FreeVS, StreetCrafter), which conditions video diffusion models on source videos and target-view colored LiDAR point projections. However, LiDAR points are inherently sparse and completely missing in distant backgrounds, sky, and reflective surfaces, leading to severe 3D geometric collapse. Moreover, due to the lack of ground-truth parallel trajectories, existing models rely on longitudinal splits along the same recording for pseudo-pair supervision, creating a severe train-test gap when performing lateral novel-trajectory synthesis during inference.

This paper tackles these challenges with a novel perspective: rather than treating 3DGS renderings as degraded images to be repaired, it exploits their structurally complete 3D scene representation as dense geometric guidance for generative diffusion, while resolving the data bottleneck via an inverted cross-trajectory pairing strategy. Core idea: eliminate expensive, sparse LiDAR conditioning by leveraging dense 3DGS renderings as a structural scaffold within a two-stage progressive diffusion framework, paired with a 3DGS-based cross-trajectory data curation strategy that constructs the 110K-pair ParaDrive dataset to align lateral training and inference transformations.

Method¶

Overall Architecture¶

Given a source trajectory video $V_s$ and a relative camera transformation $\Delta T$, ReCamDriving synthesizes a novel-trajectory video $V_t$ with lateral offsets. Directly conditioning video diffusion models on target-view 3DGS renderings induces shortcut learning, causing the network to collapse into a trivial local artifact repair tool instead of learning geometric warping. To decouple viewpoint translation from structural guidance, ReCamDriving introduces a two-stage progressive training scheme: Stage 1 learns coarse viewpoint warping conditioned strictly on relative camera poses, while Stage 2 freezes the coarse transformation parameters and introduces dedicated rendering-attention and cross-attention modules to integrate dense 3DGS renderings for fine-grained structural guidance.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Source Video + Relative Pose ΔT"] --> B["Stage 1: Relative Pose-Guided Coarse Control<br/>Disentangled pose encoding & coarse spatial warping"]
    B --> C["Stage 2: Fine-Grained 3DGS Geometric Guidance<br/>Frozen DiT trunk with rendering & cross-attention"]
    C --> D["Cross-Trajectory Data Curation (ParaDrive)<br/>Synthesize multi-iteration 3DGS source & clean target pairs"]
    D --> E["Output: High-Fidelity 3D-Consistent Novel-Trajectory Video"]

Key Designs¶

1. Stage 1: Relative Pose-Guided Coarse Camera Control To avoid latent representation ambiguity when concatenating source and target noise latents, the framework employs a decoupled pose embedding strategy. A camera encoder $E_{\text{cam}}$ independently encodes the relative camera pose $\Delta T = T_{t \leftarrow s} \in \mathrm{SE}(3)$ and the identity pose $T_I = I_4$ into relative pose embedding $c_r \in \mathbb{R}^{f \times d}$ and identity embedding $c_I \in \mathbb{R}^{f \times d}$. A learnable frame embedding $E_f \in \mathbb{R}^{f \times d}$ is further incorporated to establish temporal correspondence across frames: $$x_i = \text{Cat}(x_t + c_r + E_f, \; x_s + c_I + E_f)$$ The concatenated latent $x_i$ is fed into $N$ Diffusion Transformer (DiT) blocks. In this stage, only the self-attention and normalization layers are unfrozen, forcing the network to model spatial-temporal velocity fields that perform coarse viewpoint transformation from the source perspective before seeing any target rendering.

2. Stage 2: Fine-Grained Camera Control via 3DGS Renderings Directly introducing target-view 3DGS renderings $V_{\text{gs}}$ in an end-to-end setting leads to shortcut learning because the conditioning rendering already shares the target camera pose. In Stage 2, the self-attention parameters trained in Stage 1 are completely frozen. Two auxiliary modules—Rendering Attention and Cross Attention—are added to each DiT block. The target 3DGS rendering $V_{\text{gs}}$ is encoded by the 3D VAE into $x_{\text{gs}}$ and augmented with pose and frame embeddings as $\bar{x}_{\text{gs}} = x_{\text{gs}} + c_r + E_f$. Rendering Attention refines spatial-temporal geometry within the 3DGS rendering latent space, and Cross Attention conditions the coarsely warped diffusion latent $\bar{x}_i = \text{SelfAttn}(x_i)$ on $\bar{x}_{\text{gs}}$. Because $\bar{x}_i$ is already aligned to the target pose, the network treats 3DGS rendering tokens purely as a geometric scaffold rather than a canvas for local repair.

3. Cross-Trajectory Data Curation and the ParaDrive Dataset Single-pass autonomous driving datasets lack synchronized parallel-trajectory videos, and traditional longitudinal pseudo-pairs fail to supervise lateral lane-change transformations. The authors propose an inverted 3DGS cross-trajectory curation strategy: DriveStudio is utilized to reconstruct 3DGS scenes and render laterally shifted novel-trajectory videos ($\pm 1\text{m}, \pm 2\text{m}, \pm 3\text{m}, \pm 4\text{m}$). Because rendered novel views contain artifacts and cannot serve as ground truth, they are repurposed as the source input during training, while the pristine recorded-trajectory video serves as ground-truth supervision. To make the model robust to varying rendering quality, intermediate 3DGS checkpoints at 100, 500, and 1,000 iterations provide multi-level degraded structural conditions. In total, 1.6K scenes from WOD and NuScenes yield 110K dual-trajectory video pairs (requiring 8,240 L20 GPU hours), forming the ParaDrive dataset.

Loss & Training¶

The framework is optimized within the Rectified Flow Matching formulation. Defining the forward trajectory as a straight path between Gaussian noise $x_0 \sim \mathcal{N}(0, I)$ and data latent $x_1$, $x_t = t x_1 + (1-t) x_0$, the target velocity is $v_t = x_1 - x_0$. The model parameterizes $\epsilon_\theta$ to minimize: $$\mathcal{L}_{\text{FM}} = \mathbb{E}_{x_0, x_1, c_{\text{cam}}, t} \left\| \epsilon_\theta(x_t; c_{\text{cam}}, t) - (x_1 - x_0) \right\|_2^2$$ Both stages are trained on 64 NVIDIA A100 GPUs for 6,000 steps each, using batch size 1, learning rate 1e-4, frame resolution $480 \times 832$, and 121 frames per clip. The backbone is initialized from Wan2.1, with camera encoder and self-attention initialized from ReCamMaster.

Key Experimental Results¶

Main Results¶

Quantitative evaluations on the Waymo Open Dataset (WOD) benchmark across various lateral offsets demonstrate that ReCamDriving consistently outperforms 3DGS rendering (DriveStudio), repair models (Difix3D+), and LiDAR-conditioned generators (FreeVS, StreetCrafter), as reported in Table 1 and Table 2.

Table 1: Quantitative comparison on WOD across lateral offsets

Metric / Offset	Method	Imaging Quality IQ↑	Temporal Consistency TCE↓	Image Fidelity FID↓	Video Fidelity FVD↓	Semantic Similarity CLIP-V↑
Offset ±1m	DriveStudio	52.13	7.93	83.32	25.37	94.78
	Difix3D+	64.24	7.81	56.35	27.80	95.32
	FreeVS	62.74	11.27	63.06	37.06	88.99
	StreetCrafter	63.57	9.72	28.18	20.51	96.01
	Ours	65.18	3.38	13.76	13.27	97.96
Offset ±4m	DriveStudio	41.47	9.49	144.05	72.50	88.76
	Difix3D+	58.81	10.49	78.08	65.37	90.12
	FreeVS	56.15	14.35	107.04	58.39	85.17
	StreetCrafter	59.89	12.38	68.73	36.67	91.17
	Ours	61.32	4.30	32.36	26.76	94.91

Table 2: Camera trajectory estimation accuracy on WOD via MegaSaM

Method	Offset ±1m RErr.↓ / TErr.↓	Offset ±2m RErr.↓ / TErr.↓	Offset ±3m RErr.↓ / TErr.↓	Offset ±4m RErr.↓ / TErr.↓
Difix3D+	1.36 / 2.42	1.64 / 2.66	2.01 / 2.97	2.68 / 3.12
FreeVS	1.71 / 2.88	2.12 / 2.93	3.17 / 3.78	3.02 / 3.39
StreetCrafter	1.52 / 2.53	1.79 / 2.77	1.91 / 3.13	2.87 / 3.03
Ours	1.32 / 2.37	1.45 / 2.43	1.63 / 2.65	1.57 / 2.73

Ablation Study¶

Ablations on camera conditioning signals (Table 4) and training strategies (Table 6) on WOD validate the necessity of each architectural component.

Table 4: Ablation on camera conditioning modalities (WOD)

Camera Condition	IQ↑	FID↓	FVD↓	Rotation Error RErr.↓	Translation Error TErr.↓
Pose only	60.13	34.86	32.31	3.01	4.23
Pose + LiDAR	61.32	31.23	27.78	1.53	2.69
Pose + LiDAR + 3DGS	63.42	24.75	19.27	1.41	2.47
Pose + 3DGS (Ours)	63.63	24.88	19.18	1.49	2.55

Table 6: Ablation on training paradigm (WOD & NuScenes)

Training Strategy	Imaging Quality IQ↑	Image Fidelity FID↓	Video Fidelity FVD↓	Cross-View CLIP-V↑
One-stage joint training	59.97	32.64	25.16	94.78
Two-stage progressive training (Ours)	63.42	25.13	18.32	96.32

Key Findings¶

3DGS renders LiDAR superfluous: Table 4 reveals that adding LiDAR to the Pose + 3DGS configuration yields negligible improvement in FID (24.75 vs 24.88) and FVD (19.27 vs 19.18), while dramatically increasing data acquisition overhead. Pure vision-based dense geometric guidance provides sufficient scene constraints.
Two-stage progressive training prevents trivial shortcut repair: When all modules are trained jointly in a single stage, FID deteriorates to 32.64 and FVD to 25.16 (Table 6). Qualitative inspections confirm that single-stage optimization degrades into local image inpainting, whereas freezing Stage 1 forces the network to retain global view warping.
Lateral curation aligns real-world motion: Training on longitudinal pseudo-pairs results in a rotation error of 1.97 and FID of 34.17 (Table 7), whereas the proposed lateral curation strategy achieves a rotation error of 1.49 and FID of 24.88.

Highlights & Insights¶

Dense visual geometry over sparse active sensors: Replacing sparse LiDAR with dense 3DGS structural renderings resolves severe geometric distortion in sky and distant backgrounds, eliminating reliance on expensive LiDAR sensors.
Inverted pairing for lateral supervision: Turning degraded novel-trajectory 3DGS renderings into inputs and treating clean single-pass real videos as ground truth elegantly creates 110K parallel-trajectory training pairs without requiring physical multi-vehicle setups.
Progressive anti-collapse decoupling: Freezing the relative-pose self-attention modules before introducing 3DGS rendering tokens via cross-attention effectively prevents generative diffusion models from regressing into trivial image-repair tools.

Limitations & Future Work¶

Offline 3DGS reconstruction overhead: Reconstructing scene-level 3DGS representations with DriveStudio currently requires significant offline compute (8,240 L20 GPU hours for ParaDrive). Integrating emerging feed-forward 4D reconstruction frameworks could drastically reduce pre-processing latency.
Dynamic object reconstruction artifacts: Fast-moving surrounding traffic or pedestrians may exhibit blur and floaters in per-scene 3DGS reconstructions, occasionally introducing subtle boundary softness into the synthesized novel-trajectory videos.
Extreme viewpoint extrapolation: Under severe lateral shifts exceeding $\pm 5\text{m}$, unobserved regions expand considerably, demanding stronger visual hallucination capabilities from the video diffusion backbone.

vs Difix3D+ / GSFixer (Reconstruction-then-repair): Repair baselines perform local 2D/3D filtering directly on degraded novel-view renderings. When extrapolations exceed training distributions, repair models generate distorted lane lines and blurry roadside objects. In contrast, ReCamDriving uses 3DGS as structural conditioning for a generative diffusion model, preserving sharp details and 3D continuity.
vs StreetCrafter / FreeVS (LiDAR-guided camera control): LiDAR-conditioned models suffer from sparsity in distant backgrounds and lack lateral motion supervision. ReCamDriving relies on dense 3DGS geometry and lateral ParaDrive supervision, maintaining robust synthesis even under $\pm 4\text{m}$ lateral shifts (FVD 26.76 vs StreetCrafter's 36.67 and FreeVS's 58.39).

Rating¶

Novelty: ⭐⭐⭐⭐⭐ Elegant inverted cross-trajectory data formulation combined with a two-stage progressive diffusion scheme leveraging dense 3DGS guidance.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive benchmarking across WOD and NuScenes, supported by extensive metric evaluations and ablations on 110K video pairs.
Writing Quality: ⭐⭐⭐⭐⭐ Clear problem formulation, intuitive diagrams, and rigorous analysis.
Value: ⭐⭐⭐⭐⭐ Highly valuable for autonomous driving world models, closed-loop simulation, and vision-centric novel-trajectory synthesis.