FreeGen: Feed-Forward Reconstruction–Generation Co-Training for Free-Viewpoint Driving Scene Synthesis¶

Conference: ECCV 2026
Paper: ECCV Official
Full Cache: ../paper_cache/ECCV2026/eccv-3958.txt
Area: Autonomous Driving
Keywords: Free-Viewpoint Synthesis, Driving Scene Synthesis, Feed-Forward 3DGS, Video Diffusion Model, Closed-Loop Co-Training

TL;DR¶

FreeGen establishes a feed-forward reconstruction-generation co-training framework unifying 3D Gaussian Splatting and video diffusion, delivering consistent and realistic free-viewpoint driving scene synthesis from single-trajectory image inputs without per-scene optimization.

Background & Motivation¶

Closed-loop simulation and large-scale policy evaluation for autonomous driving require synthesizing highly realistic scenes from arbitrary, free-viewpoint camera trajectories. However, both real-world sensor logs (e.g., nuScenes, Waymo) and synthetic datasets are captured along fixed, single ego-vehicle trajectories. When simulating diverse operational maneuvers such as lane changes, turns, or overtaking, the simulator must render novel viewpoints that deviate substantially from the recorded path. Synthesizing high-fidelity off-trajectory observations under the constraint of strictly single-trajectory observations poses a major challenge.

Existing paradigms exhibit complementary strengths and fundamental shortcomings. Reconstruction methods grounded in Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) provide strong interpolation consistency within the observed camera frustum. However, they typically rely on expensive per-scene optimization and produce severe hole artifacts and blurring when extrapolating to unobserved viewpoints. Feed-forward 3D reconstruction networks accelerate inference into a single forward pass but still fail to recover plausible textures in extrapolated views. Conversely, video diffusion models pre-trained on web-scale datasets excel at hallucinating visually realistic details, but inherently lack explicit 3D geometric constraints, frequently causing structural distortions such as missing poles, mutated pedestrians, and temporal drifting. Hybrid frameworks using image warping or sparse LiDAR projections suffer from severe structural misalignment or extreme point sparsity in distant buildings and sky.

The core insight of this paper is to leverage the intrinsic complementarity between feed-forward 3D reconstruction and generative diffusion refinement. Reconstruction offers a robust 3D geometric scaffold to preserve interpolation consistency, while generative models inject realistic appearance priors to eliminate extrapolation blur. Core idea: unify feed-forward 3D Gaussian reconstruction and video diffusion refinement into an end-to-end framework, and introduce a bidirectional closed-loop co-training mechanism that distills generative priors into 3D Gaussians while reinforcing the generator with consistent structural geometry, fully removing test-time optimization.

Method¶

Overall Architecture¶

FreeGen operates as a feed-forward pipeline coupling two learnable branches through co-training. The system takes multi-view surround-camera images and camera poses as input. A monocular geometric foundation model first predicts metric depth maps to assist scene representation. A feed-forward 3DGS encoder-decoder network then extracts pixel-aligned features, fuses them into a triplane representation via deformable attention, and decodes them into 3D Gaussians to render target-view color images, depth maps, and opacity maps in real time. Next, this geometry condition triplet is projected by a condition encoder and concatenated with noisy latents inside a video diffusion model for fine-grained appearance refinement. In the co-training phase, both branches alternate between generating off-trajectory pseudo-labels and acting as structural discriminators.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}, 'subGraphTitleMargin': {'top': 8, 'bottom': 16}}}%%
flowchart TD
    In["Multi-view Surround Images + Poses"] --> GRec["Feed-Forward Gaussian Geometry Conditioning<br/>Pixel features + Triplane decoding + Rasterization"]
    GRec --> Triplet["Geometry Condition Triplet<br/>Color I_geo + Depth D_geo + Opacity A_geo"]
    Triplet --> GDiff["Geometry-Aware Video Diffusion Refinement<br/>Condition latent encoding + Channel concatenation"]
    GDiff --> Out["Refined Free-Viewpoint Video"]
    GDiff -.->|Off-trajectory pose sampling & pseudo-labeling| Loop["Closed-Loop Co-Training Mechanism<br/>Step 1: Gen-guided Rec ↔ Step 2: Rec-guided Gen"]
    Loop -.->|Distill appearance priors & supply rigid geometry| GRec

Key Designs¶

1. Feed-Forward Gaussian Geometry Conditioning: Holistic Geometry with Reliability Awareness To bypass the computational bottleneck of per-scene optimization and avoid sparse or misaligned guidance (such as LiDAR projections or depth-warped images), FreeGen employs a feed-forward 3DGS architecture based on Omni-Scene. Given \(N\)-view inputs, metric depth is initially estimated using Metric3D v2. A multi-view U-Net extracts pixel-aligned high-resolution feature maps, which are aggregated into a compact triplane representation via a Triplane Transformer with deformable attention. Gaussian decoders directly predict \(K\) 3D Gaussians with parameters covering position, opacity, scale, rotation quaternion, and spherical harmonic colors. For any requested camera trajectory, tile-based rasterization concurrently renders the color image \(I_{\mathrm{geo}}\), depth map \(D_{\mathrm{geo}}\), and opacity map \(A_{\mathrm{geo}}\):

\[C_{\mathrm{geo}} = \bigl(I_{\mathrm{geo}},\, D_{\mathrm{geo}},\, A_{\mathrm{geo}}\bigr)\]

The color image establishes structural and coarse visual layout, the depth map enforces 3D scale and boundary consistency, and the opacity map explicitly reflects the rendering reliability of each ray. Regions with low opacity highlight unobserved view cones or occluded voids, prompting the generative diffusion branch to target refinement toward uncertain regions while preserving high-confidence geometry.

2. Geometry-Aware Video Diffusion Refinement: Minimal Modification for Preserved Generalization To inject generative priors without corrupting the temporal coherence and physical realism pre-trained into large video foundation models, FreeGen adopts Stable Video Diffusion (SVD) and keeps architectural modifications minimal. The rendered geometry condition triplet \(C_{\mathrm{geo}}\) is mapped into a condition latent representation \(\mathbf{z}_c\) through a lightweight learnable condition encoder \(\mathcal{E}_c\). Simultaneously, a frozen image VAE encoder \(\mathcal{E}_v\) transforms the reference video frame into latent features, to which Gaussian noise is added at diffusion timestep \(\tau\) to yield perturbed latents \(\mathbf{z}_\tau\). The condition latent \(\mathbf{z}_c\) is directly concatenated with the noisy latent \(\mathbf{z}_\tau\) along the channel dimension and fed into the denoising network \(f_\theta\):

\[\hat{\boldsymbol{\epsilon}} = f_\theta(\mathbf{z}_\tau, \mathbf{z}_c)\]

The training objective minimizes the standard noise prediction mean squared error \(\mathcal{L}_{\mathrm{gen}} = \mathbb{E}\big[\|\boldsymbol{\epsilon} - f_\theta(\mathbf{z}_\tau, \mathbf{z}_c)\|_2^2\big]\). Channel-wise concatenation avoids re-architecting cross-attention layers, ensuring that the model leverages pre-trained spatiotemporal dynamics while tightly adhering to the underlying 3D Gaussian geometry.

3. Closed-Loop Co-Training Mechanism: Breaking Single-Trajectory Bias without Test-Time Optimization When trained solely on recorded driving data, feed-forward 3DGS models inevitably suffer from out-of-distribution geometric degradation at lateral off-trajectory viewpoints, and downstream diffusion models trained only on on-trajectory ground-truth cannot rectify these artifacts. FreeGen solves this via an alternating two-step co-training strategy that freezes one branch at a time: - Generation-guided reconstruction (Step 1): Camera viewpoints are randomly sampled with lateral shifts within \([-3\text{m}, 3\text{m}]\). The frozen diffusion model refines the Gaussian renderings to produce high-fidelity pseudo-labels. The Gaussian reconstruction network is then updated using both the original captured frames and these pseudo-labels. Setting the auxiliary loss weight to \(0.1\) effectively distills generative appearance priors into the 3D Gaussians while preventing pseudo-label noise from destabilizing the 3D structure. - Reconstruction-guided generation (Step 2): Using the updated Gaussian model, the off-trajectory refined predictions are re-rendered back onto the original trajectory viewpoints. The diffusion refinement module denoises these re-rendered views conditioned on geometry, supervised strictly by original ground-truth frames. Because supervision is anchored to uncorrupted ground-truth data, the perturbed renderings act as robust 3D data augmentation, training the diffusion network to systematically repair reconstruction artifacts. Alternating Step 1 and Step 2 establishes a mutual reinforcement loop that eliminates the need for expensive test-time optimization.

Loss & Training¶

The reconstruction model undergoes 50K pre-training steps on single-trajectory sequences followed by 20K co-training steps. The total reconstruction objective combines mean squared error, LPIPS perceptual loss, and \(L_1\) depth loss supervised by Metric3D v2 pseudo-labels:

\[\mathcal{L}_{\mathrm{recon}} = \mathcal{L}_{\mathrm{mse}} + \lambda_1 \mathcal{L}_{\mathrm{lpips}} + \lambda_2 \mathcal{L}_{\mathrm{depth}}\]

with weights \(\lambda_1 = 0.05\) and \(\lambda_2 = 0.01\). The diffusion refinement branch is pre-trained for 20K steps on cross-timestamp pairs and co-trained for 20K steps with a constant learning rate of \(1\times 10^{-5}\). The entire framework is trained on two NVIDIA A100 GPUs and runs pure feed-forward inference in seconds per 6-frame multi-camera sequence.

Key Experimental Results¶

Main Results¶

Experiments are evaluated on nuScenes across 700 training and 150 validation scenes, interpolating 2 Hz keyframes to 12 Hz. Video length is fixed to \(T=6\) at \(768 \times 432\) resolution.

In the off-trajectory extrapolation setting, target frames are sampled every 2 frames along the path and shifted laterally by \(\tau \in \{\pm 1\text{m}, \pm 2\text{m}, \pm 4\text{m}\}\). Realism and temporal coherence are measured using FID and FVD against original captured videos:

Method	Base Model	Shift ±1m FID↓	Shift ±1m FVD↓	Shift ±2m FID↓	Shift ±2m FVD↓	Shift ±4m FID↓	Shift ±4m FVD↓
PVG (IJCV 2026)	-	48.15	246.74	60.44	356.23	84.50	501.16
EmerNeRF (ICLR 2024)	-	37.57	171.47	52.03	294.55	76.11	497.85
StreetGaussian (ECCV 2024)	-	32.12	153.45	43.24	256.91	67.44	429.98
OmniRe (ICLR 2025)	-	31.48	152.01	43.31	254.52	67.36	428.20
Image Warping (ICCV 2025)	-	32.27	216.69	60.64	344.68	93.27	398.36
Omni-Scene (CVPR 2025)	-	12.92	117.49	20.96	205.99	39.47	370.79
DIFIX3D+ (CVPR 2025)	SD-Turbo	17.36	143.04	21.73	154.51	30.43	263.69
FreeVS (ICLR 2025)	SVD	51.26	431.99	62.04	497.37	77.14	556.14
DiST-4D (ICCV 2025)	SVD	10.12	45.14	12.97	68.80	17.57	105.29
FreeGen (Ours)	SVD	9.47	34.11	11.25	43.28	14.30	66.19
PhiGensis (2025)	OpenSora V2.0	9.80	43.80	11.71	67.54	15.51	103.13
WorldSplat (2025)*	OpenSora V1.2	8.25	40.17	11.26	47.41	13.38	64.07

(Note: WorldSplat relies on HDMaps and 3D bounding box annotations; FreeGen achieves superior FVD without any 3D annotations).

In the on-trajectory interpolation setting, viewpoints are evaluated across temporal intervals of 1, 2, and 4 frames (Easy, Medium, Hard):

Method	@1 (Dense) PSNR↑ / SSIM↑ / LPIPS↓ / FID↓	@2 (Medium) PSNR↑ / SSIM↑ / LPIPS↓ / FID↓	@4 (Sparse) PSNR↑ / SSIM↑ / LPIPS↓ / FID↓
PVG	23.33 / 0.73 / 0.41 / 39.47	22.08 / 0.71 / 0.41 / 47.25	20.47 / 0.67 / 0.45 / 70.94
DIFIX3D+	21.73 / 0.65 / 0.37 / 17.59	20.78 / 0.62 / 0.39 / 19.08	19.68 / 0.59 / 0.41 / 23.42
DiST-4D	19.26 / 0.66 / 0.36 / 13.42	18.47 / 0.62 / 0.39 / 15.68	17.55 / 0.58 / 0.43 / 20.64
FreeGen (Ours)	23.34 / 0.72 / 0.30 / 9.61	22.29 / 0.69 / 0.32 / 11.25	20.77 / 0.65 / 0.36 / 14.81

Ablation Study¶

Ablations conducted under a lateral shift of -2m across six camera views examine each architectural component (Table 3 & Table 4 in the original paper):

Component Breakdown (Table 3):

Configuration	Rec. Branch	Gen. Refinement	Opacity Guidance	Co-training	FID (-2m)↓	FVD (-2m)↓
(a) Feed-forward Rec. Only	✓	-	-	-	23.25	246.73
(b) + Color & Depth Refinement	✓	✓	-	-	15.49	60.53
(c) + Full Geometry Triplet (Opacity)	✓	✓	✓	-	13.86	57.21
(d) Full Model w/ Co-Training	✓	✓	✓	✓	12.87	48.39

Co-Training Dynamics (Table 4):

Training Regime	Rec. CT	Gen. CT	Rec. Only FID↓ / FVD↓	Rec. + Gen. FID↓ / FVD↓
Pre-trained Baseline	-	-	23.25 / 246.73	13.86 / 57.21
Fine-tune Rec. Only	✓	-	22.23 / 223.39	14.39 / 54.93
Fine-tune Gen. Only	-	✓	23.25 / 246.73	13.16 / 51.78
Full Alternating Co-Training	✓	✓	22.23 / 223.39	12.87 / 48.39

Key Findings¶

Reconstruction alone and generation alone both fail: Relying solely on the feed-forward reconstruction branch results in poor temporal video quality (FVD of 246.73) due to missing observations. Unconstrained diffusion models, on the other hand, suffer geometric drift as shifts widen (e.g., DiST-4D FVD deteriorates to 105.29 at ±4m).
Mutual co-training is essential to avoid distribution shift: Updating only the reconstruction branch changes the geometric condition distribution, causing the frozen diffusion model's final FID to degrade to 14.39. Alternating co-training simultaneously improves raw reconstruction rendering (FVD reduced from 246.73 to 223.39) and final refined output (FVD reduced from 57.21 to 48.39).
Opacity maps provide critical reliability masking: Incorporating rendered opacity maps drops FID from 15.49 to 13.86, demonstrating that rendering confidence cues guide the diffusion model to repair unobserved areas without hallucinating over accurate geometry.

Highlights & Insights¶

Elimination of test-time optimization: Prior works such as DIFIX3D+ and StreetCrafter distill generative priors via slow online gradient descent per scene. FreeGen integrates generative distillation directly into an offline co-training loop, enabling pure feed-forward inference in seconds.
Independence from costly 3D annotations: Unlike contemporary DiT-based models (e.g., WorldSplat) that require HDMaps and 3D bounding boxes, FreeGen relies purely on 2D surround videos and monocular depth priors, making it highly scalable for real-world driving fleets.
Reversible view supervision as spatial augmentation: Re-projecting synthesized off-trajectory views back to the original trajectory and supervising them with ground-truth frames transforms synthetic data generation into a self-regularizing, leak-free 3D data augmentation scheme.

Limitations & Future Work¶

Lack of explicit dynamic object decomposition: Dynamic actors and background environments are reconstructed jointly into static/global 3D Gaussians, which can leave faint ghosting artifacts around high-speed oncoming vehicles.
Extrapolation boundaries beyond ±4m: When lateral shifts exceed 4 meters, visual overlap with the original trajectory drops drastically, shifting the burden entirely onto diffusion hallucination and risking minor physical inconsistency.
Future directions: Integrating object-centric Gaussian representations and connecting FreeGen with end-to-end autonomous driving planners for closed-loop simulation.

vs PVG / StreetGaussian / OmniRe: Traditional per-scene 3DGS methods overfit to recorded trajectories and blur severely when extrapolated; FreeGen learns generalizable feed-forward priors and leverages diffusion refinement to synthesize sharp textures at unseen angles.
vs DiST-4D / FreeVS: DiST-4D utilizes image warping and LiDAR depth, leading to structural tears and missing distant structures; FreeVS suffers from sparse LiDAR projection. FreeGen's 3DGS representation provides dense, holistic geometry and opacity guidance.
vs WorldSplat / PhiGensis: WorldSplat and PhiGensis rely heavily on 3D spatial annotations and unidirectional refinement; FreeGen matches their generative realism using only image inputs while achieving superior FVD temporal coherence through bidirectional co-training.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ [Pioneering bidirectional co-training framework coupling feed-forward 3DGS with video diffusion for free-viewpoint driving scene generation]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive quantitative evaluation across multiple temporal and lateral displacement settings on nuScenes with rigorous ablations]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, well-formulated technical design, and self-consistent mathematical and qualitative presentation]
Value: ⭐⭐⭐⭐⭐ [Dispenses with expensive LiDAR/box annotations and online optimization, providing a highly scalable approach for closed-loop driving simulation]