DriveWeaver: Point-Conditioned Video Inpainting for Controllable Vehicle Insertion in Autonomous Driving Simulation¶

Conference: ECCV2026
arXiv: 2606.31918
Code: https://github.com/LogosRoboticsGroup/DriveWeaver
Area: Autonomous Driving
Keywords: Autonomous Driving Simulation, Video Inpainting, Point Cloud Conditioning, Vehicle Insertion, Diffusion Model

TL;DR¶

DriveWeaver proposes a video diffusion inpainting framework using vehicle point cloud renderings as pixel-level conditions. By employing a lightweight PointAdapter to inject geometric conditions into a frozen Wan2.1 diffusion transformer backbone, it generates foreground vehicles with consistent lighting integration and precise geometry. Additionally, a global-local hierarchical inpainting strategy is designed to eliminate long-horizon drift, and the generation results can be further distilled into 3D Gaussian Splatting for real-time rendering.

Background & Motivation¶

Autonomous driving simulation requires inserting controllable foreground vehicles into recorded real-world scenes to construct rare long-tail corner cases for testing and improving driving models. Current mainstream approaches are roughly divided into two lines: First, pre-reconstructing 3D vehicle asset libraries from large-scale datasets (such as 3DRealCar) and then rendering the assets into scenes based on trajectories (e.g., HUGSIM). Although this rendering approach maintains the rigid geometric structure of vehicles, there are constant discrepancies in lighting and style between the inserted vehicles and the environment. Moreover, each inserted vehicle requires manual selection of light-matched models from the asset library and additional shadow synthesis to bridge the gap, making the process highly dependent on manual intervention and difficult to scale across scenes. Second, utilizing diffusion models as perceptual evaluators to end-to-end optimize lighting parameters through differentiable rendering and inverse optimization (e.g., UrbanCAD, RealEngine). These methods require separate optimization for each scene and rely on expensive CAD mesh assets, resulting in huge computational overhead, which is also unsuitable for large-scale deployment. The common bottleneck of both lines lies in: either relying on scarce high-quality pre-reconstructed assets or suffering from prohibitively high computational costs, lacking a low-cost, plug-and-play, cross-dataset generalizable vehicle insertion method.

The core insight of this paper is that: instead of relying on expensive 3D meshes or pre-reconstructed assets, vehicle point clouds obtained via LiDAR are inherently low-cost, easily accessible geometric conditions. However, point clouds are sparse; directly projecting them onto the image plane yields massive holes, making them unsuitable as precise guidance for video inpainting models. Core Idea: A point cloud rendering technique is proposed to rasterize sparse vehicle point clouds into dense pixel-level condition maps. By driving a video diffusion inpainting model with low-cost point clouds, high-fidelity and geometrically consistent vehicle insertion is achieved. A global-local hierarchical strategy is developed to eliminate autoregressive long-horizon drift, and the generation results can be further distilled into 3DGS representations to enable real-time rendering in simulation.

Method¶

Overall Architecture¶

DriveWeaver is built on the Wan2.1 VACE-14B video diffusion model, whose backbone Diffusion Transformer (DiT) and 3D VAE are fully frozen, with only a lightweight PointAdapter DiT branch introduced to inject geometric conditions. Given a multi-frame driving video, instance vehicle point clouds, and target trajectories as input, the system first performs point cloud rendering: the sparse projection of LiDAR points is rasterized by assigning a fixed radius to each point, generating a dense pixel-level point cloud condition map and a corresponding mask, while a target inpainting region mask is generated based on the 3D bounding box projection. Then, PointAdapter takes the channel-wise concatenation of the point cloud condition map, background RGB, inpainting mask, and point cloud mask as input, encodes them, and injects them into the feature space of every 5th Wan-DiT block via residual connections, driving the backbone to inpaint and generate foreground vehicles in designated regions. Inference adopts a two-stage strategy: the global anchoring stage generates a low-frame-rate anchor video in one pass for a \(4\times\) downsampled sparse sequence, establishing a globally consistent vehicle identity and ambient lighting; the local interpolation stage uses the anchor frames as dense guidance to fill in high-frame-rate details segment by segment. Finally, the generated yield is distilled into decoupled 3DGS representations via the urban reconstruction pipeline OmniRe, bypassing the diffusion model for real-time rendering in the simulation pipeline.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input: Driving Video<br/>Vehicle Point Cloud + Trajectory"] --> B["Point Cloud Rendering<br/>Rasterization to Dense Condition Map"]
    B --> C["Wan-DiT Backbone (Frozen)"]
    C --> D["PointAdapter<br/>Lightweight DiT Branch (3B)"]
    D -->|Residual modulation every 5 layers| C
    C --> E{"Long-Horizon Generation Strategy"}
    E -->|Global Anchoring| F["Temporal Downsampling x4<br/>Generate anchor video in one pass"]
    E -->|Local Interpolation| G["Anchor frames as dense guidance<br/>Segment-wise high-frame-rate interpolation"]
    F --> G
    G --> H["3DGS Distillation (OmniRe)<br/>Decoupled foreground instances"]
    H --> I["Real-time Rendering Output"]

Key Designs¶

1. Point Cloud Rendering: Constructing Dense Pixel-Level Conditions with Low-Cost Sparse Point Clouds

Existing methods rely on asset libraries or CAD meshes, which are expensive to acquire and generalize poorly. DriveWeaver instead uses vehicle instance point clouds as geometric conditions. In each frame, LiDAR points are first projected onto the calibrated image plane and colored (by querying corresponding pixel values). Then, point clouds of each vehicle instance are extracted using 3D bounding boxes and aggregated into a canonical coordinate system. Based on the target pose and camera pose, all instance point clouds are transformed, accumulated within a temporal window, and projected onto the image plane. The key trick is that direct projection leaves numerous holes and occlusions. Therefore, a fixed radius is assigned to each point before point rasterization, generating a dense pixel-level condition map. Meanwhile, a target inpainting region mask (reactive area) is generated by projecting the 3D bounding box, marking which pixels need to be regenerated and which should remain unchanged. The elegance of this design is that the model is free to render textures and lighting over the geometric "skeleton" indicated by the point cloud, which decouples geometric precision constraints from appearance generation freedom. Furthermore, point clouds from any source (cross-dataset or even non-LiDAR point clouds) can be directly fed as conditions, making the model highly generalization-capable across datasets during inference.

2. PointAdapter: Freezing Large Models and Injecting Geometric Conditions with Lightweight Residuals

Directly fine-tuning all 14B parameters of the Wan backbone on limited autonomous driving data yields catastrophic forgetting—ablation studies reveal that FVD doubles from \(22.46\) to \(62.59\), and the massive amount of texture and motion knowledge learned during pre-training is completely washed away by the small amount of simulation data. To address this, DriveWeaver designs a lightweight PointAdapter branch: an independent DiT network with approximately 3B parameters, maintaining the same token dimension as Wan-DiT but with significantly fewer layers. Input processing is split into two paths: the point cloud condition map and background RGB image are encoded into latent features via a 3D VAE; both masks are reshaped and interpolated to align with the same latent resolution, and then concatenated along the channel dimension. Each PointAdapter block is paired with every 5th Wan-DiT block, and its output is injected into the backbone feature space in a residual manner via a linear projection. This residual modulation mechanism allows geometric guidance to act on the backbone network layer-by-layer while keeping pre-trained weights frozen, achieving high-quality conditional generation with only \(\approx 1/5\) of trainable parameters. Training requires only 20k steps (5 days on 8×H200).

3. Global-Local Hierarchical Inpainting: Non-Autoregressively Eliminating Long-Horizon Drift

The number of frames a video diffusion model can generate in a single pass is constrained by GPU memory (Wan2.1 supports up to 49 frames), whereas autonomous driving trajectories typically span hundreds of frames (the baseline here is set to 193 frames). Traditional sliding-window or autoregressive methods use the last frame of the previous segment as the condition for the next segment, accumulating errors over time and causing the vehicle's appearance to drift and its shape to deform. DriveWeaver's strategy decouples global consistency from local details: first, the original sequence is downsampled \(4\times\) (\(193 \rightarrow 49\) frames), allowing the entire time span to be covered by a single forward pass of the model to generate a coarse-grained anchor video. This step establishes globally consistent vehicle identity, lighting, and pose. Then, the original high-frame-rate sequence is divided equally into 4 segments, each utilizing the corresponding anchor frames (rather than sparse point clouds) as dense guidance conditions to execute local high-frame-rate inpainting. The model is trained in dual modes: GT-Free mode relies only on sparse point clouds for condition generation (used in the global anchoring stage), while GT-Aware mode learns local interpolation utilizing ground-truth frames as supervision (used in the local interpolation stage). This strategy reduces FVD from \(29.41\) in the autoregressive baseline to \(22.46\), and the generated videos display excellent temporal consistency, directly satisfying the strict geometric consistency requirements for downstream 3DGS reconstruction.

Key Experimental Results¶

Main Results (Waymo Dataset, Short-Horizon + Long-Horizon)¶

Method	FID↓	FVD↓	LPIPS↓	PSNR↑(Geometry)	SSIM↑(Geometry)
HUGSIM (Asset library composition)	49.06	325.89	0.304	36.05	0.973
StreetCrafter (Point cloud generation)	26.88	295.06	0.294	30.31	0.878
Wan2.1 VACE (Masked inpainting baseline)	54.67	264.74	0.171	34.47	0.956
DriveWeaver	15.50	80.17	0.157	34.69	0.960
DriveWeaver-D (3DGS distillation)	22.35	119.36	0.168	—	—

DriveWeaver significantly outperforms baseline methods on all visual realism metrics. HUGSIM maintains a slight advantage in geometric consistency metrics (PSNR/SSIM) due to its use of true 3D geometric assets, but its rendered vehicles present severe stylistic discrepancies with the environment (with an FID as high as 49.06). StreetCrafter's autoregressive generation suffers from temporal inconsistency, leading to failures in geometric reconstruction (with a PSNR of only 30.31). A consistent trend is observed on PandaSet, verifying the robustness of the method across datasets.

Ablation Study¶

Configuration	FID↓	FVD↓	LPIPS↓	Key Conclusion
Full Model	17.50	22.46	0.159	—
w/o PointAdapter (Full parameter fine-tuning)	28.82	62.59	0.166	FVD doubled, causing catastrophic forgetting of pre-trained knowledge
w/o Point Cloud Rendering (Using raw sparse projection)	21.08	28.53	0.160	Sparse conditions lead to blurriness and lack of high-frequency textures
w/o Global-Local Hierarchy (Using autoregressive instead)	18.31	29.41	0.160	Appearance drifts and shape deforms over long horizons

Key Findings¶

PointAdapter's residual modulation is significantly superior to full fine-tuning: with only 3B trainable parameters and a frozen pre-trained backbone, it preserves the texture and motion priors learned by the large model, yielding an FVD 40 points lower than full fine-tuning.
Dense rasterization in point cloud rendering is crucial compared to raw sparse projections: without rasterization, the model only receives hollow and incomplete point cloud maps, losing high-frequency texture details and increasing FVD by 6 points.
The global-local hierarchical strategy exhibits significant performance on long horizons (193 frames): the autoregressive approach yields an FVD of 29.41, while the hierarchical strategy reduces it to 22.46, verifying the central role of non-autoregressive global anchoring in eliminating temporal drift.
DriveWeaver-D, which undergoes 3DGS distillation, exhibits only minor degradation in visual realism compared to the raw generation results (FID 22.35 vs 15.50), but gains real-time rendering capabilities, which is of great practical value for closed-loop simulation environments.

Highlights & Insights¶

Replacing high-cost 3D assets with low-cost point clouds: The simple operation of point cloud rendering rasterization turns sparse LiDAR points into pixel-level conditions that video diffusion models can interpret. This allows point clouds from any source to be plug-and-play, eliminating the tedious process of asset library building and lighting matching.
Validation of the "frozen large model + lightweight adaptation" paradigm: PointAdapter demonstrates that injecting geometric conditions via a lightweight module is far more effective than fully fine-tuning a large model on limited autonomous driving data. This provides a reference paradigm for adapting large models to vertical scenarios.
Non-autoregressive global-local hierarchical scheme: This approach elegantly decouples two stages to avoid the typical "long-horizon implies autoregressive" mindset. By taking the downsampling \(\rightarrow\) single-pass generation \(\rightarrow\) segmented interpolation path, it resolves temporal consistency issues with a clean concept and remarkable efficacy.

Limitations & Future Work¶

Training data remains dependent on LiDAR point clouds, rendering it not directly applicable to simulation scenes with camera-only configurations (though point clouds from other sources can still be used as conditions during inference, offering some flexibility).
The 3DGS distillation phase requires approximately 10 minutes of offline optimization per instance. Although much faster than frame-by-frame inference, it still inflicts latency on interactive simulations requiring millisecond-level responses.
Currently, only vehicle insertion is supported. Future work can extend this to other traffic participants, such as pedestrians and cyclists, as well as scenarios involving multi-object editing.
In the two-stage pipeline of global anchoring and local interpolation, the quality of anchoring directly dictates downstream interpolation. If the anchor frames generate incorrect vehicle appearances, the subsequent local inpainting can hardly rectify it.

vs HUGSIM: HUGSIM relies on rendering combinations of pre-reconstructed 3D vehicle asset libraries, which is geometrically precise but suffers from poor lighting fusion. DriveWeaver completes vehicle appearance at the image level using generation priors from diffusion models, achieving natural coherence in lighting and style with the environment, and is unrestricted by asset library categories.
vs StreetCrafter: StreetCrafter also utilizes LiDAR point clouds as conditions to generate scenes but adopts an autoregressive generation strategy, which incurs severe geometric deformation over long horizons. DriveWeaver's global-local hierarchical strategy fundamentally bypasses this error accumulation.
vs Wan2.1 VACE: VACE supports mask-driven video-to-video generation, but mask guidance alone cannot constrain rigid geometry, leading to unstable generated vehicle shapes. DriveWeaver addresses this by introducing explicit geometric constraints via point cloud conditions.

Rating¶

Novelty: ⭐⭐⭐⭐ The framework design of driving video inpainting for vehicle insertion through point cloud rendering is novel, indicating an ingenious combination of low-cost geometric conditions with the generation priors of diffusion models.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive quantitative and qualitative comparisons are provided on both Waymo and PandaSet, covering both short- and long-horizon tracks. The ablation studies cover three core designs, showcasing rigorous experimental design.
Writing Quality: ⭐⭐⭐⭐⭐ The paper is well-structured with complete methodological descriptions. The conclusions of the main and ablation experiments are strongly supported, and the pipeline illustrations in Fig. 2 and Fig. 3 are accurate and intuitive.
Value: ⭐⭐⭐⭐⭐ Autonomous driving simulation scene editing addresses a practical pain point in industrial demand. The low-cost vehicle insertion solution has excellent practical prospects. Its framework design (point cloud conditioning + frozen large model adaptation layer + hierarchical generation) is highly transferable to other scene editing tasks.