AutoWeather4D: Autonomous Driving Video Weather Conversion via G-Buffer Dual-Pass Editing

Conference: ECCV2026
Paper: Official Page | PDF
Area: Autonomous Driving / Video Editing
Keywords: Weather conversion, G-buffer, geometry-lighting disentanglement, local relighting, video diffusion

TL;DR

AutoWeather4D extracts metric geometry and materials from driving videos, edits weather interactions and light transport separately, and uses constrained video diffusion to improve sensor realism, reaching a 0.2586 CLIP score and 0.915 vehicle detection IoU on Waymo; its main advantage is spatially grounded, parametric weather and lighting control rather than a large numerical lead over generative baselines.

Background & Motivation

Turning a sunny driving video into rain, snow, fog, or night requires more than recoloring it. Rain changes road roughness and reflections, snow should accumulate on upward-facing surfaces, and headlight illumination should depend on vehicle position, orientation, and distance. Video generators can synthesize convincing overall atmosphere, but text alone offers limited control over the spatial influence of a particular lamp and may change vehicle outlines or background structures. Real long-tail weather footage is also expensive and difficult to collect, making edits to existing videos attractive.

Prior NeRF, 3DGS, and 4DGS weather editors anchor effects in scene representations, but per-scene optimization is costly and moving traffic complicates reconstruction. More importantly, if original shadows and illumination are already baked into appearance, converting the scene to fog or night can leave incompatible lighting behind. This paper does not address that problem by training a larger weather generator. Instead, it exposes depth, normals, and materials as editable state, changes the surfaces first, and then recomputes how they should be illuminated.

Core idea: represent dynamic scenes with feed-forward metric G-buffers, modify surfaces and weather particles in a Geometry Pass, resolve local and global illumination in a Light Pass, and restrict video diffusion to refining the resulting physically grounded sequence.

Method

Overall Architecture

The inputs are a driving video and target weather, lighting, or HDR environment settings; the output is an edited video that retains the original traffic layout. The pipeline consists of metric G-buffer extraction, the Geometry Pass, the Light Pass, and VidRefiner. The two editing passes distinguish what surfaces become from how those surfaces appear under the new illumination.

Here, "feed-forward" primarily means avoiding per-scene NeRF or Gaussian-field optimization, not that every stage requires only one network evaluation: the final stage still performs diffusion denoising. Nor is the system a scale-free monocular solution. Physical distances require sparse LiDAR calibration or a camera-height prior.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Driving video and edit settings"] --> B["Metric G-buffer Extraction"]
    B --> C["Geometry Pass"]
    C --> D["Light Pass"]
    D --> E["VidRefiner"]
    E --> F["Weather and lighting edited video"]

Key Designs

1. Metric G-buffer Extraction: make geometric distance meaningful for physical rendering

Pi3 provides spatiotemporally coherent relative depth, while the zero-shot inverse-rendering branch of DiffusionRenderer supplies albedo, normals, metallicity, and roughness. These sources serve different purposes: inverse rendering separates materials from illumination, whereas 4D reconstruction supplies continuous geometry, avoiding staircase artifacts in light attenuation caused by quantized depth. The explicit, pixel-aligned state can update with moving objects instead of forcing traffic into an optimized scene representation dominated by static-scene assumptions.

Relative depth alone cannot specify how far headlights reach or how strongly fog attenuates visibility. In non-sky, non-occluded regions, the system fits a global scale and bias against sparse LiDAR. Without LiDAR, it fits a road plane to predicted points and recovers scale from camera height. Because sky depth is unstable, it is clamped to the 99th percentile of the sequence's non-sky depth distribution, with a sky mask separating invalid regions to prevent spurious local illumination there. The per-frame state is:

\[ G_t=\{D_t,N_t,A_t,R_t,M_t,S_t\}. \]

Here \(D_t\) is metric depth, \(N_t\) the normal map, \(A_t\) albedo, \(R_t\) roughness, \(M_t\) metallicity, and \(S_t\) the valid-scene/sky mask. This state definition is fully legible in the source. The detailed scale-solving procedure is deferred to supplementary material absent from the cache, so a particular robust-estimation equation cannot be inferred from the main text.

2. Geometry Pass: modify weather-affected surfaces rather than overlay a visual filter

Snow requires several spatial scales to coexist. Surface buildup uses metaballs and iterative evaluation of an SPH Poly6 kernel over normal maps, restricting accumulation to upward-facing structures. Ground coverage uses a grid and procedural density patterns, while thawing wetness lowers albedo and roughness. Airborne snowflakes use temporally persistent screen-space particle rasterization rather than independent framewise resampling. Together, these operations update weather-dependent surface descriptors instead of reducing snow to a white texture.

Rain is likewise divided into airborne motion and ground interaction. Drop velocity combines Gunn-Kinzer terminal velocities with parametric wind: the vertical component reflects gravity-drag equilibrium, while wind determines horizontal displacement. Trajectories are represented as volumetric SDFs and depth-tested against scene geometry. Fractional Brownian Motion, or FBM, generates puddle masks; within those regions, the method changes albedo and roughness and perturbs normals with procedural ripples. The next pass therefore receives a road that is already wet and locally rippled, allowing lighting to respond to those changes. The main text does not provide complete kernel, trajectory, or perturbation formulas and parameters, so none are reconstructed here.

3. Light Pass: control local lights separately from atmosphere and environment illumination

After the geometry edits, the Light Pass evaluates illumination over the modified materials. Streetlights and vehicle lights are modeled as 3D spotlights, with surface radiance evaluated through a Cook-Torrance BRDF. The same headlight beam can therefore produce different responses on dry asphalt and puddles. Unilluminated regions receive a parametric LUT adjustment toward warm nocturnal tones while retaining minimal visibility. That is a visual processing choice, so the entire output should not be described as a fully physical light-transport solution.

Vehicle lights also follow state constraints. An off-the-shelf 3D detector and tracker obtain vehicle cuboids; headlights and taillights occupy fixed relative positions on their front and rear faces. Spotlight direction follows vehicle heading, and the cones are depth-tested. Headlights activate only for moving vehicles at night or in low-visibility fog, while parked vehicles remain unlit. Streetlights are identified through semantic masks and lifted into 3D using metric depth before their radiance is accumulated with vehicle lights. The controls therefore address explicit source positions, directions, and switches rather than relying on prompts to determine overall brightness indirectly.

Fog uses a single-scattering radiative-transfer model with a Henyey-Greenstein phase function. Metric depth determines distance-dependent visibility attenuation and local light halos. In regions with sparse geometry, an HDR-conditioned neural forward renderer supplies global ambient illumination, which is linearly blended with the local-light result. The Light Pass thus combines analytical local lighting, a physical atmospheric model, and neural environment rendering; it is not a purely analytical global-illumination solver. The main text does not expose the blending weights or full scattering equation, so a textbook formula should not be presented as the implementation.

4. VidRefiner: improve camera realism while limiting generative freedom

Deterministic rendering establishes physical relationships but may lack the texture and sensor characteristics of real footage. Built from WAN-FUN 2.2, VidRefiner follows an SDEdit-style process: encode the rendered sequence into VAE latents, perturb them at a pivot timestep \(t_s\), and reverse the diffusion process. Because initialization comes from the edited video, layout, color distribution, and coarse illumination are already present instead of having to be regenerated from a text description.

Initialization alone can still drift. The system therefore extracts spatiotemporally coherent boundaries from the rendered sequence and concatenates them channel-wise at the input of an existing multi-channel conditioning backbone. Low-frequency initialization anchors global state, while boundary conditioning constrains high-frequency spatial structure. Together, they aim to keep refinement inside the resolved geometric and weather configuration. The authors report no secondary fine-tuning, but structural invariance remains a design goal supported by observations rather than a mathematical guarantee for arbitrary denoising strengths; the main text does not specify a numerical \(t_s\).

A Worked Example

Consider converting a daytime road with moving vehicles into a rainy night as a conceptual walkthrough, not an additional experiment. Depth and scale calibration first locate the road, vehicles, and sky. Rain particles are then generated, and depth tests determine whether they pass in front of or behind a vehicle. The Geometry Pass lowers road roughness and adds puddles and normal-map ripples. The Light Pass activates headlights according to tracked vehicle positions and headings, illuminating the already wet road while changing environment lighting.

VidRefiner finally adds texture using the rendered sequence and its boundaries as constraints. To isolate headlight effects, weather and scene state can remain fixed while light switches change. To study rainfall, rain parameters can change while vehicle trajectories and layout remain fixed. This is the practical distinction from prompt-only editing, although system-level driving robustness evaluation remains future work.

Loss & Training

The paper does not introduce a unified loss for retraining an end-to-end model. It combines pretrained geometry, inverse-rendering, detection, tracking, and video diffusion modules through parametric physical editing, without per-scene optimization or extra weather-data fine-tuning. The cached main text omits full VidRefiner sampling settings, physical parameters, and comprehensive ablations, referring to supplementary material instead; reproducibility details cannot be described as complete.

Downstream HRDA semantic-segmentation fine-tuning is a separate validation experiment: it uses 6,480 augmented frames for 20k iterations. That training setup should not be confused with the training cost of AutoWeather4D itself.

Key Experimental Results

Main Results

Evaluation uses the NOTR subset of Waymo: 120 source videos, each converted into rain, snow, fog, and night, with 57 frames per condition, totaling 27,360 frames. Most methods run on V100 GPUs, while resource-intensive baselines such as Cosmos-Transfer2.5 and Ditto use A100 GPUs. Those differing configurations do not directly establish a speed advantage.

The CLIP score measures adherence to the target weather or time of day. According to the main-text protocol, vehicle detection IoU compares projected 2D LiDAR ground-truth boxes with extractions from a monocular 3D detector. Despite the table heading "Vehicle 3D Detection IoU," it is not simply a reported 3D volumetric IoU. Vehicle CLIP cosine similarity compares local features before and after editing as a proxy for identity preservation.

Method	CLIP Score, higher better	Vehicle Detection IoU, higher better	Vehicle CLIP Cosine Similarity, higher better	Human Evaluation, higher better
Video-P2P	0.2448	N/A	N/A	0
Ditto	0.2532	0.805	0.769	0.425
Cosmos-Transfer2.5	0.2558	0.913	0.837	0.580
WAN-FUN 2.2	0.2577	0.888	0.794	0.668
AutoWeather4D	0.2586	0.915	0.871	0.826

Values come from Table 3. Missing Video-P2P entries result from frame cropping, not zero performance. Relative to Cosmos, the IoU advantage is only 0.002, while vehicle feature similarity improves by 0.034. The appropriate conclusion is competitive quality with explicit control, not a large or statistically established lead on every metric.

The user study uses 12 raters, 512 by 512 resolution, and 30-frame clips, assessing spatial fidelity and temporal coherence. The main text states 10 paired comparisons per baseline and 1,440 aggregated responses but does not fully explain how those counts map to each method's table entry. The original scores are therefore retained without interpreting 0.826 as a universal win rate against all baselines.

Ablation Study

The cache contains only the reconstruction ablation in main-text Figure 5. Combined-module, rain/snow-parameter, and VidRefiner-conditioning-strength ablations are deferred to unavailable supplementary material. The following is explicitly qualitative; it does not invent numerical drops.

Configuration	Main-text observation	Supported conclusion
Inverse-rendering depth without 4D reconstruction	Integer-quantized depth produces jagged local relighting and spatial steps	Depth discretization harms continuous light attenuation
With feed-forward 4D reconstruction	Continuous floating-point geometry produces smoother illumination gradients	Continuous geometry directly benefits local relighting
Other module and parameter ablations	No full numerical results in the main text; referred to supplementary Section 16	The available cache cannot rank individual module contributions

To distinguish improved visual appearance from preservation of reusable labels, the paper also tests HRDA augmentation under the Cityscapes class taxonomy. The following reports absolute Table 4 scores, not a mixture of those scores and the relative percentages in its parentheses.

Augmentation	ACDC mIoU, higher better	ACDC mAcc, higher better	Dark Zurich mIoU, higher better	Dark Zurich mAcc, higher better
No augmentation	49.20	60.72	23.92	38.29
Cosmos augmentation	49.66	62.31	23.93	39.52
AutoWeather4D augmentation	49.81	62.52	24.09	39.73

Key Findings

• ACDC mIoU increases by 0.61 percentage points over no augmentation, and Dark Zurich by 0.17 points. The respective margins over Cosmos are 0.15 and 0.16 points. These small gains lack confidence intervals and are best viewed as preliminary evidence for label usability, not a major segmentation breakthrough.
• Main-text Table 2 reports 128.1 seconds for shared semantic annotation on V100, followed by 167.1, 170.9, 2.2, and 67.6 seconds for night, fog, rain, and snow. These timings concern core editing, not established end-to-end latency including full G-buffer extraction and VidRefiner; the entire rainy-video pipeline cannot be claimed to take 2.2 seconds.
• Parametric fog density, snow density, and individual light switches enable continuous perturbations. However, the reported CLIP, IoU, and user evaluations primarily assess editing quality, not comprehensive physical-parameter calibration accuracy or driving-policy safety.

Highlights & Insights

• Controls operate on materials and light sources. Puddles change roughness before headlights are evaluated, coupling weather and illumination in an interpretable way. This is more meaningful than independent rain-texture and brightness overlays.
• Resolve structure before allowing generative detail. Latent initialization and boundary conditioning jointly constrain VidRefiner. The approach is useful for label-preserving augmentation: explicit representations determine geometry while generation supplies realism.
• Judge depth quality by its downstream use. Quantization that appears tolerable in ordinary visualization can become conspicuous under distance-sensitive relighting. Figure 5 suggests evaluating reconstructed geometry through the continuity of subsequent lighting edits.

Limitations & Future Work

• The authors explicitly identify complex long-tail fluid interactions, such as vehicle splashes, as difficult for the decoupled pipeline. Local generative priors could address those unstructured effects without relaxing geometry constraints globally.
• Real occlusion in heavy fog conflicts with retaining distant structures. The authors suggest semantic-aware attenuation masks to regulate diffusion strength; overly strong preservation of original boundaries can itself contradict realistic visibility.
• From the method's dependencies, errors in scale, sky masks, material estimation, and vehicle tracking can propagate into light localization and weather occlusion. The main text does not fully quantify these effects, and monocular road-plane fitting still requires a camera-height prior.
• From the evidence available, missing supplementary material prevents verification of comprehensive ablations, blending coefficients, and full reproduction settings. End-to-end cost, statistical significance of downstream gains, and editing generalization across datasets also need fuller reporting.

Related Work & Insights

• Relative to ClimateNeRF, RainyGS, and WeatherEdit: these methods also ground weather in 3D representations, whereas AutoWeather4D uses feed-forward G-buffers to avoid per-scene field optimization and explicitly adds local relighting. WeatherEdit itself uses 4D Gaussians, so these approaches should not all be labeled static-only.
• Relative to DiffusionRenderer: the paper reuses inverse rendering and HDR-conditioned forward rendering, adding metric 4D geometry, weather-driven surface operations, and state-aware local lights. Its contribution is a unified editable pipeline, not the invention of inverse rendering.
• Relative to WAN-FUN 2.2 and Cosmos-Transfer2.5: generative editors model overall appearance well but provide less direct access to local physical quantities. Using WAN-FUN 2.2 as a terminal refiner illustrates how explicit physics and large models can complement one another.

Rating

• Novelty: 4/5. Combining metric G-buffers, weather-surface interactions, and state-aware local lights is well targeted; the main novelty lies in system integration rather than new physical models.
• Experimental Thoroughness: 3/5. Four conditions, several generative baselines, and downstream augmentation are covered, but comprehensive main-text ablations, full costs, and statistical uncertainty are missing.
• Writing Quality: 4/5. The geometry-to-lighting-to-refinement narrative is clear, but many implementation details are deferred and parts of the human-evaluation protocol need clarification.
• Value: 4/5. The approach is useful for controllable weather data generation and diagnostic scenarios, but the evidence does not establish improved closed-loop driving safety.