Semantic-Aware, Physics-Informed, Geometry-Grounded Weather Synthesis

Conference: ECCV 2026
Paper: ECCV Official
Code: https://jumponthemoon.github.io/w-crafter/
Area: Autonomous Driving
Keywords: Weather Synthesis, Video Editing, Particle Simulation, Physical Prior, Geometry Grounding

TL;DR

By disentangling weather synthesis into semantic anchoring, physics-informed anisotropic Gaussian particle simulation, and 3D geometry grounding, this work effectively elicits latent weather and particle priors from off-the-shelf video diffusion models to synthesize realistic, temporally coherent, and controllable adverse weather videos.

Background & Motivation

Synthesizing adverse weather conditions (such as torrential rain or heavy snowfall) in natural videos is of paramount importance for safety-critical autonomous driving perception, visual robustness evaluation, and visual effects. However, real-world paired video data under extreme meteorological events is notoriously dangerous and difficult to collect. Current approaches for weather video generation fall into two primary paradigms, both facing fundamental limitations. 3D reconstruction-based pipelines (e.g., NeRF or 3D Gaussian Splatting) inject weather effects into explicit or implicit neural scene representations and re-render the sequence. While providing interpretable control, they suffer from tedious per-scene optimization, brittle sensitivity to 3D reconstruction artifacts, and handcrafted particle behaviors that lack natural diversity. Conversely, data-driven approaches retrain video diffusion models on synthetic datasets curated by graphics engines, inherently binding appearance to the simulator's style distribution and demanding costly re-training whenever adapting to new domains.

Meanwhile, general-purpose conditional video diffusion editors fail to handle heavy adverse weather. Because these foundation models are heavily trained on pristine, aesthetically clean, high-resolution video footage, their latent priors instinctively penalize and suppress severe environmental degradations and occluding particles. When prompted with simple text instructions like "make it snowy" or "heavy rain", existing video editors yield only subtle global discoloration or mild dimming, completely failing to produce dense falling snowflakes or streak-like raindrops. Furthermore, directly superimposing 2D animated particle overlays without grounding them in the scene's 3D gravity orientation and camera ego-motion leads to severe visual drift and parallax tearing.

The key insight of this paper is that pretrained video diffusion models have not lost the ability to render weather degradations; rather, their latent particle priors remain dormant because text-only prompts are underspecified and unable to override the model's clean bias. Injecting structured geometric and dynamic particle primitives into the conditioning space immediately awakens dense precipitation generation without requiring any model fine-tuning. Core idea: decompose weather synthesis into a tri-prior conditioning framework of "semantics (what to look like), dynamics (how to evolve), and geometry (where to appear)", grounding global atmospheric styles via VLM/LLM-based first-frame anchoring, simulating anisotropic Gaussian particle fields via physics dynamics, and projecting particles into camera depth space to steer off-the-shelf video diffusion models in a zero-shot manner.

Method

Overall Architecture

The proposed framework comprises three sequential, decoupled stages: Semantic-Aware Appearance Anchoring, Physics-Informed Dynamic Simulation, and Geometry-Grounded Video Synthesis. Taking an original video sequence as input, the pipeline produces a synthesized video enriched with realistic rain/snow dynamics and scene-consistent lighting and wetness degradations.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    In["Input Raw Video Sequence"] --> P1["1. Semantic-Aware Appearance Anchoring<br/>VLM parsing + LLM reasoning + I2I anchor frame"]
    In --> G0["Geometry Foundation Model<br/>3D point map + depths + camera trajectory"]
    P1 --> GAlign["3. Geometry Grounding & Projection<br/>Ground plane gravity alignment + camera projection"]
    G0 --> P2["2. Physics-Informed Dynamic Simulation<br/>Gaussian particle field evolved via gravity/wind/turbulence"]
    P2 --> GAlign
    GAlign --> Cond["Construct Tri-Prior Conditioning<br/>Appearance anchor A + particle dynamic maps {Et} + scene depths {Gt}"]
    Cond --> Out["Off-the-shelf Video Diffusion Model<br/>Zero-shot realistic weather video synthesis"]

This workflow seamlessly bridges explicit physical mechanics with implicit generative modeling: it establishes surface wetness, snow cover, and overcast illumination in an initial reference anchor; numerically integrates particle kinetics under physical forces within a canonical coordinate frame; dynamically projects particles using estimated camera parameters and aligned gravity; and finally feeds the particle-augmented depth along with appearance prompts into an off-the-shelf video diffusion model (e.g., Wan2.1-VACE) for temporally coherent rendering.

Key Designs

1. Semantic-Aware Appearance Anchoring: Resolving Text Ambiguity and Underspecification

Generic text modifiers like "heavy snow" fail to guide image generators toward fine-grained atmospheric cues such as wet reflective asphalt or overcast scattering. To resolve this ambiguity, a Vision-Language Model (VLM) first parses the initial frame \(I_0\) into a structured description capturing scene attributes, objects, and lighting conditions \(P_{\mathrm{scene}} \leftarrow \mathrm{VLM}(I_0)\). An LLM then combines this semantic context with the user's intent \(U\) (specifying weather type, duration category such as short/medium/long, and severity) to formulate a detailed, executable image editing instruction: $\(P_{\mathrm{img}} \leftarrow \mathrm{LLM}(P_{\mathrm{scene}}, U)\)$ An image-to-image model (FLUX.1-Kontext Pro) edits the initial frame using \(P_{\mathrm{img}}\), yielding an edited reference frame \(I'_0\) with realistic ground-level wetness, snow accumulation, and global diffuse lighting. Simultaneously, the LLM summarizes a video editing prompt \(P_{\mathrm{vid}}\) that encapsulates the global dynamic atmosphere. By compositing the simulated particle primitives onto \(I'_0\) to form \(\tilde{I}_0\), the pair \(\mathcal{A}=(\tilde{I}_0, P_{\mathrm{vid}})\) serves as a robust appearance anchor that constrains the global style and prevents semantic drift across video generation.

2. Physics-Informed Dynamic Simulation: Continuous Particle Fields and Turbulent Advection

To reliably trigger precipitation particles without requiring task-specific fine-tuning, weather particles are instantiated as a persistent field of anisotropic Gaussian primitives. Each particle \(i\) has a 2D local spatial footprint parameterized by an anisotropic Gaussian kernel: $\(G_i(s) = \exp\left(-\left(\frac{x^2}{2\sigma_{x,i}^2} + \frac{y^2}{2\sigma_{y,i}^2}\right)\gamma\right)\)$ where \((\sigma_{x,i}, \sigma_{y,i})\) govern particle elongation (e.g., elongated streaks for rain droplets or rounded shapes for snowflakes), and \(\gamma\) is a depth-derived scale factor. Each particle carries mass \(m\) and 3D coordinate \(\mathbf{x}_i \in \mathbb{R}^3\), evolving over time according to classical Newtonian mechanics: $\(m\dot{\mathbf{v}}_i(t) = \mathbf{F}_g + \mathbf{F}_w + \mathbf{N}_t(\mathbf{x}_i(t), t)\)$ where gravity \(\mathbf{F}_g = mg \mathbf{d}_g\) and wind force \(\mathbf{F}_w = \alpha \mathbf{d}_w\) guide macro-level trajectories. To introduce stochastic swirling perturbations characteristic of atmospheric precipitation, a divergence-free turbulence field \(\mathbf{N}_t(\mathbf{x}, t) = \beta (\nabla \times \boldsymbol{\psi}(\mathbf{x}, t))\) is incorporated via curl noise derived from 3D Perlin noise potentials \(\boldsymbol{\psi}\). The trajectory is discretized using explicit Euler integration, generating volume-preserving, physically plausible particle dynamics.

3. Geometry-Grounded Video Synthesis: Gravity Alignment and Camera-Consistent Projection

Canonical particle simulations assume an arbitrary vertical down vector, which contradicts real-world camera viewpoints featuring pitch, roll, and dynamic ego-motion. Directly pasting particles onto image planes causes implausible floating trajectories and parallax collapse. To anchor the dynamics in real scene geometry, a foundation geometry model (Depth Anything V3) extracts a dense 3D point map \(S_p\), per-frame depth maps \(D\), and camera extrinsics/intrinsics \((K_t, R_t, \mathbf{t}_t)\) from input video \(V\). The method fits a dominant ground plane equation \(\pi: \mathbf{c}^\top \mathbf{p} + d = 0\) over sampled 3D points and assigns the inverted normal as true gravity \(\mathbf{d}'_g = -\mathbf{c}\). A minimal Rodrigues rotation aligns the canonical gravity vector with \(\mathbf{d}'_g\), ensuring that precipitation falls perpendicularly toward the actual ground regardless of camera inclination. Next, particles \(\mathbf{x}_i^t\) are projected onto the image plane using camera parameters \((K_t, R_t, \mathbf{t}_t)\) to obtain pixel coordinates \(\mathbf{u}_{i,t}\), while each particle stamp is dynamically oriented along its projected velocity vector \(\theta_{i,t} = \mathrm{atan2}((\mathbf{v}_i^t)^y, (\mathbf{v}_i^t)^x)\). The rendered particle maps are fused with scene depth to yield particle-augmented depth maps \(\{\mathcal{E}_t\}\), which together with original depth \(\{\mathcal{G}_t\}\) and appearance anchor \(\mathcal{A}\) steer Wan2.1-VACE to produce the final video \(\tilde{V}\).

Key Experimental Results

Main Results

Quantitative evaluations are conducted across 58 video sequences from DAVIS, DL3DV-10k, PandaSet, and nuScenes. Comparisons include prompt-only baselines (CogVideoX, WeatherEdit) and conditional diffusion editors (LTX-Video, VACE). Metrics include CLIP Directional Alignment (CLIP-D), CLIP Semantic Similarity (CLIP-S), and GPT-4V Multimodal Quality Score (VLM).

Method	Snow CLIP-D ↑	Snow VLM ↑	Rain CLIP-D ↑	Rain VLM ↑	Overall CLIP-D ↑	Overall CLIP-S ↑	Overall VLM ↑
CogVideoX	0.08	0.56	0.06	0.52	0.07	0.82	0.54
WeatherEdit	0.14	0.45	0.12	0.36	0.13	0.91	0.41
LTX-Video	0.15	0.68	0.12	0.67	0.13	0.82	0.67
VACE	0.18	0.76	0.14	0.74	0.16	0.82	0.75
Ours	0.19	0.84	0.15	0.81	0.17	0.81	0.83

Note: Table 1 from the original paper. WeatherEdit achieves high CLIP-S because its modifications are minimal and conservative; in contrast, our method achieves superior prompt alignment (CLIP-D) and perceived visual realism (VLM) while preserving underlying scene structure.

Downstream Perception and Realism Evaluation

To assess the practical value for autonomous driving perception under adverse weather, two domain-adaptive segmentation models (DAFormer and HRDA) are fine-tuned using data synthesized by our method and evaluated on ACDC and MUSES benchmarks (mIoU, %):

Baseline Model	ACDC (Snow)	ACDC (Rain)	MUSES (Snow)	MUSES (Rain)
DAFormer	50.9	50.8	41.4	33.8
DAFormer + Ours	56.3 (+5.4)	56.6 (+5.8)	47.2 (+5.8)	41.5 (+7.7)
HRDA	50.0	51.6	40.4	35.4
HRDA + Ours	60.4 (+10.4)	66.1 (+14.5)	50.4 (+10.0)	44.4 (+9.0)

Note: Table 4 from the original paper. In addition, in a user study with 46 participants (5-point Likert scale), our method achieved 4.16 in photo-realism (vs. 3.58 for VACE) and 4.12 in physical-realism (vs. 3.52 for VACE); user preference rate reached 41.3% for rain and 56.5% for snow (Table 2 and Table 3).

Key Findings

• Substantial Downstream Perception Gains: Incorporating our synthesized adverse weather data yields remarkable robustness gains, improving HRDA mIoU by +9.0% on MUSES rain (35.4% to 44.4%) and +14.5% on ACDC rain (51.6% to 66.1%), confirming that physics-grounded weather synthesis provides high-utility training data for real-world perception.
• Criticality of Geometry & Gravity Alignment: Ablation experiments confirm that omitting ground-plane gravity matching causes precipitation to fall vertically across tilted camera views, violating real physics. Omitting camera projection prevents particles from displaying forward parallax and approaching ego-motion, destroying 3D spatial coherence.
• Hallucination Suppression via Structured Anchoring: Feeding particle-altered depth maps directly into video diffusion without detailed first-frame semantic anchoring causes the model to hallucinate objects, as depth perturbations conflict with the model's structural priors. The detailed semantic prompt contextualizes these depth perturbations as legitimate atmospheric phenomena.

Highlights & Insights

• Activating Latent Priors over Retraining: Demonstrates that large video foundation models inherently possess precipitation generation capabilities that are suppressed by training curation biases; explicit physical and geometric guidance activates these capabilities zero-shot.
• Bridging Classical Simulation with Modern Generative Models: Seamlessly embeds curl noise fluid turbulence and anisotropic Gaussian primitives into modern latent diffusion conditioning streams, granting interpretable slider-like control over wind direction, speed, and particle geometry.
• Scalable Corner Case Engine for Autonomous Systems: Enables synthetic generation of safety-critical weather extremes directly from sunny driving logs, establishing an effective data flywheel for autonomous vehicle perception.

Limitations & Future Work

• Cascading Upstream Sensitivity: The pipeline relies on upstream foundation models for depth estimation and plane fitting. Scenes lacking a distinct ground plane (e.g., steep mountainous terrains or aerial drone views) can lead to suboptimal gravity alignment.
• Simplified Surface Interactions: Current particle dynamics primarily focus on airborne motion; surface splash dynamics, water film pooling, vehicle tire spray, and windshield wiper wiping marks are not explicitly simulated.
• Inference Runtime Overhead: While preprocessing requires ~3 minutes per video, video diffusion synthesis dominates execution time at ~38 minutes on a single NVIDIA RTX A6000 GPU, limiting immediate interactive simulation.

Related Work & Insights

• vs. WeatherWeaver (ICCV 2025): WeatherWeaver fine-tunes video diffusion models on synthetic simulator data, causing synthesized outputs to inherit simulator domain gaps; this method operates zero-shot on an off-the-shelf editor, yielding superior photorealism and generalizability.
• vs. ClimateNeRF / RainyGS: Neural rendering approaches require dense multi-view captures and expensive per-scene optimization; this method operates on in-the-wild, single-view driving videos with moving cameras.

Rating

• Novelty: ⭐⭐⭐⭐☆ [Disentangles weather synthesis into semantic, physical, and geometric priors, activating dormant precipitation priors in video foundation models zero-shot.]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive quantitative evaluation, human studies, ablation analyses, and domain-generalization segmentation benchmarks on ACDC and MUSES.]
• Writing Quality: ⭐⭐⭐⭐⭐ [Exemplary structure, crisp motivation, and rigorous mathematical formulations.]
• Value: ⭐⭐⭐⭐⭐ [Provides a scalable data engine for generating rare adverse-weather corner cases in autonomous driving.]

Related Papers

• [ECCV 2026] A Physics-Grounded Benchmark for Multi-Agent Dynamics in World Models
• [ECCV 2026] Real-Time LiDAR Gaussian Splatting SLAM via Geometry-Aware Covariance Coupling
• [ECCV 2026] FreeGen: Feed-Forward Reconstruction–Generation Co-Training for Free-Viewpoint Driving Scene Synthesis
• [ECCV 2026] Sparse-Aware Vector Quantization for Bandwidth-Efficient Collaborative 3D Semantic Occupancy Prediction
• [ECCV 2026] MobileOcc: A Human-Aware Semantic Occupancy Dataset for Mobile Robots