WaterGen: Decoupling Scene and Medium in Underwater Image Generation

Conference: ECCV 2026
Paper: ECCV Paper
Code: https://github.com/jiayi-wu-umd/WaterGen
Area: Image Generation / Underwater Vision
Keywords: underwater image generation, latent diffusion model, scene-medium decoupling, underwater image formation model, conditional decoding

TL;DR

WaterGen decouples underwater image generation into degradation-free semantic scene synthesis and physics-grounded conditional decoding, achieving independent and precise control over scene geometry and water medium parameters (attenuation and backscatter) while consistently boosting downstream restoration and segmentation performance as a scalable synthetic data engine.

Background & Motivation

Underwater visual exploration is vital for marine biology, maritime archaeology, and subsea infrastructure inspection. However, developing robust computer vision algorithms—such as object detection, image restoration, and semantic segmentation—is severely bottlenecked by the extreme scarcity of diverse, high-quality annotated underwater datasets. Capturing images in aquatic environments is logistically arduous and expensive, plagued by wavelength-dependent light attenuation, severe scattering, and low contrast. Furthermore, collecting accurate ground-truth depth and dense pixel-wise semantic annotations using divers or autonomous underwater vehicles (AUVs/ROVs) under harsh visibility and strict regulatory conditions remains impractical at scale.

Existing underwater synthesis paradigms fail to simultaneously deliver semantic diversity and physical fidelity. Model-based approaches rely on the Underwater Image Formation Model (UIFM) to degrade terrestrial images, inheriting strong semantic domain bias and lacking native underwater structures. Data-driven translation models (e.g., GANs) are constrained by fixed source domain layouts, only transferring superficial color casts while ignoring depth-dependent optical physics. Meanwhile, fine-tuning large-scale latent diffusion models directly on degraded underwater images entangles scene content with optical scattering into a monolithic "underwater style." Consequently, diffusion backbones produce over-saturated, cartoonish imagery with poor prompt adherence, and altering medium-related text prompts frequently distorts the underlying object geometry.

The fundamental tension stems from the mismatch between the tasks: scene synthesis is a high-level semantic generative task requiring global contextual reasoning and diversity, whereas water medium degradation is inherently an extrinsic, radiometric, pixel-wise physical process governed by optical parameters and scene depth. Injecting degradation into the diffusion backbone disrupts the pre-trained denoising prior and fails to enforce fine-grained radiometric control. Core Idea: decouple underwater image synthesis into clean semantic scene latent generation via a fine-tuned diffusion backbone, and physics-consistent medium degradation via a multi-scale conditional decoder, enabling independent "one scene, multiple waters" controllable synthesis.

Method

Overall Architecture

WaterGen decomposes underwater image generation into two sequential stages: Semantic Scene Latent Diffusion and Multi-scale Medium-conditioned Decoding. In Stage 1, an SDXL diffusion backbone adapted with LoRA is fine-tuned exclusively on restored, degradation-free underwater images to synthesize a pristine scene latent \(z_{clean}\) conditioned on a text prompt. In Stage 2, physical water parameters \(\Phi = \{B^\infty, \beta^D, \beta^B\}\) (background light, direct attenuation, and backscatter coefficients) together with a depth estimation module generate pixel-wise transmission and backscattering maps. These physical condition maps are encoded and injected via Zero-Convolutions into the image decoder to reconstruct the final degraded underwater image while preserving scene geometry.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Text Prompt + Water Parameters"] --> B["Stage 1: Clean Scene Latent Diffusion<br/>LoRA models pristine underwater semantics"]
    B --> C["Clean Scene Latent"]
    C --> D["Stage 2: Medium-Conditioned Decoding<br/>Physical parameter modulation via Zero-Conv"]
    D --> E["Paired Physical Outputs<br/>Degradation-free Scene / Degraded Image"]
    E --> F["Downstream Tasks<br/>Restoration / Semantic Segmentation"]

Key Designs

1. Clean Scene Latent Diffusion: Degradation-Free Semantic Modeling

Directly fine-tuning diffusion backbones on degraded underwater imagery causes latent space corruption, resulting in stylized artifacts and degraded prompt adherence. To establish a strong inductive bias towards clean underwater semantics, the authors curated WaterGen-Clean by processing images from six standard underwater datasets (UIIS10K, USIS10K, SUIM, USOD10K, UIEB, and UF7D) using the state-of-the-art SLURPP restoration model, paired with detailed BLIP-2 captions. During Stage 1 training, the VAE and base U-Net are frozen, and only low-rank adaptation layers (LoRA, rank=32, \(\alpha=16\)) within the attention blocks are optimized against pristine scene latents. Calculating denoising gradients strictly against degradation-free targets forces the diffusion model to learn native underwater structures (corals, marine life, divers) while treating residual medium artifacts as noise to be eliminated, effectively preserving latent high-frequency fidelity.

2. Multi-Scale Medium-Conditioned Decoding: Pixel-Wise Physical Injection via Revised UIFM

Light attenuation and scattering in water are continuous physical processes determined by distance and optical parameters. According to the Revised Underwater Image Formation Model:

\[I_c = J_c \cdot e^{-\beta^D_c d} + B_c^\infty \cdot (1 - e^{-\beta^B_c d})\]

where \(c \in \{R, G, B\}\) denotes the color channel, \(d\) is scene depth, \(J\) is the water-free scene radiance, \(B^\infty\) is background light at infinity, and \(\beta^D, \beta^B\) denote direct attenuation and backscattering coefficients, respectively. WaterGen operationalizes this formation model during decoding. A monocular depth estimation module (Depth Pro) infers metric depth \(d\) from the clean representation, allowing explicit analytical formulation of the transmission map \(T_c = e^{-\beta^D_c d}\) and backscattering map \(B_c = B_c^\infty \cdot (1 - e^{-\beta^B_c d})\). These maps are processed by a medium encoder and injected into the latent image decoder via Zero-Convolutions across multiple scales. By confining physical modulation to the decoding stage, WaterGen inherently supports "one scene, multiple waters": a single latent \(z_{clean}\) can be rendered under diverse water types (DeepBlue, Green, Turbid, etc.) by merely altering \(\Phi\), without altering the underlying spatial geometry.

3. Bidirectional UIFM Consistency & Stochastic Medium Noise Injection

When training the conditional decoder, relying solely on synthetic pairs can cause the network to overfit as a naive weighted superposition module, while remaining vulnerable to subtle medium residuals in the Stage 1 latents. The authors construct \((J, I, \Phi)\) triplets by applying physical UIFM sampling to clean terrestrial images. During training, a stochastic medium noise injection strategy applies slight random physical perturbations to input images prior to encoding. This forces the decoder to disregard latent color biases and strictly follow the injected parameters \(\Phi\). Furthermore, each iteration performs a dual forward pass (\(\Phi=0\) for clean reconstruction \(\hat{J}\) and \(\Phi=\Phi_{input}\) for degraded synthesis \(\hat{I}\)), penalized by a Bidirectional UIFM Consistency loss:

\[\mathcal{L}_{UIFM} = \|\hat{I} - (\hat{J} \cdot T + B)\|_1 + \left\|\hat{J} - \frac{\hat{I} - B}{T}\right\|_1\]

Enforcing consistency in both the forward degradation (\(\hat{J} \to I_{gt}\)) and inverse restoration (\(\hat{I} \to J_{gt}\)) pathways ensures algebraic and physical closure, effectively preventing color drift and error accumulation.

Loss & Training

Stage 1 is optimized with standard diffusion MSE denoising loss. Stage 2 trains the conditional latent decoder on a single NVIDIA A6000 GPU in approximately 1 day using Adam with a learning rate of \(10^{-5}\). The total Stage 2 objective is:

\[\mathcal{L}_{stage2} = \mathcal{L}_{rec}(I) + \mathcal{L}_{rec}(J) + \lambda_{uifm}\mathcal{L}_{UIFM}\]

where the reconstruction loss combines multi-scale objectives \(\mathcal{L}_{rec} = \lambda_1 L_1 + \lambda_{ssim} L_{SSIM} + \lambda_{lpips} L_{LPIPS}\). The loss weights are empirically set to \(\lambda_1 = 1.0\), \(\lambda_{ssim} = 1.0\), \(\lambda_{lpips} = 0.5\), and \(\lambda_{uifm} = 0.3\).

Key Experimental Results

Main Results

Evaluated across 5,451 SynTIDE prompts against state-of-the-art baselines Atlantis and TIDE, WaterGen demonstrates superior perceptual quality and semantic alignment. Following standard protocol to isolate scene fidelity, baseline generations are post-processed with SLURPP restoration and compared against WaterGen's native degradation-free outputs (\(B=0, T=1\)) using reference-free metrics (UIQM, MUSIQ) and CLIP Score (Table 1).

Method	UIQM ↑	MUSIQ ↑	CLIP Score ↑	Controllability
Atlantis (CVPR 2024)	2.8338 ± 0.1927	67.5437 ± 1.7373	0.2457 ± 0.0274	Text-only
TIDE (CVPR 2025)	2.3725 ± 0.3816	66.4304 ± 2.1780	0.2305 ± 0.0118	Text-only
WaterGen (Ours)	3.0239 ± 0.1317	69.2638 ± 0.8813	0.2614 ± 0.0073	Text + Medium

To validate downstream utility as a synthetic data engine: 1. Underwater Image Restoration: Augmenting training with 20,000 paired WaterGen images consistently enhances restoration baselines on the real-world UIIS10K benchmark across both UIQM and MUSIQ (Table 2).

Model	UIQM Baseline	UIQM (+Ours)	MUSIQ Baseline	MUSIQ (+Ours)
WaterNet	3.067	3.141 (+0.074)	66.866	68.322 (+1.456)
Phaseformer	2.239	2.272 (+0.033)	67.400	69.165 (+1.765)
DeepWaveNet	2.685	2.731 (+0.046)	66.843	67.937 (+1.094)
Histoformer	2.994	3.048 (+0.054)	66.178	66.924 (+0.746)

1. Underwater Semantic Segmentation: Using 40,000 paired clean/degraded images with pseudo-masks extracted on clean scenes, training SegFormer on Real + WaterGen data achieves 75.6% mIoU on UIIS (+5.4% over Real Only) and 76.7% mIoU on USIS10K (+2.1% over Real Only), outperforming SynTIDE augmentation.

Ablation Study

Medium control precision was quantified by extracting background light parameters from generated images and computing Root Mean Square Error (RMSE), Mean Angular Error (MAE), and CIEDE2000 color difference (\(\Delta E_{00}\)) against the input condition (Table 4).

Medium Injection Mechanism / Config	RMSE (↓)	MAE (↓)	\(\Delta E_{00}\) (↓)	Note
ControlNet + SDXL	0.45	21.05°	43.70	Severe color and attenuation drift in U-Net
T2I-Adapter + SDXL	0.30	9.96°	30.10	Adapter fails to capture low-frequency physics
Ours (w/o Degradation Injection)	0.07	7.01°	7.27	Decoder overfits to residual latent style
Ours (w/o Bidir UIFM Consistency)	0.07	7.31°	6.90	Slight degradation mapping drift without inverse constraint
Ours (Full Model)	0.06	4.42°	5.44	Decoupled decoder yields highest physical fidelity

Key Findings

• Decoder vs. U-Net Injection: Standard adapter mechanisms (ControlNet, T2I-Adapter) injected into the diffusion U-Net fail dramatically on medium control (\(\Delta E_{00} = 43.70\) and \(30.10\)). Denoising U-Nets are optimized for high-level structure and conflict with low-frequency radiometric shifts. Relocating medium modulation to the decoder drops \(\Delta E_{00}\) to 5.44.
• Bidirectional Physics Consistency Prevents Drift: Omitting the inverse formulation in the UIFM loss increases MAE from 4.42° to 7.31°, confirming that mathematical reversibility is crucial for grounding the physical simulation.
• Clean Pseudo-Labeling Solves Annotation Deficits: Segmenting clean generated scenes with off-the-shelf models produces razor-sharp masks that remain perfectly aligned with degraded counterparts, effectively mitigating false negatives under heavy backscatter and turbidity.

Highlights & Insights

• Physically Grounded Scene-Medium Decoupling: Allocating semantic layout to the diffusion latent space and radiometric degradation to the conditional decoder cleanly divides generative imagination from physical simulation.
• Scalable "One Scene, Multiple Waters" Data Engine: By freezing the clean latent \(z_{clean}\) and varying optical coefficients \(\Phi\), WaterGen effortlessly synthesizes perfectly registered degradation pairs across arbitrary water types without structural parallax.
• Noise-Injected Decoder Invariance: Injecting stochastic physical noise into Stage 2 training prevents the decoder from memorizing latent artifacts, enforcing strict adherence to explicitly specified physical parameters.

Limitations & Future Work

• Simplified Optical Assumptions: The framework relies on single-scattering, homogeneous medium assumptions (JM / revised UIFM), omitting localized artificial lighting (flashlight caustics), forward multiple-scattering blur, and surface wave refraction.
• Dynamic Particulate Matter Simulation: High-turbidity marine snow and large suspended particulates are modeled as diffuse attenuation rather than discrete volumetric particles with independent dynamics.
• Future Directions: Extending the decoupled latent formulation to 3D representations (e.g., 3D Gaussian Splatting) or video diffusion models for view-consistent underwater novel view synthesis and trajectory simulation.

Related Work & Insights

• vs. Atlantis (CVPR 2024): Atlantis guides Stable Diffusion using depth ControlNets but trains directly on degraded underwater images, leading to stylized, hyper-saturated scenes and text-only medium control. WaterGen decouples the pipeline, yielding higher fidelity (UIQM 3.02 vs. 2.83) and explicit parameter control.
• vs. TIDE (CVPR 2025): TIDE attempts joint generation of images and dense annotations in a single unified model, resulting in blurry object boundaries. WaterGen generates pristine scenes to yield superior pseudo-labels, consistently outperforming TIDE on downstream restoration and segmentation benchmarks.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Pioneering decoupling of scene semantics and optical medium degradation within latent diffusion frameworks]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Comprehensive evaluations across synthesis fidelity, physical control accuracy, restoration, and segmentation tasks]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, mathematically rigorous formulation, and well-structured comparative analysis]
• Value: ⭐⭐⭐⭐⭐ [Serves as both a state-of-the-art generative framework and an indispensable synthetic data generator for subsea robotics and vision]