AquaStereo: Enabling Underwater Stereo Matching via Depth-Conditioned Diffusion and Geometry Self-Distillation¶
Conference: ECCV 2026
Paper: ECCV Official
Area: 3D Vision
Keywords: Stereo Matching, Underwater Vision, Diffusion Models, Geometry Self-Distillation, Perception Enhancement
TL;DR¶
To tackle extreme data scarcity and severe feature degradation in underwater stereo matching, AquaStereo establishes a depth-conditioned diffusion pipeline with left-right consistency checking to synthesize geometry-faithful stereo datasets, combined with cross-domain geometry self-distillation from a frozen terrestrial teacher and a temporal perception-enhanced matcher, setting new zero-shot state-of-the-art accuracy without target-domain fine-tuning.
Background & Motivation¶
Stereo matching serves as a fundamental building block for robotic perception, 3D scene reconstruction, and autonomous underwater vehicle (AUV) navigation. While modern learning-based stereo algorithms exhibit impressive accuracy on terrestrial benchmarks, transferring these models directly to underwater environments induces severe performance degradation. Underwater imaging operates under drastically different physical principles: light attenuation, forward scattering, and backscatter severely degrade image contrast, induce wavelength-dependent color casts, and obscure discriminative textures. More critically, these optical corruptions are frequently view-dependent, directly violating the fundamental photometric consistency and left-right epipolar assumptions upon which modern cost-volume stereo architectures depend.
The primary bottleneck hindering progress in underwater depth estimation is the extreme scarcity and poor quality of dense supervisory ground truth. Capturing real underwater stereo data is prohibitively expensive and logistically challenging, where water turbidity, scattering, and refraction render active sensors like LiDAR and structured light severely range-limited, resulting in sparse, noisy, or distorted disparity labels. Conversely, synthetic datasets generated via computer graphics (CG) engines provide dense disparity maps but suffer from simplified optical approximations and lack realistic water appearance, opening a significant domain gap to real ocean imagery. Earlier attempts using GAN-based image translation frequently introduce non-rigid geometric distortions and cross-view misalignments that corrupt stereo correspondence.
To break this impasse, the authors abandon both expensive physical capture and simplistic CG rendering in favor of controllable generative modeling. Core idea: condition a ControlNet-guided diffusion model on monocular depth maps and physics-inspired prompts to synthesize diverse, geometry-preserving underwater stereo pairs, enforce epipolar coherence via a lightweight left-right consistency module, and couple cross-domain geometry self-distillation with a temporal perception-enhanced matcher to achieve robust zero-shot stereo matching.
Method¶
Overall Architecture¶
The AquaStereo pipeline consists of three core stages: controllable underwater stereo data generation, cross-domain geometry self-distillation, and disparity inference via a perception-enhanced stereo network. In the generation stage, monocular depth maps extracted from clean terrestrial stereo pairs and a physics-grounded text prompt pool guide a ControlNet-driven Stable Diffusion model to render diverse underwater appearances. A lightweight left-right consistency module filters epipolar drift to assemble the synthetic UW-Dataset (40K pairs). In the training stage, a frozen terrestrial teacher network provides multi-scale feature alignment for a student model processing perturbed underwater pairs, stabilized by pseudo-disparity supervision on clean inputs. During inference, the matcher expands the stereo input into a pseudo-temporal sequence using learnable perception frames, fusing temporal representations from a video backbone with high-level DINOv2 semantics before iterative disparity regression.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Terrestrial Stereo Pair & Monocular Depth"] --> B["1. Depth-Conditioned Diffusion Generation<br/>ControlNet + Physics-Inspired Prompt Pool"]
B --> C["2. Coherence-Enhanced Consistency Module<br/>Epipolar Patch Sampling & Disparity-Aware Confidence"]
C --> D["Synthetic Underwater Dataset UW-Dataset"]
D --> E["3. Cross-Domain Geometry Self-Distillation<br/>Frozen Terrestrial Teacher + Perturbation Alignment + Pseudo GT"]
E --> F["4. Perception-Enhanced Stereo Matcher<br/>Learnable Perception Frames + Video Backbone + DINOv2 Fusion"]
F --> G["High-Precision Disparity & Clean Point Cloud"]
Key Designs¶
1. Depth-Conditioned Diffusion Generation: Controllable Synthesis of Geometry-Preserving Stereo Pairs Supervised stereo networks require dense, metric disparity annotations that are virtually unobtainable at scale in natural marine environments. To overcome this limitation, this module leverages a pretrained foundation stereo model to extract high-quality pseudo disparities \(D_{\mathrm{GT}} = F_{\mathrm{FStereo}}(I_L, I_R)\) from terrestrial stereo pairs alongside monocular depth maps extracted via Depth-Anything V2. To realistically model water-type variations and degradation profiles, the authors construct a physics-inspired prompt pool \(\mathcal{P}\) summarizing optical absorption, backscatter, and veiling light equations coupled with CLIP-retrieved underwater descriptors. During sampling, a prompt \(p \sim \mathcal{P}\) and depth maps \((Depth_L, Depth_R)\) condition ControlNet to steer Stable Diffusion, synthesizing diverse underwater stereo pairs \((I'_L, I'_R)\) while strictly preserving the underlying 3D scene structure.
2. Coherence-Enhanced Consistency Module: Eliminating Epipolar Stereo Misalignment in Generative Diffusion Independent diffusion sampling across views easily introduces asymmetric artifacts and color shifts that break epipolar geometry and corrupt cost-volume construction. To enforce binocular fidelity, a lightweight cross-view consistency module samples patch pairs along corresponding epipolar lines. For left patch feature \(\mathbf{f}_i^L\) and right candidate feature \(\mathbf{f}_j^R\), the module computes a disparity-aware affinity score: $\(s_{ij} = \frac{\langle\mathbf{f}_i^L, \mathbf{f}_j^R\rangle}{\|\mathbf{f}_i^L\|_2 \|\mathbf{f}_j^R\|_2} (1 - \lambda) \exp\left(-\frac{|d_i - d_j|}{\tau}\right)\)$ where \(d_i, d_j\) represent coarse disparity priors, and \(\lambda\) balances visual similarity against geometric proximity. The normalized scores produce soft confidence modulation masks that modulate intermediate ControlNet features, slashing the cross-view inconsistency rate (Break px) from 15% to 5% with negligible runtime overhead, guaranteeing epipolar-compliant stereo pairs for downstream training.
3. Cross-Domain Geometry Self-Distillation: Mitigating Feature Degradation and Metric Scale Drift Directly feeding degraded underwater imagery into stereo models causes feature smearing and erroneous correspondences. To transfer clean geometric representations into the degraded domain, the authors institute a dual-branch self-distillation framework. The teacher network \(T\) processes clean terrestrial stereo images \((I_L, I_R)\) and remains completely frozen. The student network \(S\) receives perturbed underwater counterparts \((\tilde{I}'_L, \tilde{I}'_R) = \mathrm{Perturb}(I'_L, I'_R)\) subjected to simulated turbidity, severe color shifts, and patch mixups. Multi-scale feature representations are aligned via an \(\ell_1\) distance loss \(\mathcal{L}_{\mathrm{feat}} = \sum_{k=1}^K \|F_S(\tilde{I}'_L, \tilde{I}'_R) - F_T(I_L, I_R)\|_1\). Concurrently, a supervised loss on unperturbed pairs \(\mathcal{L}_{\mathrm{sup}} = \sum_{k=1}^K \|D_S(I'_L, I'_R) - D_{\mathrm{GT}}\|_1\) grounds the student to the shared pseudo ground-truth disparity \(D_{\mathrm{GT}}\), preventing metric scale drift and stabilizing convergence.
4. Perception-Enhanced Stereo Matcher: Incorporating Degradation Priors and High-Level Semantics In turbid or low-texture aquatic regions, conventional 2D CNN feature extractors struggle to extract discriminative cues due to poor cross-view communication prior to cost aggregation. The authors reframe the rectified stereo pair as a pseudo-temporal sequence by appending two learnable perception frames \(I_{P1}\) and \(I_{P2}\) to the stereo pair. Passing the 4-frame sequence through a video encoder \(\mathcal{F}_{\mathrm{enc}}\) facilitates cross-frame temporal attention as an effective proxy for cross-view feature alignment: $\(\mathbf{f}_{\mathrm{left}}, \mathbf{f}_{\mathrm{right}} = \mathcal{F}_{\mathrm{enc}}(I_{P1} \mathbin{\textcircled{c}} I_L \mathbin{\textcircled{c}} I_R \mathbin{\textcircled{c}} I_{P2})\)$ The learnable perception frames act as domain-adaptive visual prompts, absorbing underwater degradation priors and directing attention toward physical object boundaries. Simultaneously, frozen DINOv2 backbones extract high-level semantic features \(F^{\mathrm{sem}}_L, F^{\mathrm{sem}}_R\). Channel concatenation and linear projection produce robust matching descriptors \(p_L, p_R\) that feed into an IGEV++ cost-volume refinement head, yielding crisp, artifact-free disparity estimates.
Loss & Training¶
The entire network is trained end-to-end under the joint objective \(\mathcal{L} = \mathcal{L}_{\mathrm{feat}} + \lambda_{\mathrm{sup}} \mathcal{L}_{\mathrm{sup}}\) with balancing weight \(\lambda_{\mathrm{sup}} = 0.2\). Optimization is carried out on 4 NVIDIA RTX 4090 GPUs (24GB) using AdamW with an initial learning rate of \(10^{-4}\) and an effective batch size of 8 per GPU over 3 days. Image crops are set to \(384 \times 736\) pixels. Data synthesis is driven by Stable Diffusion v1.5 with ControlNet depth conditioning and a 50-step DDIM sampler, compiling 40,000 synthetic pairs in the UW-Dataset.
Key Experimental Results¶
Main Results¶
Zero-shot generalization is comprehensively evaluated on the physics-based synthetic benchmark UWStereo across four scene splits (Coral, Default, Industry, Ship), tracking mean End-Point Error (EPE) and the percentage of disparity outliers exceeding 3 pixels or 5% error (D1).
| Dataset / Scene Split | Metric | AquaStereo (Ours) | FoundationStereo | MonSter | IGEV | PSMNet |
|---|---|---|---|---|---|---|
| UWStereo (Coral) | EPE↓ / D1(%)↓ | 1.528 / 7.180 | 1.811 / 7.590 | 1.882 / 9.010 | 2.262 / 12.310 | 2.680 / 13.170 |
| UWStereo (Default) | EPE↓ / D1(%)↓ | 0.338 / 1.320 | 0.844 / 3.090 | 0.664 / 2.940 | 1.088 / 5.580 | 2.360 / 7.210 |
| UWStereo (Industry) | EPE↓ / D1(%)↓ | 0.373 / 1.510 | 0.718 / 2.490 | 1.061 / 4.870 | 1.990 / 7.590 | 2.850 / 10.580 |
| UWStereo (Ship) | EPE↓ / D1(%)↓ | 0.500 / 2.240 | 2.750 / 8.890 | 1.254 / 6.340 | 2.654 / 12.300 | 4.950 / 15.700 |
| UWStereo (Total) | EPE↓ / D1(%)↓ | 0.590 / 2.616 | 1.669 / 5.768 | 1.195 / 5.744 | 2.138 / 9.758 | 3.540 / 12.360 |
On real-world forward-looking underwater benchmarks, training IGEV++ on the proposed UW-Dataset achieves 3.05 EPE / 18.31% D1 on FLSea-Canyon and 1.80 EPE / 10.52% D1 on Squid, substantially outperforming models trained on SceneFlow (5.41 / 35.82%) and UE-rendered StereoAdapter (4.05 / 20.51%).
Ablation Study¶
Ablation experiments systematically isolate the impact of the left-right consistency module, cross-domain self-distillation schemes, and feature encoder architectures on the UWStereo benchmark.
| Module / Configuration | EPE↓ | D1(%)↓ | Note |
|---|---|---|---|
| Left-Right Consistency Module | |||
| Off (w/o consistency module) | 3.237 | 14.820 | 8.4s/pair gen. time, 15% inconsistent pixels |
| On (with consistency module) | 2.953 | 12.140 | 8.1s/pair gen. time, 5% inconsistent pixels |
| Self-Distillation Variants (IGEV++ Backbone) | |||
| Baseline (without self-distillation) | 2.9530 | 12.144 | Standard supervised training |
| Self-distillation (no weight sharing) | 2.7510 | 11.195 | Pure cross-domain feature alignment |
| Self-distillation + Shared feature extractor | 2.7740 | 11.262 | Tight parameter binding offers no advantage |
| Self-distillation + \(\ell_2\) regularization | 3.0510 | 13.838 | Over-constrains representation adaptation |
| Self-distillation + Supervised clean branch (\(\mathcal{L}_{\mathrm{sup}}\)) | 2.6350 | 10.549 | Anchors metric scale to prevent drift |
| Feature Extractor Architectures | |||
| Standard IGEV++ CNN backbone | 2.9530 | 12.144 | Lacks cross-view modeling prior to cost volume |
| VGG19 + DINOv2 | 3.0790 | 12.955 | High-level semantics without degradation cues |
| ResNet + DINOv2 | 3.1410 | 13.197 | Standard CNN struggles under turbidity |
| VGGT-style cross-frame attention | 3.4090 | 15.462 | Unguided attention gets corrupted by scatter |
| AquaStereo Perception-Enhanced Backbone (Ours) | 2.4830 | 10.824 | Fuses learnable perception frames and DINOv2 |
Key Findings¶
- Enforcing epipolar geometric coherence during diffusion is indispensable: disabling the LR-consistency module triples the rate of inconsistent pixels from 5% to 15%, degrading downstream stereo matching accuracy from 2.953 to 3.237 EPE.
- Metric scale calibration prevents unconstrained feature drift: while cross-domain distillation alone reduces EPE from 2.9530 to 2.7510, incorporating shared pseudo-target supervision (\(\mathcal{L}_{\mathrm{sup}}\)) on the clean branch further lowers EPE to 2.6350.
- Learnable perception frames act as degradation-adaptive visual prompts: attention maps confirm that incorporating perception frames shifts the network's focus away from floating backscatter particles and locks attention onto true structural silhouettes.
Highlights & Insights¶
- Physics-Informed Generative Data Engine: By wedding a depth-conditioned ControlNet with an LLM- and CLIP-curated prompt pool of oceanographic optical properties, the authors synthesize 40,000 photo-realistic, geometry-faithful underwater pairs, circumventing the twin traps of costly field capture and unrealistic CG rendering.
- Pseudo-Temporal Formulation for Cross-View Modeling: Conceptualizing a binocular stereo pair as a pseudo-temporal video clip flanked by learnable perception frames elegantly repurposed video attention mechanisms for stereo matching, providing intrinsic resilience to non-uniform haze and waterborne particles.
- Asymmetric Distillation with Clean Anchor Supervision: Pairing a frozen terrestrial teacher observing clean geometry with a student subjected to severe turbidity perturbations effectively injects canonical geometric priors into degraded underwater representations without scale distortion.
Limitations & Future Work¶
- Performance in Zero-Visibility Extreme Turbidity: Under catastrophic backscatter conditions where target structures are completely enveloped by dense floating particulate, diffusion priors and attention mechanisms can still experience boundary hallucination.
- Computational and Latency Overhead: Running a video backbone in tandem with large foundation vision encoders (DINOv2) introduces non-trivial memory and compute demands, presenting deployment hurdles for real-time edge computing on battery-constrained AUV platforms.
- Future Directions: Exploring lightweight, distilled perception encoders, integrating polarized underwater imaging cues, and investigating online physics-guided neural radiance field fine-tuning for dynamic ocean survey missions.
Related Work & Insights¶
- vs. StereoAdapter: StereoAdapter relies on Unreal Engine rendering, which fails to faithfully reproduce volumetric backscatter and wavelength attenuation; AquaStereo leverages generative diffusion with consistency checking, achieving vastly superior zero-shot performance on the real FLSea benchmark (3.05 vs. 4.05 EPE).
- vs. FoundationStereo: While FoundationStereo demonstrates formidable general stereo capability via massive terrestrial pretraining, severe underwater optical degradations degrade its cost-volume aggregation; AquaStereo surpasses FoundationStereo on UWStereo Total EPE by a wide margin (0.590 vs. 1.669).
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Seamlessly unifies depth-conditioned diffusion synthesis, epipolar consistency checking, and pseudo-temporal perception framing for underwater stereo.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated extensively across four diverse underwater benchmarks with rigorous ablations dissecting consistency, distillation, and encoder topologies.
- Writing Quality: ⭐⭐⭐⭐⭐ Exceptionally clear narrative, rigorous problem formulation, and highly informative qualitative and quantitative visualizations.
- Value: ⭐⭐⭐⭐⭐ Delivers an off-the-shelf, zero-shot robust stereo matching solution of immense practical utility for marine robotics, underwater archaeology, and oceanic exploration.