RefracGS: Novel View Synthesis Through Refractive Water Surfaces with 3D Gaussian Ray Tracing

Conference: ECCV 2026
Paper: ECCV Official Page
Full Cache: /Users/zy/workspace/paper_cache/ECCV2026/eccv-3476.txt
Area: 3D Vision
Keywords: 3D Gaussian Splatting, Dynamic Water Surface Refraction, Neural Height Field, Differentiable Ray Tracing, Novel View Synthesis

TL;DR

Addressing the non-linear optical distortion and multi-view inconsistency of underwater scene reconstruction through non-planar refractive water surfaces, this paper introduces a hybrid surface representation combining a contiguous neural height field with coarse-to-fine recursive subdivision tracing, coupled with an end-to-end differentiable 3D Gaussian ray tracing pipeline that backpropagates gradients through Snell's law to optimize the surface geometry, achieving ~15x faster training, 200+ FPS real-time rendering, and high-fidelity novel view synthesis.

Background & Motivation

Novel view synthesis (NVS) of underwater scenes captured through non-planar refractive water surfaces is a long-standing challenge with critical applications in shallow water bathymetry, environmental monitoring, and underwater cultural heritage preservation. In standard in-air novel view synthesis, light travels along straight paths, allowing radiance field representations such as NeRF and 3D Gaussian Splatting (3DGS) to achieve sub-millimeter high-fidelity reconstruction by relying heavily on linear projective geometry and multi-view epipolar consistency. However, when light rays penetrate dynamic, undulating water-air interfaces, mismatched refractive indices cause abrupt changes in ray directions. This refraction-induced non-linear trajectory bending fundamentally invalidates standard multi-view consistency and linear projection assumptions.

Existing refractive neural radiance field techniques (such as NeRFrac) explicitly account for ray bending within volumetric rendering frameworks, but they suffer from prohibitive computational costs, requiring hours of training and operating at sub-FPS rendering speeds. Even more detrimentally, many existing approaches parameterize the refractive interface using view-dependent representations—such as predicting surface normals as functions of individual camera rays—which lacks multi-view geometric consistency across varying viewpoints, frequently causing surface geometry drift and severe ghosting artifacts in the underlying scene. While 3DGS has demonstrated exceptional speed through tile-based rasterization, and recent extensions like 3DGRT introduce ray tracing to handle secondary light transport, their formulations remain non-differentiable with respect to refractive boundary geometry; meanwhile, methods tailored for transparent solid objects (e.g., TransparentGS) fail to model continuous and wavy fluid boundaries.

The core tension lies in reconciling high-frequency water wave dynamics with fast, differentiable ray tracing that maintains global multi-view geometric consistency. The critical angle of attack is to explicitly decouple the refractive boundary from the underlying scene, establishing a representation that combines mesh-like hardware ray intersection efficiency with continuous implicit field fidelity. Core idea: explicitly decouple the refractive water surface and model it as a contiguous neural height field queried via coarse-to-fine recursive subdivision tracing, unified with a fully differentiable 3D Gaussian ray tracing pipeline under Snell's law that allows photometric loss gradients to backpropagate through refracted paths to jointly optimize surface geometry and the underlying scene.

Method

Overall Architecture

RefracGS aims to simultaneously reconstruct both the refractive water surface geometry and the underlying underwater 3D scene from multi-view images captured above water along with calibrated camera parameters. The overall pipeline integrates two synergistic innovations: (1) a Water Height Map, which serves as a hybrid refractive surface representation combining a continuous neural height field with a recursively subdivided triangular mesh proxy; and (2) a Refraction-Aware Gaussian Ray Tracing framework that tracks refracted rays governed by Snell's law and accumulates color and opacity over 3D Gaussian primitives with end-to-end gradient backpropagation.

During the forward rendering pass, a camera ray is cast for each pixel and tested against a global coarse proxy mesh accelerated by hardware Bounding Volume Hierarchies (BVHs). Intersected triangles are locally subdivided while unhit triangles are aggressively pruned. New vertices query the neural height field, and smooth surface normals are derived via barycentric interpolation. Snell's law is evaluated at the intersection point to compute the refracted ray's origin and direction. This refracted ray traverses the underlying scene, intersecting 3D Gaussian primitives and accumulating radiance via front-to-back alpha blending. In backpropagation, photometric and opacity losses deliver gradients not only to 3D Gaussian spatial and appearance attributes, but also across the refracted trajectory back to the surface intersection and normal, directly updating the MLP weights of the neural height field.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-view Camera Rays r = (o, d)"] --> B["Neural Height Field & Proxy Mesh<br/>Continuous 3D height field & global BVH query"]
    B --> C["Recursive Subdivision Tracing & Normal Smoothing<br/>Local triangle subdivision, pruning & barycentric interpolation"]
    C --> D["Refraction-Aware Gaussian Ray Tracing<br/>Snell's law refraction & Gaussian radiance accumulation"]
    D --> E["End-to-End Differentiable Optimization<br/>Photometric & opacity loss backpropagation"]

Key Designs

1. Neural Height Field & Proxy Mesh: Decoupling View-Consistent Refractive Surfaces

Prior implicit surface approaches parameterize refraction boundaries conditioned on camera ray origins and directions, which inevitably creates view-dependent ambiguities and violates physical spatial consistency. RefracGS observes that natural open water surfaces predominantly align horizontally across the XY-plane with waves oscillating along the vertical axis. Setting the upward vertical direction as the \(Z\)-axis, the refractive boundary is parameterized in world space as a continuous single-valued neural height field \(\mathcal{H}(x, y) = z\), defined implicitly as the zero isosurface \(\mathcal{H}(x, y) - z = 0\) via a lightweight MLP. Representing the surface directly in global Cartesian space guarantees strict multi-view geometric consistency across all camera perspectives. To avoid computationally prohibitive ray marching or iterative binary searches across implicit fields during rendering, the method predefines a coarse 2D triangular grid \(\mathcal{M}_{2D}(V_{2D}, F)\) and queries \(\mathcal{H}\) at its vertices to construct a 3D piecewise linear proxy mesh. Ray-surface intersection is thereby converted into standard ray-triangle intersection that natively leverages GPU RT Cores and BVH acceleration.

2. Recursive Subdivision Tracing & Normal Smoothing: Capturing Wave Details with Continuous Shading

A fixed-resolution coarse mesh cannot resolve high-frequency geometric ripples, while globally densifying the mesh leads to severe memory blowup and redundant MLP evaluations. RefracGS solves this dilemma through Recursive Subdivision Tracing. Utilizing the coarse BVH intersection mask, all non-intersected triangles are pruned away. Each hit triangle is then locally subdivided into four smaller sub-triangles (quaternary subdivision), and the neural height field \(\mathcal{H}\) is evaluated only at the newly generated sub-vertices. This coarse-to-fine local hierarchy avoids reconstructing the entire global BVH while maintaining field-level geometric accuracy. Furthermore, using raw discontinuous face normals from triangular facets causes severe step discontinuities across triangle edges, producing optical banding artifacts. Inspired by Phong shading, RefracGS computes smooth surface normals via barycentric interpolation:

\[\mathbf{n}_p = \frac{\alpha \mathbf{n}_{v_i} + \beta \mathbf{n}_{v_j} + \gamma \mathbf{n}_{v_k}}{\|\alpha \mathbf{n}_{v_i} + \beta \mathbf{n}_{v_j} + \gamma \mathbf{n}_{v_k}\|_2}\]

where \(\alpha + \beta + \gamma = 1\) are barycentric coordinates of intersection point \(p\) on triangle \((v_i, v_j, v_k)\), and vertex normals are obtained by averaging adjacent face normals. This yields a spatially smooth normal field essential for accurate directional refraction.

3. Refraction-Aware Gaussian Ray Tracing: Fully Differentiable Non-Linear Transport

While 3D Gaussian ray tracing methods such as 3DGRT allow forward traversal through particle fields, their interfaces treat reflection and refraction as non-differentiable transformations, precluding inverse reconstruction of unknown refractive surfaces. RefracGS closes this loop with an end-to-end differentiable path formulation. Given intersection point \(\mathbf{p} \in \mathbb{R}^3\) and interpolated unit normal \(\mathbf{n} \in SO(3)\), the refracted ray direction \(\mathbf{\omega}_o\) is computed according to the vector form of Snell's law:

\[\mathbf{\omega}_o = \eta (\mathbf{\omega}_i - c_1 \mathbf{n}) - c_2 \mathbf{n}\]

where \(\mathbf{\omega}_i\) is the incident ray direction, \(\eta = n_1 / n_2\) is the relative refractive index of air over water, \(c_1 = \mathbf{n} \cdot \mathbf{\omega}_i\), and \(c_2 = \sqrt{1 - \eta^2(1 - c_1^2)}\). The refracted ray originating at \(\mathbf{p}\) traverses the underlying scene, accumulating colors and opacities across intersected anisotropic 3D Gaussian primitives. During backward propagation, first-order partial derivatives of the rendered pixel color with respect to the refracted origin \(\mathbf{p}\) and direction \(\mathbf{\omega}_o\) are calculated and chained back to the surface normal and the MLP weights of \(\mathcal{H}(x, y)\). This unified gradient flow drives simultaneous convergence of both the water surface wave geometry and the underlying 3D scene parameters without requiring explicit 3D surface supervision.

Loss & Training

The framework is optimized end-to-end under multi-view photometric supervision, augmented with an opacity regularization penalty to suppress floating artifacts:

\[\mathcal{L}_{\text{total}} = \lambda_1 \mathcal{L}_1 + \lambda_2 \mathcal{L}_{\text{SSIM}} + \lambda_3 \mathcal{L}_\alpha\]

The photometric term combines \(\mathcal{L}_1 = \|\mathbf{C} - \mathbf{C}_{gt}\|_1\) and structural similarity \(\mathcal{L}_{\text{SSIM}} = 1 - \text{SSIM}(\mathbf{C}, \mathbf{C}_{gt})\). In sparsely viewed regions under complex refraction, 3DGS often forms dense floating Gaussians that occlude subsequent primitives and induce gradient vanishing. To prevent this, the opacity loss penalizes squared opacity values across all \(N\) Gaussian primitives:

\[\mathcal{L}_\alpha = \frac{1}{N} \sum_{i=1}^N o_i^2\]

The loss weights are set to \(\lambda_1 = 0.8\), \(\lambda_2 = 0.2\), and \(\lambda_3 = 0.007\). Built upon PyTorch and NVIDIA OptiX with custom CUDA kernels, the model trains for 15k iterations in approximately 10 to 12 minutes on a single NVIDIA RTX 4090 GPU.

Key Experimental Results

Main Results

Evaluation is performed across three benchmark settings: the NeRFrac Real and Synthetic datasets (8 training views looking vertically down), and the challenging RefracGS dataset featuring wide-baseline surround-view setups with steep incidence angles and ground-truth water surface geometry (24 training views, 6 test views). Baselines include general radiance field models (Mip-NeRF, TensoRF, Plenoxels, 3DGS, 3DGRT) and specialized refractive pipelines (WSGS, NeRFrac). Surface reconstruction accuracy is quantified by the root mean square error (RMSE, in cm) of ray-surface intersection distances against ground-truth geometry.

Dataset	Method	PSNR ↑	SSIM ↑	LPIPS ↓	Training Time ↓	Rendering FPS ↑	Surface RMSE(cm) ↓
NeRFrac Real	Mip-NeRF	10.385	0.221	0.916	~15 h	< 1	-
	TensoRF	22.973	0.809	0.139	11 min	< 1	-
	Plenoxels	14.867	0.389	0.631	9 min	61	-
	3DGS	27.711	0.864	0.130	3 min	857	-
	3DGRT	27.732	0.842	0.164	16 min	157	-
	WSGS†	22.449	0.679	0.279	106 min	129	-
	NeRFrac†	28.146	0.876	0.167	149 min	< 1	-
	Ours†	30.671	0.913	0.129	10 min	242	-
NeRFrac Synthetic	Mip-NeRF	12.414	0.382	0.870	~15 h	< 1	-
	TensoRF	16.223	0.532	0.601	11 min	< 1	-
	Plenoxels	13.334	0.407	0.806	12 min	72	-
	3DGS	22.186	0.730	0.300	3 min	704	-
	3DGRT	21.929	0.689	0.363	27 min	89	-
	WSGS†	20.868	0.726	0.271	107 min	153	-
	NeRFrac†	34.381	0.944	0.149	77 min	< 1	-
	Ours†	35.816	0.952	0.136	12 min	203	-
RefracGS	Mip-NeRF	13.874	0.479	0.808	~15 h	< 1	-
	TensoRF	14.125	0.362	0.658	13 min	< 1	-
	Plenoxels	13.983	0.383	0.790	12 min	23	-
	3DGS	17.482	0.428	0.486	5 min	593	-
	3DGRT	18.019	0.424	0.545	14 min	182	-
	WSGS†	17.915	0.424	0.484	105 min	124	-
	NeRFrac†	17.837	0.405	0.624	164 min	< 1	3.655
	Ours†	30.224	0.933	0.098	11 min	124	0.115

(Note: † indicates methods specifically tailored for refractive environments)

Ablation Study

The impact of each design component was evaluated across the NeRFrac datasets, as detailed in the table below:

Config	PSNR ↑	SSIM ↑	LPIPS ↓	Training Time ↓	Rendering FPS ↑	Note
(a) w/o Water Height Map (NeRFrac field)	27.782	0.869	0.168	22 min	57	View-dependent field causes multi-view inconsistency; PSNR drops by 5.09 dB
(b) w/o Normal Smoothing (raw facet normals)	32.643	0.928	0.133	11 min	232	Edge normal discontinuities introduce refractive optical artifacts
(c) w/o Recursive Intersection (fixed low-res proxy)	32.966	0.929	0.132	20 min	218	Training time nearly doubles from 11 min to 20 min without pruning
(d) w/o Opacity Loss \(\mathcal{L}_\alpha\)	31.927	0.920	0.125	12 min	224	Dense floaters appear in sparse-view regions, degrading PSNR by ~0.95 dB
Full Model	32.876	0.929	0.132	11 min	223	Best balance of synthesis fidelity, geometric accuracy, and computational efficiency

Key Findings

• Decoupled Height Field Enforces Multi-View Rigidity: Replacing the Water Height Map with NeRFrac's view-dependent refraction field causes PSNR to plummet from 32.88 dB to 27.78 dB and reduces rendering speed from 223 to 57 FPS, confirming that explicit world-space height field parameterization is essential for spatial consistency.
• Superior Robustness Under Steep Incidences: On the wide-angle RefracGS dataset, where rays strike water at sharp angles, NeRFrac completely collapses to 17.84 dB PSNR and 3.655 cm surface RMSE. In contrast, RefracGS maintains 30.22 dB PSNR and recovers the water surface with an RMSE of just 0.115 cm—achieving an order-of-magnitude reduction in geometric error.
• Efficiency of Recursive Tracing: The recursive subdivision strategy cuts training time in half (from 20 minutes to 11 minutes) through hit-mask triangle pruning, while yielding virtually identical perceptual and photometric fidelity.
• Downstream Water Removal and Mesh Extraction: By simply disabling ray bending during inference, RefracGS synthesizes pristine de-refracted views at 23.15 dB PSNR (compared to NeRFrac's 14.95 dB). These water-removed images are clean enough to directly feed surface extraction algorithms like PGSR to reconstruct watertight underwater 3D meshes.

Highlights & Insights

• Hybrid Implicit-Explicit Representation Synergy: Projecting a continuous neural height field onto an adaptive triangular mesh proxy harnesses hardware OptiX RT Core speed while retaining the resolution independence of continuous implicit functions.
• Differentiable Snell's Law for Particle Fields: This work bridges Gaussian ray tracing with analytical Snell's law derivation, allowing gradients from 2D pixel losses to backpropagate directly to the 3D refractive interface.
• Physical Decoupling for Downstream Editing: Decoupling the water surface from underwater 3D Gaussians allows zero-shot water removal and post-training surface editing (such as substituting real waves with synthetic sine meshes) without retraining.

Limitations & Future Work

• Single-Valued Height Field Assumption: Parameterizing the water boundary as \(H(x, y)\) assumes that light traverses at most one surface intersection along \(Z\), making it incapable of handling overturning breakers, splashing droplets, or turbulent foams.
• Omission of Surface Reflection and Participating Media: The current model models refraction geometry but omits Fresnel reflection glints, water surface caustics, and subsurface volumetric light scattering.
• Static Scene Limitation: The pipeline assumes fixed surface waves and static underwater geometry across multi-view captures; extending to dynamic video streams with moving waves remains an open challenge.
• Potential for 2D Gaussian Primitives: The underwater scene is parameterized using 3D Gaussians. Transitioning to 2D Gaussian Splatting could offer better-aligned surface normals for exact underlying mesh extraction.

Related Work & Insights

• vs NeRFrac: NeRFrac employs implicit volumetric NeRF and view-dependent ray bending, taking several hours to train and running at sub-FPS rates, while failing at oblique angles. RefracGS models the surface in world coordinates using 3D Gaussian ray tracing, boosting training speed by ~15x, rendering at 200+ FPS, and resolving geometric inconsistency.
• vs 3DGRT: 3DGRT provides forward ray tracing for 3D Gaussians but treats reflection/refraction interfaces as non-differentiable; RefracGS analytically derives gradients through Snell's law back to the surface MLP, enabling inverse reconstruction of unknown refractive surfaces.
• vs WSGS: WSGS strictly assumes an idealized flat water plane, causing severe artifacts on real undulating wave surfaces; RefracGS accommodates arbitrary non-planar wave geometry.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Pioneered fully differentiable 3D Gaussian ray tracing through complex non-planar refractive boundaries with recursive neural height field tracing.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous validation across real and synthetic datasets under both narrow and wide baselines, covering novel view synthesis, surface accuracy, ablations, and water removal applications.
• Writing Quality: ⭐⭐⭐⭐⭐ Mathematically sound derivations, intuitive system design figures, and thoroughly substantiated claims.
• Value: ⭐⭐⭐⭐⭐ Provides an efficient, physically grounded paradigm for shallow water bathymetry, underwater archaeology, and marine robotics.