title: >- [Paper Note] DR-GS: Physically-Based Deformable and Relightable 2D Gaussians description: >- [ECCV 2026][3D Vision][2D Gaussian Splatting] Decoupling geometry, materials, and illumination with multiple importance sampling and mesh ray tracing for physically plausible deformation and relighting. tags: - ECCV 2026 - 3D Vision - 2D Gaussian Splatting - Inverse Rendering - Deformable Gaussians date: 2026-09-19 content_hash: ef17ad2cd22e1fd1

DR-GS: Physically-Based Deformable and Relightable 2D Gaussians¶

Conference: ECCV 2026
Paper: ECCV Official Portal
Project: Project Page
Area: 3D Vision
Keywords: 2D Gaussian Splatting, Inverse Rendering, Physics-Based Rendering, Deformable Objects, Material Decoupling

TL;DR¶

DR-GS establishes a unified Gaussian framework integrating physically-based inverse rendering, relighting, and deformation-aware manipulation, effectively resolving baked-illumination artifacts on glossy surfaces under dynamic motion through decoupled parameter learning and accelerated mesh ray tracing.

Background & Motivation¶

Gaussian Splatting (GS) has quickly established itself as a foundational representation in interactive virtual reality, digital asset authoring, and embodied simulations due to its explicit formulation and high rasterization throughput. To animate reconstructed static assets, contemporary deformable GS approaches typically adhere to a reconstruct-then-drive paradigm: a static Gaussian radiance field is first optimized from calibrated multi-view imagery, followed by particle-based physical simulation (e.g., MPM or PBD) or triangle-mesh deformation. However, this snapshot-based reconstruction suffers from an intrinsic physical limitation: environmental illumination and high-frequency specular reflections are irreversibly baked into the per-Gaussian base colors or low-order spherical harmonics. When the reconstructed object undergoes non-rigid deformation, bending, or exposure to unfamiliar lighting environments, the specular highlights drift rigidly with the surface geometry instead of adapting to dynamically rotated surface normals and incoming light, producing severe visual artifacts, especially on glossy or metallic surfaces.

The core tension lies in the structural conflict between rigorous inverse-rendering decomposition and efficient continuous geometric deformation. While NeRF-based inverse rendering methods accurately recover global illumination via ray marching, their computational burden makes dynamic interactive simulation infeasible. Conversely, existing physics-based Gaussian inverse rendering frameworks are almost strictly confined to static scenes; they either rely on split-sum approximations that degrade specular fidelity or employ prohibitively slow 2D Gaussian ray tracing (2DGRT), whose expensive BVH rebuilds and quadratic-surfel intersection tests drop dynamic frame rates to unworkable levels. Furthermore, static snapshot binding prevents 3D artists from performing post-reconstruction editing on physical material attributes such as roughness, metallic properties, and diffuse albedo.

To address these challenges, the authors argue that material attributes, geometry, and lighting must be disentangled within a unified Gaussian primitive, while separating the expensive ray-tracing proxy from the microscopic rasterization primitives during motion. The core idea is to build a fully decoupled 2D Gaussian inverse rendering and driving pipeline, where 2D surfels capture microfacet BRDF properties (albedo, roughness, metallic) via the full rendering equation during static pretraining, while a TSDF-extracted surface mesh acts as an accelerated ray-tracing proxy during dynamic deformation, driven by low-sample multiple importance sampling and cross-bilateral SVGF denoising to deliver physically faithful relighting at 24 FPS.

Method¶

Overall Architecture¶

The DR-GS framework operates through three interconnected stages: Static Geometry and Mesh-Based Gaussian Initialization, Material–Lighting Parameter Decoupling, and Dynamic Driving with Accelerated Relighting. In the first stage, initial 2D Gaussians are acquired and a triangular mesh is extracted via TSDF. To ensure geometric consistency and eliminate boundary tears, the 2D Gaussians are reparameterized onto the mesh faces with learnable normal offsets. Next, each Gaussian is augmented with physically-based material parameters (albedo, roughness, metallic) and rendered via deferred rasterization into screen-space G-buffers; the rendering equation is then solved across the upper hemisphere using importance sampling and ray tracing to decouple materials from direct and indirect illumination. Finally, in the dynamic stage, the object is driven either by Material Point Method (MPM) particle simulations via Generalized Moving Least Squares (GMLS) interpolation or by direct mesh cage deformation. Incident illumination is re-evaluated efficiently by swapping Gaussian ray tracing for mesh-based ray tracing, combined with Multiple Importance Sampling (MIS) and SVGF cross-bilateral denoising.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Calibrated Multi-View Images"] --> B["Mesh Extraction & Gaussian Reinitialization<br/>TSDF mesh extraction + face-attached 2D surfels"]
    B --> C["Parameter Decoupling via Rasterization<br/>Deferred G-buffer generation: position, normal, albedo, roughness, metallic"]
    C --> D["Full Rendering Equation Solution<br/>Environment direct lighting + 2DGRT visibility & indirect bounces"]
    D --> E["Dual-Mode Dynamic Deformation<br/>MPM particle cloud with GMLS interpolation / Mesh affine driving"]
    E --> F["Mesh-Assisted Dynamic Ray Tracing & MIS<br/>32-sample multiple importance sampling + SVGF edge-preserving denoiser"]
    F --> G["Physically Plausible Dynamic Relighted Renderings"]

Key Designs¶

1. Mesh-Constrained 2D Gaussian Reinitialization: Enforcing Geometric Regularity
Standard 3D Gaussians lack explicit surface definitions and well-defined normal vectors, frequently resulting in noisy outliers and non-manifold artifacts under extreme deformations. DR-GS adopts 2D Gaussian Splatting (2DGS) as its geometric primitive, where the cross-product of orthogonal tangent vectors analytically yields consistent surface normals $\mathbf{n} = \mathbf{t}_u \times \mathbf{t}_v$. To anchor the Gaussians to underlying mesh topology and facilitate downstream ray tracing, the pretrained Gaussians are projected onto triangular faces $f \in \mathcal{F}$ extracted via TSDF. Each face-attached Gaussian center $\mathbf{p}$ is parameterized through barycentric interpolation weights $\boldsymbol{\omega} = (\omega_a, \omega_b, \omega_c)$ alongside a learnable normal offset $d$: $$\mathbf{p} = (\omega_a \mathbf{v}_a + \omega_b \mathbf{v}_b + \omega_c \mathbf{v}_c) + d \cdot r \mathbf{n}$$ where $r$ represents the circumradius of the triangle formed by vertices $\{\mathbf{v}_a, \mathbf{v}_b, \mathbf{v}_c\}$. Optimizing the normal offset $d$ and individual Gaussian opacity $\alpha$ gives the explicit surfels the elastic flexibility to compensate for discretization errors and geometric inaccuracies in the extracted mesh, preventing spiky distortions and localized surface tears.

2. Physical Parameter Decoupling: Solving the Full Rendering Equation
To strip away baked illumination, DR-GS equips each 2D Gaussian with physical material attributes $\Phi_{\text{mat}} = \{\mathbf{a}, r, m\}$, designating diffuse albedo $\mathbf{a} \in [0, 1]^3$, roughness $r \in [0, 1]$, and metallic $m \in [0, 1]$. In deferred shading passes, depth-ordered Gaussian blending aggregates these properties into screen-space normal, depth, and material maps. Surface shading is governed by the complete rendering equation under Cook-Torrance microfacet BRDF: $$L_o(\boldsymbol{\omega}_o, \mathbf{x}) = \int_{\Omega} f(\boldsymbol{\omega}_o, \boldsymbol{\omega}_i, \mathbf{x}) L_i(\boldsymbol{\omega}_i, \mathbf{x}) (\boldsymbol{\omega}_i \cdot \mathbf{n}) d\boldsymbol{\omega}_i$$ Incident radiance $L_i$ is decomposed into distant environment illumination $L_{\text{dir}}$ modulated by geometric visibility $V(\boldsymbol{\omega}_i, \mathbf{x})$ and inter-reflection radiance $L_{\text{ind}}$, both computed via 2D Gaussian Ray Tracing (2DGRT). Computing this integral across sampled rays during static training enables the network to cleanly disentangle true surface reflectance from environmental lighting conditions.

3. Dual-Mode Dynamic Deformation: Particle-Driven and Mesh-Driven Control
To accommodate both complex soft-body physical simulations and artist-controlled rigging animations, DR-GS supports two complementary driving mechanisms. For particle-driven dynamics, the internal volume of the extracted mesh is seeded with a dense particle cloud and simulated using the Material Point Method (MPM). The resulting deformation gradient $\mathbf{F}_i$ and spatial displacement are transferred to each 2D Gaussian via Generalized Moving Least Squares (GMLS) interpolation. Performing polar decomposition $\mathbf{F}_i = \bar{\mathbf{R}}_i \bar{\mathbf{S}}_i$ isolates local rotation $\bar{\mathbf{R}}_i$ and scaling-shearing strain $\bar{\mathbf{S}}_i$, which are applied directly to the Gaussian's rotation matrix and tangential scaling parameters: $$\mathbf{p}'_i = \bar{\mathbf{p}}_i, \quad \mathbf{R}'_i = \bar{\mathbf{R}}_i \mathbf{R}_i, \quad \mathbf{s}'_i = \text{diag}(\bar{\boldsymbol{\Lambda}}_i) \mathbf{s}_i$$ For mesh-driven dynamics, affine transformations computed across deformed triangle vertices are mapped to face-bound Gaussians through barycentric coordinates, ensuring that surface normals and tangent planes adapt consistently to dynamic physical strain.

4. Mesh-Accelerated Ray Tracing with Multiple Importance Sampling (MIS)
During real-time dynamic manipulation, precomputed Gaussian radiance values are invalidated by altered surface orientations and novel light sources. Performing per-frame Gaussian ray tracing (2DGRT) over hundreds of thousands of moving primitives creates an acute computational bottleneck. DR-GS resolves this by observing that macroscopic light transport (shadows and visibility) is predominantly determined by gross geometry. Consequently, learned material parameters are stored on the TSDF mesh vertices, allowing hardware-accelerated OptiX mesh ray tracing to handle intersection lookups during dynamic sequences. To curb Monte Carlo variance under narrow specular lobes without excessive ray counts, DR-GS introduces Multiple Importance Sampling (MIS) using the balance heuristic, combining cosine-weighted distributions, GGX microfacet normal distributions, and environment-map luminance sampling. The per-pixel ray budget is reduced from 512 rays during static training to only 32 rays during dynamic driving (16 cosine, 8 GGX, 8 environment light). A depth- and normal-guided Spatiotemporal Variance-Guided Filter (SVGF) removes residual stochastic noise, achieving clean, interactive dynamic relighting at 24 FPS.

Loss & Training¶

During the parameter decoupling phase, the 2D Gaussian geometry, material parameters $\Phi_{\text{mat}}$, and environment lighting are optimized jointly using the multi-term loss: $$\mathcal{L} = \mathcal{L}_c + \lambda_1^{\text{pbr}}\mathcal{L}_1^{\text{pbr}} + \lambda_{\text{light}}\mathcal{L}_{\text{light}}$$ where $\mathcal{L}_c$ denotes the standard 3DGS RGB reconstruction loss combining $L_1$ and D-SSIM, $\mathcal{L}_1^{\text{pbr}}$ measures the absolute discrepancy between physically-based Monte Carlo shading and ground-truth pixel values, and $\mathcal{L}_{\text{light}}$ enforces a white-balance regularization on the recovered environment illumination to prevent scale ambiguity between lighting intensity and diffuse albedo. Models are pretrained following Ref-Gaussian and fine-tuned for 20,000 iterations.

Key Experimental Results¶

Main Results¶

The method was evaluated on the GlossySynthetic dataset across eight reflective scenes (angel, bell, cat, horse, luyu, potion, tbell, teapot). Table 1 reports static reconstruction quality against leading dynamic GS baselines, while Table 2 benchmarks relighting accuracy and runtime efficiency.

Table 1: Static Reconstruction Quality Comparison on GlossySynthetic (from paper Table 1)

Scene	Metric	PhysGaussian (P)	GSP (P)	SuGaR (M)	Mani-GS (M)	GaussianMesh (M)	Ours (P)	Ours (M)
angel	PSNR↑ / SSIM↑ / LPIPS↓	25.38 / 0.801 / 0.092	25.70 / 0.916 / 0.083	12.81 / 0.775 / 0.202	26.39 / 0.911 / 0.075	25.84 / 0.901 / 0.079	29.64 / 0.943 / 0.052	25.04 / 0.887 / 0.094
bell	PSNR↑ / SSIM↑ / LPIPS↓	24.31 / 0.904 / 0.122	29.26 / 0.940 / 0.090	12.40 / 0.740 / 0.240	24.58 / 0.901 / 0.114	24.02 / 0.890 / 0.118	31.94 / 0.962 / 0.056	28.12 / 0.917 / 0.103
cat	PSNR↑ / SSIM↑ / LPIPS↓	29.48 / 0.947 / 0.085	30.82 / 0.955 / 0.068	12.51 / 0.730 / 0.203	29.90 / 0.950 / 0.072	28.39 / 0.936 / 0.083	31.25 / 0.965 / 0.052	27.07 / 0.931 / 0.091
potion	PSNR↑ / SSIM↑ / LPIPS↓	28.71 / 0.923 / 0.132	28.41 / 0.932 / 0.113	14.68 / 0.613 / 0.254	27.83 / 0.922 / 0.110	27.53 / 0.918 / 0.112	30.55 / 0.941 / 0.095	26.75 / 0.887 / 0.145
teapot	PSNR↑ / SSIM↑ / LPIPS↓	21.14 / 0.877 / 0.122	22.67 / 0.898 / 0.108	13.62 / 0.744 / 0.194	21.18 / 0.878 / 0.106	20.64 / 0.871 / 0.109	26.72 / 0.945 / 0.062	22.93 / 0.882 / 0.107

Table 2: Relighting Performance and Computational Efficiency on GlossySynthetic (from paper Table 2)

Scene	Metric	GS-ROR2	Ref-Gaussian	IRGS	DR-GS (Ours)
angel	PSNR↑ / SSIM↑	20.81 / 0.878	21.39 / 0.900	20.58 / 0.860	24.15 / 0.905
bell	PSNR↑ / SSIM↑	24.49 / 0.927	22.90 / 0.919	20.98 / 0.878	25.81 / 0.931
cat	PSNR↑ / SSIM↑	26.28 / 0.942	20.54 / 0.912	22.43 / 0.888	27.34 / 0.940
potion	PSNR↑ / SSIM↑	25.67 / 0.918	20.06 / 0.868	22.92 / 0.866	27.35 / 0.923
teapot	PSNR↑ / SSIM↑	21.17 / 0.893	21.78 / 0.924	19.27 / 0.870	23.08 / 0.925
Training Time	Hours (total pipeline)	1.5 h	0.58 h	1.0 h	0.68 h
Rendering Speed	FPS (dynamic relighting)	122 FPS	208 FPS	0.5 FPS	24 FPS (interactive)

Because ground truth is unavailable for dynamic deformations, the authors conducted a blind perceptual user study with 30 participants across Physical Plausibility (PP), Temporal Coherence (TC), and Deformation Consistency (DC) on a 0-5 scale (paper Table 3). Under dynamic relighting conditions, DR-GS scored 4.62 (PP), 4.27 (TC), and 4.21 (DC), significantly outperforming GSP's ratings of 3.82 (PP), 3.56 (TC), and 3.65 (DC).

Ablation Study¶

The paper evaluates the individual contributions of the learnable reinitialization parameters and dynamic rendering acceleration components (paper Figures 12 and 13): 1. Ablation on Learnable Parameters (Figure 12): - Removing the learnable normal offset $d$ (w/o learnable $d$): Discretization artifacts from the initial mesh extraction bleed directly onto the surfels, resulting in prominent surface spikes and structural blurring. - Removing learnable opacity $\alpha$ (w/o learnable $\alpha$): Localized surface noise and crack-like boundary seams appear across complex curvatures. - Full Model: Preserves clean, razor-sharp specular highlights and smooth surface continuity. 2. Ablation on Acceleration Components (Figure 13): - Removing MIS (w/o MIS): Causes significant variance and salt-and-pepper noise in specular regions, reducing PSNR by 2–4 dB at equivalent sample counts. - Removing Mesh Acceleration (w/o mesh): Forces per-frame Gaussian ray tracing, driving rendering time per frame up to hundreds of milliseconds. - Removing the Denoiser (w/o denoiser): Requires more than an 8× increase in ray sampling to achieve comparable visual quality.

Key Findings¶

Decoupling is non-negotiable for specular fidelity: While PhysGaussian scores respectably on static scenes, its dynamic deformations fail because highlights slide across the mesh as if glued to the texture. Disentangling intrinsic material parameters is essential for realistic light-surface interactions.
Mesh proxies unlock real-time relighting: IRGS achieves high-fidelity static decomposition via 2DGRT but crawls at 0.5 FPS; switching the dynamic ray-tracing proxy to a TSDF mesh accelerates rendering to 24 FPS (a 48× speedup) without compromising inter-reflection or shadow quality.

Highlights & Insights¶

Dual Representation Synergy: Using 2D Gaussians for detailed deferred screen-space rasterization while using an underlying TSDF mesh for ray-traced geometric occlusion neatly sidesteps the intractable cost of Gaussian self-intersection tests.
Post-Reconstruction Asset Editing: Because albedo, roughness, and metallic properties are independently parameterized, users can directly edit physical parameters post-hoc—turning stone sculptures into polished gold or rusted bronze without retraining.

Limitations & Future Work¶

Throughput Gap Relative to Pure 3DGS: While 24 FPS is sufficient for interactive manipulation, solving the rendering equation and casting 32 rays per pixel is still significantly more demanding than vanilla 3DGS rasterization (>100 FPS).
Vulnerability to Severe Mesh Inversion: In mesh-driven mode, violent topological tears or degenerate self-intersecting triangles can compromise ray-mesh intersection tests, indicating a need for future work on adaptive mesh remeshing.

vs PhysGaussian [53]: PhysGaussian pioneered MPM-driven physical simulation for 3D Gaussians but baked lighting into colors; DR-GS preserves physical simulation capabilities while introducing rigorous PBR material decoupling.
vs GSP [7]: GSP supports both deformation and relighting via PBD, but suffers from hole-ridden surface reconstructions and lacks importance sampling, resulting in noisy specularities; DR-GS ensures smooth, continuous reflections via 2DGS normals and MIS.
vs IRGS [11]: IRGS implements full 2D Gaussian ray tracing for static scenes at a prohibitive 0.5 FPS; DR-GS introduces mesh-assisted ray tracing and SVGF filtering, accelerating dynamic relighting by ~48× to 24 FPS.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ [The first unified framework combining physically-based inverse rendering, relighting, and dual-mode dynamic deformation within 2D Gaussians.]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Thorough benchmarking across static reconstruction, relighting, user studies, parameter editing demonstrations, and component ablations.]
Writing Quality: ⭐⭐⭐⭐⭐ [Logically organized, with clear mathematical formulations and informative pipeline diagrams.]
Value: ⭐⭐⭐⭐⭐ [Provides a practical, physically sound paradigm for dynamic 3D asset generation, game development, and interactive simulation.]