Dynamic Inverse Rendering for Enhanced Material-Lighting Decomposition¶

Conference: ECCV2026
Paper: Official page · PDF
Project: https://razayunus.github.io/DIR
Area: Video Understanding / 3D Reconstruction and Inverse Rendering
Keywords: Dynamic inverse rendering, material-lighting decomposition, rigid pose tracking, SDF Gaussians, relighting

TL;DR¶

DIR turns rigid motion during hand-held capture into illumination constraints, using three stages of tracking, geometry refinement, and physical inverse rendering to recover relightable assets; on synthetic diffuse objects, hand-held capture with estimated poses achieves 40.33 dB albedo PSNR versus 32.68 dB for static multiview capture with ground-truth poses.

Background & Motivation¶

Reconstructing an object well enough to reproduce its photographs does not mean recovering its intrinsic color. Static multiview capture changes the viewing direction, but largely preserves the relationship between each surface normal and the environment. A dark region could therefore reflect low albedo or a shadow. An optimizer can put object color into the environment map or bake shading into the material while still fitting the training photographs. Replacing the illumination exposes this incorrect decomposition. Existing approaches introduce material or lighting priors, multiple capture environments, or additional light sources, at the cost of extra data, capture requirements, or more complicated illumination models.

This paper asks what happens when the environment stays fixed but the object rotates freely in a person's hands. The same surface point faces different environment directions, so an incorrect material-lighting combination that explains one frame may no longer explain the rest. The important difference is not simply having more video frames: the relationship between the surface and the light actually changes. A single-axis turntable provides some of this variation, but less than arbitrary-axis rotation. Moving only the camera primarily adds viewing constraints rather than incident-light constraints.

The challenge then shifts to tracking. Exploiting large rotations requires recovering both the shape and the poses of an unknown object, while incorrect normals can contaminate material estimation. The authors first obtain stable coarse tracking with a smooth representation, then introduce detailed Gaussians, and only afterward optimize physical materials and illumination. Core idea: use diverse surface-light interactions generated by rigid motion in one shared, stationary far-field environment to constrain a single material model, with coarse-to-fine reconstruction making those cross-frame constraints geometrically consistent.

Method¶

Overall Architecture¶

The input is a monocular RGB sequence of a rigid object moving in front of a stationary camera, together with object masks. Neither a pre-existing object CAD model nor per-frame poses are required. The outputs are canonical geometry, albedo and roughness, per-frame object poses, and an environment map shared over time. SAM3 supplies the masks for real captures; the hand itself is not explicitly modeled as an illumination occluder.

The pipeline proceeds through progressive pose and coarse geometry estimation, global SDF Gaussian refinement, and dynamic physical inverse rendering. The first two stages establish where the surface is and how it is oriented. The final stage requires shared materials to explain photographs across poses, instead of allowing each frame to choose an independent environment that absorbs its errors.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["RGB sequence and object masks"] --> B["Progressive Pose and<br/>Coarse Geometry Estimation"]
    B --> C["Global SDF Gaussian Refinement"]
    C --> D["Dynamic Physical Inverse Rendering"]
    E["Environment shared across frames"] --> D
    D --> F["Geometry, materials, poses,<br/>and a relightable asset"]

Key Designs¶

1. Progressive Pose and Coarse Geometry Estimation

Optimizing detailed geometry, all poses, and materials from the start lets incorrect geometry compensate for incorrect poses, potentially producing a solution that fits photographs but is physically wrong. Following the progressive strategy of FMOV, DIR initially represents the object with the neural signed distance field of NeuS. MLP smoothness and unit-sphere geometric initialization help keep the surface intact under sparse observations. A low number of samples per ray deliberately limits this stage to coarse geometry, preventing poses from following high-frequency noise. View-dependent color here is an early reconstruction tool, not the final disentangled albedo.

Frames are incorporated sequentially, with each pose initialized from the preceding frame and represented by a per-frame pose MLP. Centering the object with its mask in a virtual camera image removes two translation degrees of freedom during initialization. This is a tracking convenience rather than the final physically complete camera model. Geometry is reset after sufficiently large pose changes so that a degraded surface does not derail subsequent tracking. Learned matchers such as RoMa v2 and MASt3R provide neighboring-frame correspondences; the cross-frame matching loss helps constrain tracking on weakly textured and specular objects. Each step updates only the pose MLPs of the current frame and its matched frame, rather than immediately solving for the entire sequence jointly.

2. Global SDF Gaussian Refinement

Once coarse poses are available, DIR switches to detailed 3D Gaussians, but not ordinary 3DGS with freely optimized opacity. Each Gaussian additionally stores an SDF value that determines opacity, with a learnable global parameter controlling the mapping. Its smallest-scale axis defines the normal. Backfacing Gaussians are culled during rasterization to avoid mixing front and back surfaces, and depth is computed from the ray's 3D intersection with the Gaussian rather than simply its center. Together with normal, depth-variance, and disc-shape constraints, this stage is intended to produce reliable surface normals for the later BRDF, not merely sharper photographs.

Object poses transform Gaussian positions and rotations, carrying their normals with them. Spherical-harmonic appearance is evaluated by inversely rotating the query direction. This preserves a fixed environment coordinate system while allowing rendering gradients to reach the pose network. The virtual-camera estimates from the first stage are converted into full 6-DoF initializations using mesh-derived 3D-to-2D correspondences and RANSAC EPnP. A global MLP with per-pose embeddings then learns corrections while all frames are optimized jointly. FastGS multi-view-consistent densification and pruning concentrate Gaussians on useful surfaces and control the cost of subsequent ray tracing. Refinement therefore strengthens geometric constraints and pose consistency rather than simply increasing the Gaussian count.

3. Dynamic Physical Inverse Rendering

Each Gaussian now receives albedo and roughness parameters. The material model combines Lambertian diffuse reflection with GGX microfacet specular reflection, fixing normal-incidence reflectance to \(F_0=0.04\) for dielectric materials. Following SVG-IR, material properties and normals are stored at the four corners of a Gaussian plane and interpolated at the ray intersection, avoiding a completely uniform local surface within each Gaussian. Illumination remains a single shared far-field environment map. Optimization begins at low resolution and progressively upsamples the map bilinearly so that high-resolution lighting does not immediately become a container for reconstruction errors.

The motion-induced constraint can be expressed as follows. These are normalized expressions based on the prose, not literal transcriptions of equations damaged during text extraction: a canonical surface point keeps the same material over time, while its position and normal follow the rigid pose.

\[ x_t=R_t x+t_t,\qquad \hat n_t=R_t\hat n. \]

\[ L_o(x_t,\omega_o)=\int_{\Omega_t} f_t(\omega_i,\omega_o)\left[L_{\mathrm{direct}}(\omega_i)V(x_t,\omega_i)+L_{\mathrm{indirect}}(x_t,\omega_i)\right](\omega_i\cdot\hat n_t)_+\,d\omega_i. \]

Here \(f_t\) is the shared material's BRDF in the current pose, \(\Omega_t\) is the integration domain of directions, and the positive-part operator removes negative cosine values. The same direction in the shared environment produces different observed brightness as normals and visibility change. One material model and one environment must explain all of those observations. If the optimizer permanently darkens a region to explain a shadow in one frame, rotating that region back into the light creates a cross-frame error. This is the additional information supplied by motion relative to static multiview capture.

To prevent self-occlusion from being mistaken for low albedo, DIR uses 3DGRT to trace secondary rays from the surface. Thresholded accumulated opacity yields environment visibility, while traced color approximates indirect incident light from that direction. The implementation uses deferred shading: transformed Gaussians are first rasterized into depth, normal, and material maps, then physical rendering is evaluated at the recovered surface points. Optimization uses Fibonacci sphere sampling, accepting fixed-pattern bias for stability. Relighting switches to BRDF- and lighting-based importance sampling. Gaussian radiance learned under the original environment is no longer a valid indirect-light estimate after the environment changes, so relighting instead uses a split-sum approximation to compute one-bounce indirect radiance.

A Worked Example¶

Imagine rotating an object with a white patch around different axes in the same room. This is an illustration of the mechanism, not an additional experiment. The first stage uses masks and neighboring-frame matches to associate the patch with the same coarse surface across frames. The second recovers detailed normals and globally consistent poses. The third observes that the patch alternates between dark and bright while its canonical material should remain unchanged, so the changes must primarily be explained through the rotated lighting directions and self-occlusion.

With only single-axis rotation, some normals still encounter a limited range of illumination directions. Under uniform white illumination, rotation does not generate new directional lighting evidence at all. The benefit therefore comes from the interaction between motion and a nonuniform environment, not from video being inherently superior to images. When the environment map is finally replaced, the recovered materials stay fixed while direct light and approximate one-bounce illumination are recomputed, testing whether the material has actually been separated from the original shading.

Loss & Training¶

The first stage combines photometric reconstruction, the NeuS eikonal regularizer, masks, and cross-frame matching losses, prioritizing a stable surface and sequential tracking. The second combines photometric reconstruction, normal regularization, depth variance, a rank constraint encouraging disc-like Gaussians, SDF median and projection losses, and masks for global geometry and pose refinement.

The third stage primarily minimizes photometric error after physical rendering, with additional smoothness losses on environment and material maps. The method overview also states that geometry and poses are jointly optimized at this stage rather than completely frozen. The local cache contains only the main paper; loss weights, learning rates, iteration counts, and the environment-resolution schedule cannot be verified from it and are not supplied as definite hyperparameters here. The damaged SDF-to-opacity equation is likewise not reconstructed speculatively.

Key Experimental Results¶

Main Results¶

The synthetic dataset uses 10 objects from HOT3D assets, either keeping their original, generally more specular materials or replacing them with Lambertian materials while preserving albedo textures. The three capture configurations share a set of evaluation views. Synthetic hand-held sequences contain arbitrary-axis object rotations without explicitly rendering a hand. Albedo and relighting scales are normalized using ground-truth albedo, so these metrics do not establish recovery of absolute radiometric scale.

The following comparison comes from paper Table 3 and covers four hand-held sequences. PSNR is measured in dB and is higher-is-better; normal MAE is mean angular error and is lower-is-better. Asterisks identify static methods adapted to rotating objects, not results of their unmodified versions.

Method	Environment maps	Albedo PSNR	Relighting PSNR	Normal MAE
ReCap	10	21.91	25.74	8.06
ReCap	40	22.42	25.58	8.01
ReCap	100	22.31	25.20	8.08
SVG-IR*	1	33.40	32.22	3.69
IRGS*	1	40.26	32.99	0.56
DIR	1	41.03	39.65	0.51

Relative to IRGS*, DIR improves albedo and relighting PSNR by 0.77 and 6.66 dB, respectively. ReCap shares environment maps across groups of consecutive frames: excessive sharing produces an overly coarse approximation, whereas too little sharing introduces too many illumination parameters. Increasing the count from 10 to 100 maps does not resolve this mismatch.

Paper Table 2 also evaluates the rendering pipeline on the static TensoIR benchmark, checking that its strengths do not depend entirely on dynamic inputs.

Method	Albedo PSNR	Relighting PSNR	Normal MAE
SVG-IR	30.27	31.19	4.19
IRGS	33.41	30.63	3.99
DIR	33.96	32.17	4.19

DIR has the highest values for both PSNR metrics, but not the lowest normal error. The distinction matters: the evidence supports better decomposition and relighting, not universal superiority on geometry metrics.

Ablation Study¶

The following extraction from paper Table 1 compares capture configurations and pose sources. It is an ablation of acquisition conditions, not a network-component removal study. Static dome capture moves the camera over the upper hemisphere around a stationary object; turntable capture rotates the object around one axis; hand-held capture rotates it around arbitrary axes. The setting icons did not survive text extraction, so setting names are assigned from the prose and the original table's grouping order.

Material	Capture	Poses	Albedo PSNR	Relighting PSNR	Normal MAE
Original	Static dome	Ground truth	36.72	38.14	0.83
Original	Turntable	Estimated	34.10	33.02	2.45
Original	Turntable	Ground truth	36.15	36.01	1.20
Original	Hand-held	Estimated	40.37	38.54	1.56
Original	Hand-held	Ground truth	40.72	40.24	0.53
Diffuse	Static dome	Ground truth	32.68	34.85	1.59
Diffuse	Turntable	Estimated	32.97	33.04	1.83
Diffuse	Turntable	Ground truth	35.65	35.10	1.42
Diffuse	Hand-held	Estimated	40.33	40.37	0.75
Diffuse	Hand-held	Ground truth	40.55	41.29	0.58

Key Findings¶

For diffuse materials, estimated-pose hand-held capture improves albedo by 7.65 dB and relighting by 5.52 dB over ground-truth-pose static capture. Tracking errors do not erase the benefit of motion.
For original materials, estimated-pose hand-held capture has a normal error of 1.56 versus 0.83 for ground-truth-pose static capture, yet still achieves better albedo. More accurate normals are therefore not the sole explanation for the improvement.
Replacing estimated hand-held poses with ground truth increases relighting from 38.54 to 40.24 dB for original materials and from 40.37 to 41.29 dB for diffuse materials. Pose accuracy remains a meaningful bottleneck.
The roughness analysis in Figure 5 indicates that sharp specular highlights already constrain illumination, whereas diffuse objects depend more strongly on motion-induced directional evidence. Figure 6 shows that uniform white light supplies no additional motion-induced lighting information, while complex environments widen the gap between dynamic and static decomposition. Complete numerical curve data are unavailable in the cache, so no numerical table is invented for them.
Real experiments cover only 2 objects captured in static and hand-held configurations under the same environment. They provide qualitative evidence of cleaner albedo and more plausible relighting, not ground-truth material metrics.

Highlights & Insights¶

A capture trajectory can itself be a design for physical constraints. Rather than learning another material prior, the method changes object orientation to obtain observations that better distinguish the same unknown quantities.
Representation changes address different sources of error at different stages. Early NeuS smoothness prevents poses from chasing noise, while later SDF Gaussians provide the details and stable normals required for material estimation.
Relighting should not reuse indirect-light estimates tied to the training environment. Recomputing an approximate single bounce after illumination changes respects the underlying physical dependency better than retaining old radiance colors, a lesson applicable to other relightable Gaussian systems.

Limitations & Future Work¶

The authors explicitly identify hand occlusions, near-field illumination, and non-rigid objects as future directions. Modeling object self-occlusion does not imply that hand shadows, reflected light from hands, or position-dependent nearby lighting are solved.
A distant, time-invariant environment is a central assumption. Uniform illumination and restricted rotation offer limited extra constraints. Motion improves conditioning, but the paper does not establish unique material-lighting identifiability in general.
Fixing \(F_0=0.04\) targets dielectric materials and does not directly cover arbitrary metals, transparent materials, or complex scattering. This scope restriction follows from the material parameterization.
The 10 synthetic objects, 4 dynamic benchmark sequences, and 2 real objects establish feasibility rather than large-scale real-world generalization. Scale normalization also leaves an evidence gap for absolute illumination intensity recovery.
The local main paper does not provide verifiable component-removal ablations, the complete pose benchmark, or runtime costs. It is therefore not possible to assign separate numerical contributions to the SDF, matchers, or ray tracing, or to claim real-time performance. Useful follow-up experiments would independently control rotation coverage, pose noise, and hand occlusion.

FMOV supplies the virtual-camera and progressive tracking ideas. DIR uses stronger matchers and adds global SDF Gaussian refinement and physical material decomposition, extending the target from geometric reconstruction to relightable assets.
DiscretizedSDF, SVG-IR, FastGS, and 3DGRT support SDF Gaussian geometry, locally varying materials, efficient density control, and ray tracing, respectively. The contribution is primarily their integration into reliable dynamic inverse rendering and the validation of motion-induced decomposition gains, not the invention of every underlying component.
ReCap and IRGS address cross-environment illumination modeling and inter-reflection, respectively. DIR's key constraint is object motion with shared illumination; it is not simply independent lighting estimation for every video frame.
Transferable direction: active capture could prioritize poses that expose already observed surfaces to new lighting directions, rather than maximizing camera coverage alone. This is a research direction suggested by the mechanism, not an automatic capture algorithm already validated by the authors.

Rating¶

Novelty: 4/5. The empirical argument for rigid motion as a decomposition constraint is clear, while the underlying representations and rendering components largely build on existing methods.
Experimental Thoroughness: 4/5. Capture, pose, material, and illumination analyses complement static and dynamic baselines, but real-world coverage is small and the available cache lacks fine-grained component ablations.
Writing Quality: 4/5. The three-stage rationale and experimental logic are clear; reproduction-level details require the supplement. Damaged equations in the cache do not imply errors in the original paper.
Value: 4/5. The method offers useful guidance for accessible hand-held asset capture and relighting, subject to rigid-motion, far-field-lighting, and material assumptions.