[ECCV 2026][3D Vision][Inverse Rendering] This paper proposes regularizing physically disentangled inverse rendering with similarity kernels derived from diffusion-predicted G-buffers (albedo, roughness, metallic, and normals): rather than fitting these predictions as target values, it uses them to define material similarity between surface regions, applying joint bilateral filtering to encourage consistent optimized parameters within the same material region, thereby suppressing artifacts caused by lighting baked into materials and significantly outperforming SOTA methods such as Neural-PBIR and MaterialFusion on Stanford-ORB.

Tags: ECCV 2026 · 3D Vision · Inverse Rendering · Material Regularization · Diffusion Model · Physically Based Rendering · Joint Bilateral Filtering

Diffusion-Based Material Regularization for Physics-Based Inverse Rendering¶

Conference: ECCV 2026
arXiv: 2606.31065
Code: None
Area: 3D Vision
Keywords: Inverse Rendering, Material Regularization, Diffusion Model, Physically Based Rendering, Joint Bilateral Filtering

TL;DR¶

This paper proposes regularizing physically disentangled inverse rendering with similarity kernels derived from diffusion-predicted G-buffers (albedo, roughness, metallic, and normals): rather than fitting these predictions as target values, it uses them to define material similarity between surface regions, applying joint bilateral filtering to encourage consistent optimized parameters within the same material region, thereby suppressing artifacts caused by lighting baked into materials and significantly outperforming SOTA methods such as Neural-PBIR and MaterialFusion on Stanford-ORB.

Background & Motivation¶

Reconstructing an object's geometry, materials, and lighting from multi-view images (inverse rendering) is a central problem in graphics and vision. The reconstructed PBR assets can be used directly in standard rendering pipelines for relighting and editing. Physics-based inverse rendering (PBIR) provides an accurate image formation model through differentiable rendering, but the problem itself is highly underconstrained: the same input image can be explained by multiple (material, lighting) combinations. The optimizer can easily "bake" lighting effects into material maps—for example, treating shadows as dark albedo textures—resulting in extremely poor generalization to novel views or lighting.

Meanwhile, data-driven diffusion models such as DiffusionRenderer and RGB-X can predict visually plausible G-buffers from a single image or video. However, these predictions do not satisfy the rendering equation, and using them directly for physically based rendering can introduce color shifts and inaccurate materials.

Existing methods typically treat diffusion predictions as supervision targets (for example, MaterialFusion uses score distillation, while VideoMat uses a global scale-invariant loss). This creates tension between photometric fitting and prior constraints: indiscriminately pulling results toward predicted values sacrifices fidelity to the input images and cannot accommodate the separate corrections needed by different material regions in a scene. The paper's central insight is that diffusion predictions need not serve as target values. Their most valuable information is the implicit grouping of material regions: predictions vary smoothly within the same material region and change abruptly across different regions. This grouping can constrain inverse rendering optimization to maintain material consistency within a region without forcing the optimized results to equal the predictions. Core Idea: use diffusion predictions as the guidance kernel for joint bilateral filtering to construct an "implicit material clustering" regularizer, benefiting from data-driven priors without sacrificing the accuracy of physically based rendering.

Method¶

Overall Architecture¶

The input consists of N multi-view images of a static object under unknown lighting. The goal is to reconstruct standard PBR assets for the object (a triangle mesh + spatially varying Disney BRDF parameters + an environment map), such that relighting under new illumination matches the ground truth. The method executes three stages sequentially:

Preprocessing: After calibrating the camera parameters, DiffusionRenderer (image mode) predicts a G-buffer $\mathbf{G}_i = [\mathbf{A}_i, \mathbf{R}_i, \mathbf{M}_i, \mathbf{N}_i]$ (albedo, roughness, metallic, and normals) for each view.
Neural surface reconstruction: A voxel-grid SDF + neural volume rendering reconstructs the object's shape, with an additional normal supervision loss (the Huber distance between predicted normals and diffusion-predicted normals) to reduce concavity artifacts on smooth surfaces. Marching Cubes then extracts the initial mesh.
Physics-based inverse rendering (PBIR): Starting from the initial mesh, the method jointly optimizes shape, spatially varying materials (the base color / roughness / metallic parameters of the Disney BRDF), and the environment map. The optimization objective is $\mathcal{L} = \mathcal{L}_{\text{img}} + \lambda_{\text{mat}}\mathcal{L}_{\text{mat}}$, where $\mathcal{L}_{\text{img}}$ is a relative MSE rendering loss (adapted to HDR inputs), and $\mathcal{L}_{\text{mat}}$ is the paper's core implicit material clustering regularizer.

The PBIR stage parameterizes the spatially varying BRDF and environment map using Dictionary Fields (neural fields with shared basis functions). Rendering uses the path replay backpropagation integrator in Mitsuba 3 (maximum path depth 3, supporting one-bounce indirect illumination).

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-view images<br/>+ camera parameters"] --> B["Preprocessing<br/>DiffusionRenderer predicts<br/>per-view G-buffers"]
    B --> C["Neural surface reconstruction<br/>Voxel SDF + normal supervision<br/>Marching Cubes mesh extraction"]
    C --> D["Physics-based inverse rendering（PBIR）<br/>Mitsuba 3 differentiable rendering<br/>Joint shape+material+lighting optimization"]
    B -->|G-buffer guidance kernel| D
    D --> E["Output PBR assets<br/>Mesh + BRDF + environment lighting"]

Key Designs¶

1. Implicit material clustering regularization: using diffusion predictions as similarity kernels rather than target values

Optimizing spatially varying materials through physics-based inverse rendering under sparse views and unknown lighting is severely underconstrained: each surface point is observed from only a limited number of views, so the optimizer can easily bake lighting effects into materials. DiffusionRenderer can predict visually plausible G-buffers whose values vary smoothly within the same material region. Rather than treating predicted values as targets, this paper exploits the property that "predictions are similar within regions of the same material" to construct an implicit material clustering constraint.

Specifically, for each view, let the diffusion-predicted material G-buffer be $\mathbf{g} = [\mathbf{A}, \mathbf{R}, \mathbf{M}]$. The similarity kernel between pixels p and q is defined as:

\[k_{p,q}(\mathbf{g}) = \exp\left(-\frac{\lVert \mathbf{g}_p - \mathbf{g}_q \rVert_2^2}{2\sigma_g^2}\right)\]

Let $\hat{\mathbf{g}}$ denote the differentiably rendered G-buffer of the current optimized parameters. Joint bilateral filtering (JBF) smooths $\hat{\mathbf{g}}$ using the diffusion-predicted kernel for guidance:

\[\mathrm{JBF}(\hat{\mathbf{g}}; \mathbf{g})_p = \frac{\sum_q k_{p,q}(\mathbf{g}) \hat{\mathbf{g}}_q}{\sum_q k_{p,q}(\mathbf{g})}\]

The intuition behind JBF is to compute a weighted average of the optimized materials across all pixels within a "material-similar" region defined by diffusion predictions. If the optimizer bakes a shadow into part of the region, the material there deviates from the regional mean, and the regularizer pulls it back. JBF smooths only within similar regions defined by the kernel, not across material boundaries, so regions with different materials can each have their own independent deviations.

The final regularization loss is the L1 distance between the rendered G-buffer and its JBF-filtered version:

\[\mathcal{L}_{\text{mat}} = \left\lVert \hat{\mathbf{g}} - \mathrm{JBF}(\hat{\mathbf{g}}; \mathbf{g}) \right\rVert_1\]

Importantly, a single kernel based on $\mathbf{g} = [\mathbf{A}, \mathbf{R}, \mathbf{M}]$, which concatenates all channels, regularizes every channel (rather than constructing a separate kernel for each channel). This allows information from other channels to prevent over-regularization when one predicted channel is overly smooth. Ablation experiments also demonstrate that the concatenated kernel outperforms separate per-channel kernels. JBF is implemented efficiently and differentiably using a permutohedral lattice. In essence, this design resembles reflectance sharing: diffusion predictions implicitly partition the surface into "same-material regions," sharing observations within each region to reduce degrees of freedom without forcing materials to be consistent across regions.

2. Scale-independent albedo transformation: decoupling albedo and lighting scales

Applying material regularization directly to albedo introduces a subtle bias: the optimizer can lower $\mathcal{L}_{\text{mat}}$ by reducing albedo while increasing lighting brightness to compensate in the rendered results, producing abnormally intense light sources. To address this, the paper introduces a scale-independent albedo transformation:

\[\psi(\hat{\mathbf{A}}) = \operatorname{sg}(\hat{\mathbf{A}}) \odot \log([\hat{\mathbf{A}}]_\epsilon)\]

Here, $\operatorname{sg}(\cdot)$ denotes stop-gradient, and $[\hat{\mathbf{A}}]_\epsilon = \max(\hat{\mathbf{A}}, \epsilon)$ ensures numerical stability. In log space, multiplicative scaling within a region becomes an additive offset that cancels out in the JBF difference $\hat{\mathbf{g}} - \mathrm{JBF}(\hat{\mathbf{g}}; \mathbf{g})$, making the regularizer insensitive to overall albedo scaling. Meanwhile, the log function scales the gradient by $1/\hat{\mathbf{A}}$. Multiplying by the stop-gradient factor $\operatorname{sg}(\hat{\mathbf{A}})$ exactly cancels this scaling and appropriately reduces gradients in dark regions. When computing $\mathcal{L}_{\text{mat}}$, use $\psi(\hat{\mathbf{A}})$ in $\hat{\mathbf{g}}$ in place of its $\hat{\mathbf{A}}$ component.

3. Normal-supervised neural surface reconstruction

When reconstructing shape with a voxel SDF + neural volume rendering in the second stage, the diffusion-predicted normal maps are directly introduced as supervision:

\[\mathcal{L}_{\text{shape}} = \sum_{i=1}^{N} \sum_{p} H_\delta\left(1 - \hat{\mathbf{N}}_{i,p}^\top \mathbf{N}_{i,p}\right)\]

Here, $H_\delta$ is a Huber penalty ($\delta = 0.03$, with linear decay beyond an angular difference of approximately $15^\circ$), improving robustness to noise in diffusion-predicted normals. This loss enhances geometric surface detail, particularly by reducing concavity artifacts on smooth surfaces—artifacts that specular highlights would severely amplify during subsequent relighting.

Loss & Training¶

The total loss is $\mathcal{L} = \mathcal{L}_{\text{img}} + \lambda_{\text{mat}}\mathcal{L}_{\text{mat}}$, with $\lambda_{\text{mat}} = 0.1$. $\mathcal{L}_{\text{img}}$ is relative MSE:

\[\mathcal{L}_{\text{img}} = \sum_{i=1}^{N} \left\lVert \frac{\hat{I}_i - I_i}{\operatorname{sg}(\hat{I}_i) + \epsilon} \right\rVert_2^2, \quad \epsilon = 10^{-2}\]

The relative loss prevents a small number of excessively bright pixels in HDR inputs from dominating the gradients. The PBIR stage starts with a learning rate of $3 \times 10^{-2}$, cosine-annealed to $10^{-3}$, and runs for 900 iterations, randomly sampling 6 views per iteration. Rendering uses 256 spp for the primal pass and 64 spp for reverse-mode automatic differentiation. BRDF parameters and the environment map are parameterized with Dictionary Fields (2D configuration). The environment map uses exponential activation; BRDF outputs have no activation but are clamped to [0,1], with an $L_1$ penalty on out-of-bounds values (weight $10^{-2}$). On a single RTX 5090, the SDF stage takes approximately 10 minutes, DiffusionRenderer preprocessing approximately 15 minutes, and the PBIR stage approximately 7 minutes per scene.

Key Experimental Results¶

Main Results¶

Evaluation covers three datasets: Stanford-ORB (42 scenes, real captures + relighting ground truth), Synthetic4Relight (4 scenes), and DTC-Synthetic (7 scenes, including highly smooth surfaces and cast shadows). Comparison methods are Neural-PBIR (pure analysis-by-synthesis inverse rendering) and MaterialFusion (a score-distillation diffusion prior). Metrics include PSNR-H/L (high/low dynamic range), SSIM, and LPIPS.

Dataset	Method	Relighting PSNR-H	Relighting PSNR-L	Relighting SSIM	Relighting LPIPS	Novel-view PSNR-H
Stanford-ORB	Neural-PBIR	26.07	33.39	0.980	0.023	28.90
Stanford-ORB	MaterialFusion	23.52	31.18	0.968	0.037	26.17
Stanford-ORB	Ours	27.22	34.98	0.981	0.021	29.58

Dataset	Method	Relighting PSNR	Relighting SSIM	Aligned Albedo PSNR	Albedo PSNR	Roughness MSE
Synthetic4Relight	Neural-PBIR	27.83	0.974	25.72	22.67	0.016
Synthetic4Relight	MaterialFusion	20.20	0.944	24.28	15.65	0.015
Synthetic4Relight	Ours	32.02	0.975	27.83	27.04	0.013
DTC-Synthetic	Neural-PBIR	39.18	0.9928	-	-	-
DTC-Synthetic	MaterialFusion	28.63	0.9746	-	-	-
DTC-Synthetic	Ours	43.21	0.9961	-	-	-

On Stanford-ORB, the method improves relighting PSNR-H from Neural-PBIR's 26.07 to 27.22 (+1.15 dB) and reduces shape error (Shape metric) from 0.42 to 0.30. Although MaterialFusion can remove baked-in shadows, it severely underfits, yielding an albedo PSNR of only 15.65 (vs Ours 27.04). Qualitatively, this method completely removes baked-in artifacts in scenes with cast shadows from strong directional lighting (the side of the red block in Block_RedBlue and around the cup handle in cup_scene006) and recovers more accurate reflections on glossy metallic surfaces (pitcher_scene001 and TeaPot_EmeraldGoldTop).

Ablation Study¶

Config	Stanford-ORB PSNR-H	PSNR-L	SSIM	Note
Full model (Ours)	27.22	34.98	0.981	full model
w/o reg. (no material regularization)	26.11	33.78	0.978	Removing regularization causes a 1.11/1.20 dB drop
d-s corr. (diffuse-specular autocorrelation regularization)	26.37	34.10	0.979	Non-data-driven autocorrelation regularization, misled by baked-in albedo
scale inv. (global scale-invariant loss)	26.38	34.21	0.980	Global adjustment cannot handle per-region material deviations
Diffusion-BP (direct back-projection of diffusion predictions)	26.06	33.09	0.976	Diffusion predictions themselves do not satisfy the rendering equation; worst performance

The ablations show that removing material regularization reduces relighting PSNR on Synthetic4Relight from 32.02 to 28.94 (-3.08 dB), and on DTC-Synthetic from 43.21 to 40.23 (-2.98 dB). Ablating the scale-independent albedo transformation shows that without it, the optimizer lowers albedo and compensates with an abnormally bright environment map, departing from the true lighting. For normal supervision, removing $\mathcal{L}_{\text{shape}}$ introduces concavity artifacts on smooth surfaces, causing severe distortions in specular regions. Supplementary experiments also compare concatenated and per-channel kernels: the concatenated kernel outperforms the per-channel alternative in both albedo estimation (27.15 vs 26.46 PSNR) and roughness (0.013 vs 0.024 MSE).

Key Findings¶

Material clustering regularization is the largest contributor to performance; removing it reduces Synthetic4Relight relighting performance by 3.08 dB. Its central value is not "how accurate the diffusion predictions are," but the property that "diffusion predictions are sufficiently smooth within regions of the same material."
Using diffusion predictions directly as initialization without regularization (w/o reg.) quickly reintroduces baked-in artifacts. This indicates that the initialization is far from the convergence point and optimization easily becomes trapped in local minima; the regularizer helps escape these local minima by constraining the solution space.
Replacing DiffusionRenderer with RGB-X as the upstream model still leaves the proposed regularization ahead of direct back-projection and a global scale-invariant loss (PSNR-H: 26.58 vs 25.78/26.02), demonstrating that the main benefit comes from the regularizer design rather than a particular upstream diffusion model.
Transferring the regularizer to IRGS (an inverse rendering pipeline based on 2D Gaussian Splatting) in place of its original smoothness term yields, across the four Synthetic4Relight scenes, an average gain of +0.80 dB PSNR, showing that the regularizer is not tied to a particular PBIR pipeline.

Highlights & Insights¶

Demoting diffusion predictions from "target values" to "similarity kernels" is an elegant conceptual shift: No regression or distillation is performed; only the local smoothness of predictions within regions of the same material is used. This gives data-driven priors and physically based rendering constraints distinct responsibilities: the diffusion model determines "where things should be the same," while PBIR determines "what the actual values should be." It fundamentally avoids the dilemma of sacrificing physical accuracy to move closer to predicted values.
JBF is an elegant engineering choice for inverse rendering regularization: Joint bilateral filtering has a long history in image processing, but using it for cross-pixel material smoothing within a differentiable rendering pipeline, with diffusion predictions defining the kernel, is new in inverse rendering. Its efficient, differentiable permutohedral-lattice implementation allows JBF to be embedded in every SGD iteration without significantly increasing computational overhead.
The scale-independent log transformation + stop-gradient trick transfers directly: Any regression-plus-regularization setting with scale ambiguity (such as brightness assumptions in decomposition problems) can draw on the design $\psi(x) = \operatorname{sg}(x) \odot \log(x)$. Smoothing in log space turns multiplicative scaling into additive offsets that cancel out, while the stop-gradient multiplier corrects gradient scaling. This trick also applies to decomposition tasks beyond inverse rendering.
The regularizer's transferability validates its generality: Successful transfer to IRGS (a different representation and renderer) indicates that this regularizer fundamentally does not depend on Mitsuba, a mesh representation, or Dictionary Fields. It requires only "a differentiably rendered G-buffer $\hat{\mathbf{g}}$ and an externally predicted guidance G-buffer $\mathbf{g}$."

Limitations & Future Work¶

The authors acknowledge that when DiffusionRenderer fails to predict rare appearances outside its training distribution and its predictions are insufficiently smooth within regions of the same material, the regularizer can overconstrain optimization and blur materials. Diffusion models also have limited resolution, leaving high-frequency details insufficiently constrained; higher-resolution diffusion models could directly alleviate this issue.
The similarity kernel currently uses simple channel concatenation + a Gaussian kernel. As diffusion models evolve, more sophisticated nonlinear kernel combinations (such as learned metrics or attention-based kernels) may work better. Although ablations establish the advantage of concatenated over per-channel kernels, a richer kernel design space remains unexplored.
Residual artifacts remain near shadow boundaries, which the authors attribute to insufficiently accurate geometry reconstruction. Future work could extract shadow-boundary information from the regularization signal to feed back into geometry optimization.
The current method assumes a single static object. Extending it to multi-object or dynamic scenes would require additional instance segmentation and temporal consistency mechanisms. Moreover, although the PBIR stage is fast at 7 minutes per scene (RTX 5090), the 10-minute SDF stage and 15-minute DiffusionRenderer preprocessing bring the end-to-end runtime to approximately 32 minutes—still some distance from real-time interactive reconstruction.
The experiments do not evaluate in-the-wild data captured with real mobile phones (such as the scenes of Alzayer et al.), leaving robustness to camera noise, motion blur, and strong lighting changes without systematic evaluation.

vs Neural-PBIR: Both use mesh-based inverse rendering pipelines, but Neural-PBIR relies entirely on analysis-by-synthesis optimization without learned priors. Under sparse views, severe material-lighting ambiguity causes baked-in artifacts. This paper adds diffusion-guided implicit material clustering regularization to that framework, improving Stanford-ORB relighting PSNR-H by 1.15 dB.
vs MaterialFusion: Both use diffusion priors to assist inverse rendering, but MaterialFusion uses score distillation (SDS) to sample from diffusion predictions as a target distribution. Without per-region correction, it produces excessive smoothing and color shifts. This paper's JBF regularizer allows each material region to deviate independently from diffusion predictions, substantially improving albedo and roughness reconstruction accuracy.
vs VideoMat / IntrinsicAnything: These methods use a global scale-invariant loss to guide materials, which cannot eliminate local baked-in shadows (such as at cast-shadow boundaries) and tends to underfit local material details (such as metallic foil lettering). This paper's per-pixel JBF regularizer relies on neighborhood similarity rather than global statistics, naturally supporting differentiated treatment of individual regions.
vs reflectance sharing / basis BRDF methods: Classical methods constrain material complexity through hard clustering or explicit segmentation, but accurate segmentation is itself difficult to obtain. This paper uses diffusion predictions to implicitly define "soft" material similarity, avoiding the model-selection problems (such as the number of bases) and boundary sensitivity of explicit segmentation.

Rating¶

Novelty: Four stars. Demoting diffusion predictions from target values to similarity kernels is an elegant conceptual shift. Although applying JBF to inverse rendering is not entirely new, the specific design (concatenated kernels + scale-independent transformation) offers substantive innovation.
Experimental Thoroughness: Four and a half stars. Three datasets + multiple baselines + transfer across upstream models (RGB-X) + transfer across pipelines (IRGS) provide very solid validation. The ablations cover regularization, kernel design, scale transformation, normal supervision, and other dimensions.
Writing Quality: Four stars. The motivation is clearly developed, the method balances formulas with intuition, and the figures are abundant and well targeted. The appendix provides extensive intrinsic G-buffer visualizations and supplementary ablations.
Value: Four stars. The paper introduces a transferable, general regularization design paradigm for inverse rendering, independent of any particular pipeline or upstream model, with the potential to become a standard component in subsequent work. In practice, it can substantially reduce baked-in artifacts, directly benefiting downstream applications that require relightable PBR assets (AR/VR, e-commerce, and film production).