Under One Sun: Multi-Object Generative Perception of Materials and Illumination¶
Conference: ECCV 2026
Paper: ECCV Official
Code: https://vision.ist.i.kyoto-u.ac.jp/research/onesun/
Area: 3D Vision
Keywords: Inverse Rendering, Generative Perception, Radiometric Disentanglement, Diffuse Texture, Illumination Estimation
TL;DR¶
MultiGP addresses the ill-posed radiometric disentanglement problem in single-image inverse rendering by leveraging the physical consensus of multiple objects sharing the same scene illumination, using a cascaded diffusion architecture with Coordinated Scheduling and Axial Attention to jointly sample diffuse textures, uniform reflectances, and high-frequency illumination.
Background & Motivation¶
Surface radiance observation is jointly dictated by surface geometry, the bidirectional reflectance distribution function (BRDF), and incident ambient lighting. However, recovering individual radiometric constituents (texture, reflectance, and illumination) from a single image is fundamentally ambiguous and ill-posed. Under the signal processing framework of inverse rendering, surface reflection acts as an angular convolution of incident illumination with a BRDF kernel; different surface materials attenuate high angular-frequency illumination details via low-pass filtering, leading to an irreversible loss of spectral information and severe material-lighting entanglement in single-object observations.
Conventional deterministic optimization and regression methods enforce statistical smoothness priors or suffer from regression to the mean, yielding blurry low-frequency lighting and failing to capture the intrinsic multi-modal ambiguity of plausible physical explanations. Meanwhile, recent generative perception models either hallucinate lighting environment maps without physically grounded material estimation, or remain confined to textureless single objects (e.g., DRM) while ignoring complex global light transport. A single object offers only a narrow range of surface normals and a single material frequency filter, inherently lacking the capability to serve as a comprehensive illumination probe.
The core idea is to exploit the physical consensus that diverse objects in a scene share the exact same illumination, treating them as complementary spatio-spectral probes within an end-to-end cascaded diffusion framework to jointly disentangle texture, reflectance, and global lighting.
Method¶
Overall Architecture¶
MultiGP approximates the joint posterior via a cascaded factorization: \(q(\mathbf{T}, \mathbf{R}, L \mid I, \mathbf{S}) \approx q_\phi(\mathbf{T} \mid I, \mathbf{S}) q_\theta(\mathbf{R}, L \mid I, \mathbf{S}, \mathbf{T})\). Given a single image \(I\) with \(M\) objects of known shapes \(\mathbf{S}=\{\mathbf{n}(x)_m\}_{m=1}^M\), the pipeline first disentangles diffuse textures \(\mathbf{T}\) from object appearances in image space using a latent diffusion model \(q_\phi\). The resulting texture-free appearances are projected onto shape-invariant raw reflectance maps over a Gaussian sphere. A multi-object diffusion model \(q_\theta\) then simultaneously recovers individual Disney BRDF reflectances \(\mathbf{R}\) and shared high-frequency mirror reflectance illumination \(L\), coordinated by Coordinated Scheduling and cross-object Axial Attention. Finally, a ControlNet conditioned on Mitsuba 3 physics-based rendering residuals refines the textures for strict observation consistency.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Image + Known Shape Normals"] --> B["Stage 1: Texture Extraction Prior q_phi<br/>Latent diffusion isolates diffuse textures and texture-free appearance"]
B --> C["Reflectance Map Mapping & GI Mitigation<br/>Partial normal projection to Gaussian sphere and completion"]
C --> D["Stage 2: Multi-Object Diffusion q_theta with Coordinated Scheduling<br/>Linear interpolation converging to shared mirror illumination"]
D --> E["Cross-Object Axial Attention Fusion<br/>Aligning normal directions to share complementary frequency bands & FOV"]
E --> F["Stage 3: ControlNet Texture Refinement via Renderer Residuals<br/>Mitsuba 3 physical re-rendering guides observation consistency"]
F --> G["Output: Individual Textures T + Reflectances R + Shared Illumination L"]
Key Designs¶
1. Cascaded Image-Domain and Angular-Domain Factorization: Decoupling Spatial Textures from Angular Material and Lighting Jointly optimizing high-frequency spatial texture patterns and angular surface reflection in a single step leads to severe texture artifacts leaking into estimated lighting. MultiGP decomposes the problem into two decoupled representation domains: a latent diffusion model \(q_\phi\) conditioned on appearance and shape latents isolates spatially-varying diffuse textures \(\mathbf{T}=\{\rho_d(x)_m\}_{m=1}^M\) and intermediate texture-free radiance appearances \(L_r^{(m)}(x)\); these appearances are then mapped onto geometry-free spherical reflectance maps, allowing the subsequent multi-object diffusion model \(q_\theta\) to focus purely on spatially-uniform Disney BRDF parameters \(\Psi^{(m)}=\{\text{metallic } \gamma, \text{roughness } r, \text{specular } \rho_s\}\) and environment illumination without interference from spatial albedo patterns.
2. Multi-Object Coordinated Scheduling: Synchronizing Stochastic Reverse Diffusion Toward a Common Environment Map Under reverse diffusion, each object begins with distinct estimated material properties; independent denoising chains would diverge into contradictory illumination hypotheses. MultiGP introduces Coordinated Scheduling, linearly transitioning each object's reflectance from its estimated initial state \(\hat{\Psi}^{(m,K)}\) to a pure mirror reflectance \(\Psi_0=\{1, 0, 1\}\) over \(K\) shared denoising steps: $$ \hat{\Psi}^{(m,k)} = \frac{k}{K}\hat{\Psi}^{(m,K)} + \left(1 - \frac{k}{K}\right)\Psi_0 $$ The total diffusion step count \(K\) is adaptively determined by the average Euclidean distance of the input materials to the ideal mirror state: $$ K = \left\lfloor \frac{K_{\max}}{M}\sum_{m=1}^M \left\lVert \frac{\hat{\Psi}^{(m,K)} - \Psi_0}{\sqrt{3}} \right\rVert \right\rfloor $$ At \(k=0\), every object's reflectance map stochastically converges to the exact mirror reflectance map \(L_r^{(0)}(\mathbf{n}) = \hat{L}_i(\mathbf{n})\) representing the true scene lighting, avoiding the infinite step requirement of prior single-object formulations.
3. Multi-Object Axial Attention: Cross-Object Spatio-Spectral Complementarity Sharing Single objects exhibit limited surface normal coverage and rough materials act as aggressive low-pass filters that erase high-frequency lighting information. MultiGP introduces Axial Attention operating across the \(M\) reflectance maps at matching surface normal coordinates. Highly specular objects inject high-frequency angular boundaries and directional source cues into diffuse objects, while wide-angle surfaces complete missing normal orientations for partially occluded geometry; local spatial attention is combined to provide robustness against minor surface normal misalignments.
4. Renderer Residual ControlNet Texture Refinement: Physics-Likelihood Guided Consistency Enforcement To eliminate non-physical generative hallucinations, sampled components \((\mathbf{T}, \mathbf{R}, L)\) are re-rendered through the Mitsuba 3 physically-based path tracer to produce synthesized radiance \(\hat{L}_r\). The image-space residual \(c_f = \mathcal{E}(I - \hat{L}_r)\) is injected into a ControlNet architecture that refines the initial diffuse textures \(\mathbf{T}\), enforcing rigorous photometric fidelity against global light transport while preserving generative diversity.
Loss & Training¶
The multi-object diffusion network \(q_\theta\) is trained under a variational lower bound (ELBO) on the rendering equation, using an illumination residual objective: $$ \mathcal{L}i = \mathbb{E} \right) \right\rVert_2^2 \right] $$ combined with a material prediction loss } \left[ \left\lVert \mu'_{\theta_i,\theta_r}\left(L_r^{(m,k)}, k\right) - \left( L_r\left(\Psi^{(m,k-1)}, L_r^{(0)}\right) - L_r^{(m,k)\(\mathcal{L}_r\) as \(\mathcal{L} = \lambda_i \mathcal{L}_i + \lambda_r \mathcal{L}_r\). The texture extraction prior and ControlNet optimize standard latent diffusion epsilon-prediction objectives. Training is conducted end-to-end on multi-object scenes synthesized with path-traced interreflections and cast shadows from Adobe 3D Assets and Poly Haven HDRIs.
Key Experimental Results¶
Main Results¶
On synthetic test sets (Adobe 3D Assets with MV-Adapter textures and held-out HDRIs), evaluating illumination (128×256) and reflectance across 10 samples (reporting the mean of the top 3) shows that MultiGP decisively outperforms state-of-the-art baselines:
| Method | Illumination logRMSE ↓ | Illumination PSNR ↑ | Illumination SSIM ↑ | Illumination LPIPS ↓ | Reflectance logRMSE ↓ |
|---|---|---|---|---|---|
| DPI (ACM TOG 2023) | 1.64 | 11.96 | 0.37 | 0.56 | 2.22 |
| DRM (CVPR 2024) | 1.48 | 12.62 | 0.34 | 0.61 | 1.85 |
| DiffusionLight (CVPR 2024) | 1.60 | 11.37 | 0.38 | 0.63 | - |
| MultiGP (single-object) | 1.44 | 12.67 | 0.38 | 0.58 | 2.05 |
| MultiGP (Full Multi-Object) | 1.28 | 13.54 | 0.42 | 0.56 | 1.81 |
For diffuse texture estimation (512×512, known geometry):
| Method | Texture RMSE ↓ | Texture PSNR ↑ | Texture SSIM ↑ | Texture LPIPS ↓ |
|---|---|---|---|---|
| DPI | 0.150 | 24.86 | 0.92 | 0.084 |
| DiffusionRenderer | 0.110 | 27.37 | 0.96 | 0.051 |
| RGB↔X (HDR) | 0.110 | 27.59 | 0.96 | 0.054 |
| MultiGP | 0.081 | 29.99 | 0.97 | 0.031 |
Ablation Study¶
Ablation of key architectural components on synthetic test data:
| Config | Illum logRMSE ↓ | Illum PSNR ↑ | Reflectance logRMSE ↓ | Texture PSNR ↑ | Note |
|---|---|---|---|---|---|
| Full model | 1.28 | 13.54 | 1.81 | 29.99 | best performance across all metrics |
| w/o Coordinated Scheduling | 1.29 | 13.50 | 1.87 | - | lack of synchronized mirror boundary degrades reflectance |
| w/o Axial Attention | 1.37 | 12.97 | 1.82 | - | disabling spatio-spectral sharing causes major illum degradation |
| w/o Texture Refinement | 1.29 | 13.49 | 1.87 | 29.32 | absence of renderer feedback drops texture PSNR by 0.67 dB |
| Global Illumination off | 1.33 | 13.08 | 1.81 | 29.35 | synthetic training without GI hurts real-world handling |
Ambiguity-aware evaluation in PCA space over order-32 Spherical Harmonics (1089 coefficients, 99% variance) confirms the probabilistic superiority: - Heterogeneous reflectances: Joint MultiGP achieves a multivariate normal log-likelihood of 171.9 (Mahalanobis distance 16.98), significantly beating the sharpest single input A (169.2 / 18.42) and diffuse input C (-243.6 / 28.91). - Heterogeneous masks: Joint MultiGP achieves a log-likelihood of 61.5 (Mahalanobis distance 24.18), drastically outperforming the largest visible area object A (-56.8 / 27.60).
Key Findings¶
- Axial Attention is the most critical driver of illumination accuracy; its removal spikes illumination logRMSE from 1.28 to 1.37, demonstrating the vital role of cross-object frequency compensation.
- High runtime efficiency: on a single NVIDIA A100 GPU for a 1024×1024 multi-object input, MultiGP runs in 25.3 seconds, offering an approximate 50× speedup over optimization-based baselines like DPI (1167.7s) and DiffusionLight (1280.1s).
Highlights & Insights¶
- Multiple Objects as Multi-Band Physical Probes: Framing distinct objects as concurrent spatial and frequency filters of the same illumination environment elegantly breaks the fundamental ambiguity barrier of single-image inverse rendering.
- Bounded Stochastic Convergence via Coordinated Scheduling: Linearly steering heterogeneous BRDF states toward an invariant mirror target enables principled multi-stream diffusion convergence within a finite, adaptive step budget.
- Ambiguity-Aware Spherical Harmonics Evaluation Metric: Introducing log-likelihood and Mahalanobis distances over order-32 SH PCA distributions provides a much sounder benchmark for evaluating stochastic inverse rendering than fragile point distances.
Limitations & Future Work¶
- Dependency on Known 3D Geometry: The current framework relies on pre-scanned meshes and surface normals; noisy geometry estimates or registration errors degrade reflectance map projection.
- Distant Illumination Assumption: The formulation assumes infinite directional illumination; complex indoor scenes with strong near-field point lights introduce spatially varying lighting discrepancies.
- Homogeneous Specular Reflectance: Objects are assumed to have spatially uniform specular BRDF parameters; extending this to fully spatially-varying specular properties remains an open problem.
Related Work & Insights¶
- vs DRM (CVPR 2024): DRM introduced diffusion on reflectance maps but was restricted to single textureless objects and required infinite steps; MultiGP generalizes to multi-object textured scenes with finite-step Coordinated Scheduling.
- vs DPI (ACM TOG 2023): DPI relies on slow per-scene differentiable inverse rendering optimization taking ~20 minutes; MultiGP uses feed-forward cascaded diffusion sampling in 25 seconds with higher accuracy.
- vs DiffusionLight (CVPR 2024): DiffusionLight hallucinates reflective chrome spheres without explicit physical BRDF decomposition; MultiGP achieves physically interpretable, joint estimation of textures, materials, and lighting.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Elegant physical consensus formulation for multi-object radiometric disentanglement.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive validation across synthetic benchmarks, real-world datasets (Stanford-ORB, nLMVS-real, self-captured), and novel ambiguity-aware SH metrics.
- Writing Quality: ⭐⭐⭐⭐⭐ Rigorous mathematical formulation, clear narrative, and highly transparent experimental analysis.
- Value: ⭐⭐⭐⭐⭐ Provides an impactful paradigm shift for generative inverse rendering, neural relighting, and embodied visual perception.