DANTE-W: Diffuse Albedo Neural Texturing in the Wild¶
Conference: ECCV 2026
Paper: ECCV Official
Project Page: https://dante-wild.github.io
Area: 3D Vision
Keywords: Neural Texturing, Diffuse Albedo Estimation, Intrinsic Decomposition, Multi-Resolution Hash Encoding, Relighting
TL;DR¶
Addressing the baked-in shading and shadowing artifacts in classical multi-view mesh texturing of unstructured in-the-wild scenes, Dante-w lifts view-space diffusion priors onto a UV-space neural texture and leverages the frequency discrepancy between albedo and irradiance through physically principled neural rendering to recover high-fidelity, 3D-consistent diffuse albedo maps ready for standard graphics engines.
Background & Motivation¶
Reconstructing high-quality textured 3D assets of real-world, large-scale outdoor scenes is foundational for applications across virtual reality, visual effects, and digital twins. However, classical image-based 3D reconstruction and texturing pipelines (such as Metashape Pro and Waechter et al.) operate primarily by blending multi-view observations directly or optimizing Markov Random Field (MRF) patch selection over surface meshes. Because these paradigms lack intrinsic material reasoning, they cannot disentangle intrinsic surface reflectance (e.g., diffuse albedo) from incident illumination effects (e.g., diffuse irradiance, cast shadows, and self-occlusions). Consequently, intense noon-day highlights and sharp cast shadows are permanently baked into the resulting UV texture maps, causing severe visual artifacts when relighting the asset under novel environments.
While physics-based neural inverse rendering approaches attempt to jointly estimate geometry, materials, and environment lighting using neural implicit fields, they consistently struggle to scale to unconstrained in-the-wild captures due to the ill-posed lighting-material ambiguity and oversimplified shading formulations. On the other end of the spectrum, generative intrinsic decomposition models (such as RGB↔X and Cosmos-DiffusionRenderer) harness rich 2D diffusion priors pretrained on web-scale synthetic data. Nevertheless, these models operate on single images or short video clips, suffering from 3D multi-view inconsistency, random generative hallucinations, and high-frequency textural blur caused by VAE latent compression. Crucially, their generative nature makes them incompatible with standard graphics pipelines and physically based ray tracers.
To overcome these barriers, the authors introduce Dante-w, a hybrid neural texturing framework that seamlessly bridges generative 2D diffusion priors with explicit 3D mesh parameterization. The key observation is rooted in physical optics: for in-the-wild scenes, diffuse irradiance fields vary smoothly over space (accurately approximated by low-order spherical harmonics), whereas intrinsic diffuse albedo contains rich, high-frequency textural variations. Core idea: lift view-space diffusion albedo priors into a high-resolution 2D multi-resolution hash neural texture, impose an explicit frequency-band bottleneck by bounding the maximal resolution of the 3D irradiance field, and employ physically principled neural rendering on raw images to direct 3D-consistent high-frequency details solely into the albedo representation, which is ultimately baked into standard UV texture maps.
Method¶
Overall Architecture¶
Dante-w takes as input an unstructured collection of multi-view photographs and a reconstructed, UV-parameterized triangle mesh from off-the-shelf Structure-from-Motion (SfM) and Multi-View Stereo (MVS) pipelines, producing a high-fidelity diffuse albedo UV map ready for standard rendering engines such as Blender Cycles. The workflow proceeds through three main stages: first, view-space diffuse albedo guidance buffers are generated using pretrained intrinsic diffusion models; second, decoupled neural representations are initialized—a high-resolution 2D multi-resolution hash encoding on the UV texture manifold for diffuse albedo, and a low-resolution 3D volumetric hash encoding paired with per-image exposure embeddings for scene irradiance; third, the representations are jointly optimized via diffusion prior distillation and tone-mapped physical neural rendering loss, allowing fine-grained textural details to flow exclusively into the albedo field.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Input Multi-View Images + Parameterized Mesh"] --> B["View-Space Diffusion Albedo Prior Distillation<br/>Multi-view prior projection and consistency aggregation"]
B --> C["Frequency-Decoupled Neural Representation Construction<br/>High-resolution 2D texture hash vs low-resolution 3D lighting hash"]
C --> D["Physically Principled Neural Rendering Reprojection<br/>Irradiance and albedo multi-band tone-mapped reconstruction"]
D --> E["Standard UV Albedo Texture Map Baking<br/>Seamless export to Blender Cycles for free-view relighting"]Key Designs¶
1. View-Space Diffusion Albedo Prior Distillation: suppressing view flickering and anchoring low-frequency albedo To resolve the severely ill-posed decomposition problem in unconstrained multi-view captures, this design introduces state-of-the-art intrinsic decomposition diffusion models (such as Cosmos-DiffusionRenderer or RGB↔X) as robust low-frequency guidance. Input images \(\{I_k\}\) are processed offline to generate view-space albedo buffers \(\{A'_k\}\). During training, ray rasterization identifies the surface intersection and its UV coordinates \(\hat{x}\), queries the neural albedo prediction \(a_d\), and minimizes the screen-space distillation loss: $\(\mathcal{L}_a = \sum_{k} \sum_{r} \| a_d(\hat{x}_r) - A'_k(r) \|_1\)$ Because 2D diffusion models produce view-dependent fluctuations and lack spatial geometric awareness, aggregating their predictions onto a unified 2D UV neural manifold acts as a multi-view consensus filter, effectively eliminating single-view hallucinations while anchoring the base reflectance.
2. Frequency-Decoupled Neural Representation Construction: resolving lighting-material ambiguity via spatial frequency discrepancy Direct distillation of diffusion outputs results in blurry textures due to latent VAE compression and generative smoothing. Dante-w resolves this issue by exploiting the fundamental frequency discrepancy between smooth irradiance and sharp albedo. Diffuse albedo is parameterized on the 2D surface UV domain via a multi-resolution hash encoding \(\psi^{2D}_a(\hat{x})\) with a lightweight MLP \(F_{\Theta_a}\), where the maximum grid resolution \(V_{L_a}^{(a)}\) is set to an ultra-dense level (e.g., 8192 for large outdoor scenes). In contrast, global diffuse irradiance \(s_d\), which accounts for multi-bounce light transport and shadowing across 3D space, is modeled as a 3D volumetric hash field \(\psi^{3D}_s(x)\) concatenated with a per-image latent embedding \(e_k\) and decoded by an MLP \(F_{\Theta_s}\). Crucially, the maximum grid resolution of irradiance is strictly capped at a low frequency \(V_{L_s}^{(s)}\) such that \(V_{L_a}^{(a)} \gg V_{L_s}^{(s)}\). This structural bottleneck prevents the lighting representation from absorbing fine-grained spatial variations.
3. Physically Principled Neural Rendering Reprojection: guiding high-frequency image details into the albedo field Following the Lambertian diffuse formation model \(c = a_d \odot s_d\), predicted surface radiance is composed from the decoupled fields. To handle real-world radiometric differences across cameras and viewpoints, per-image learnable scaling and bias scalars \((w_k, b_k)\) and a fixed sRGB tone mapping function \(\gamma(\cdot)\) are introduced. The model is trained to reconstruct the original raw camera observations \(I_k(r)\): $\(\mathcal{L}_c = \sum_{k} \sum_{r} \| \gamma(w_k \cdot (a_d \odot s_d) + b_k) - I_k(r) \|_1\)$ Because the irradiance representation is strictly frequency-constrained and cannot accommodate high-frequency variations (such as fine brick cracks, carved inscriptions, or painted motifs), back-propagated gradients from \(\mathcal{L}_c\) naturally funnel high-frequency details exclusively into the expressive 2D neural texture hash grid, recovering crisp, photographic-level surface textures free from baked-in lighting.
Loss & Training¶
The entire framework is optimized end-to-end using a joint objective combining the neural rendering loss and the diffusion albedo regularization: $\(\mathcal{L} = \mathcal{L}_c + \lambda \mathcal{L}_a\)$ The balancing hyper-parameter is set to \(\lambda = 1.0\) across all synthetic and real-world experiments. To maximize throughput, ray-surface intersection buffers (3D coordinates, UV coordinates, and foreground hit masks) are pre-rasterized and cached. During optimization, mini-batches of foreground rays are sampled randomly across all views. Built upon the high-performance tiny-cuda-nn library, Dante-w completes optimization in only 5 to 8 minutes per scene containing thousands of 4K images on a single NVIDIA RTX 3090 GPU, requiring 0.8GB of VRAM for synthetic objects and 3.3GB for large-scale outdoor scenes. Once optimized, the continuous neural field is queried over the UV domain to bake standard 4K/8K diffuse albedo texture files.
Key Experimental Results¶
Main Results¶
Quantitative evaluations are conducted on the newly curated GigaLit benchmark, comprising 10 high-resolution synthetic objects with 4K textures rendered under 5 diverse training and 5 held-out novel lighting conditions, alongside 6 real-world outdoor scenes. Dante-w is systematically compared with classical texturing approaches, intrinsic diffusion models, and inverse rendering baselines on both Diffuse Albedo recovery and novel-view Relighting.
| Method / Model | Albedo PSNR (dB)↑ | Albedo SSIM↑ | Albedo LPIPS↓ | Relighting PSNR (dB)↑ | Relighting SSIM↑ | Relighting LPIPS↓ |
|---|---|---|---|---|---|---|
| Waechter et al. (Classical texturing) | 21.43 | 0.889 | 0.103 | 23.78 | 0.915 | 0.092 |
| Bi et al. (Patch-based optimization) | 21.48 | 0.885 | 0.111 | 23.81 | 0.902 | 0.097 |
| Metashape Pro (Commercial MVS texturing) | 21.50 | 0.892 | 0.103 | 23.83 | 0.917 | 0.089 |
| LightSwitch (Diffusion material estimation) | 16.89 | 0.807 | 0.185 | 17.79 | 0.808 | 0.189 |
| RGB↔X (Diffusion intrinsic model) | 21.65 | 0.860 | 0.125 | 20.30 | 0.849 | 0.134 |
| Cosmos-DiffusionRenderer (DR) | 23.36 | 0.879 | 0.120 | 21.64 | 0.873 | 0.130 |
| Dante-w [Ours + RGB↔X] (Full) | 25.50 | 0.911 | 0.093 | 27.45 | 0.933 | 0.076 |
| Dante-w [Ours + DR] (Full) | 25.99 | 0.921 | 0.091 | 27.77 | 0.938 | 0.071 |
Ablation Study¶
The ablation study validates the efficacy of the Physically Principled Neural Rendering (PR) framework and the frequency-band discrepancy design on the GigaLit synthetic benchmark:
| Configuration | Albedo PSNR (dB)↑ | Albedo SSIM↑ | Albedo LPIPS↓ | Relighting PSNR (dB)↑ | Relighting LPIPS↓ | Note |
|---|---|---|---|---|---|---|
| Ours + RGB↔X (w/o PR, prior lifting only) | 25.32 | 0.890 | 0.120 | 27.27 | 0.099 | Textures lack fine-grained edge details |
| Ours + RGB↔X (Full, with PR) | 25.50 | 0.911 | 0.093 | 27.45 | 0.076 | Sharp high-frequency texture recovery |
| Ours + DR (w/o PR, prior lifting only) | 25.69 | 0.907 | 0.115 | 27.45 | 0.092 | Over-smoothed albedo due to VAE compression |
| Ours + DR (Full, with PR) | 25.99 | 0.921 | 0.091 | 27.77 | 0.071 | Superior fidelity in albedo and relighting |
| Balanced frequency grid (\(V_{L_a}^{(a)} \approx V_{L_s}^{(s)}\)) | Qualitative drop | Qualitative drop | Qualitative drop | Qualitative drop | Qualitative drop | High-frequency textures leak into irradiance field |
Key Findings¶
- Physical Neural Rendering (PR) dramatically cuts perceptual error: Integrating PR reduces albedo LPIPS by approximately 24% (from 0.115-0.120 to 0.091-0.093) and relighting LPIPS by over 20%, proving that re-rendering original camera views effectively restores textural details lost during generative VAE compression.
- Substantial margins over diffusion-based renderers: Dante-w outperforms standalone Cosmos-DiffusionRenderer by +2.63 dB in albedo PSNR and +6.13 dB in relighting PSNR, while reducing relighting LPIPS by ~45%, confirming the superiority of 3D-consistent neural textures over per-frame 2D generative outputs.
- Unprecedented training efficiency: In processing complex outdoor monuments (e.g., The Pavilion of Prince Teng with over 2,500 4K photographs), the full optimization converges in 5 to 8 minutes on a single consumer GPU—over 7× faster than commercial texturing in Metashape Pro—while consuming only 0.8GB to 3.3GB of GPU memory.
Highlights & Insights¶
- Frequency-band bottleneck as an elegant physics regularizer: Rather than imposing heuristic spatial smoothness penalties, Dante-w translates the physical smoothness of diffuse lighting into asymmetric grid resolution limits in multi-resolution hash encodings, directing fine detail gradients cleanly into the albedo channel.
- Seamless bridge between 2D foundation priors and standard 3D graphics: Instead of relying on un-editable neural volumetric fields or bulky radiance fields, the method outputs explicit, standard UV texture maps that can be directly baked and rendered in Blender, Unreal Engine, or Unity.
- Uncovering capture flaws in legacy benchmarks: The paper critically demonstrates that existing real-world datasets like Stanford-ORB contain severe measurement noise and texture degradation in their ground-truth albedo annotations, justifying the introduction of the high-fidelity GigaLit benchmark.
Limitations & Future Work¶
- Lambertian reflectance assumption: The current formulation specifically models diffuse albedo and diffuse irradiance, assuming approximately Lambertian surfaces, which does not account for complex non-Lambertian specularities, roughness, or metallic properties.
- Dependency on upstream reconstruction quality: The framework relies on an initial geometry mesh and UV parameterization provided by upstream SfM/MVS tools; geometric topological flaws or severe UV distortions directly impact texturing quality.
- Future directions: The authors outline extending the frequency-decoupled neural texturing paradigm to explicit point-based radiance representations (e.g., 3D Gaussian Splatting) and incorporating multi-parameter PBR BRDF decomposition for specular materials.
Related Work & Insights¶
- vs Metashape Pro / Waechter et al. (Classical mesh texturing): Classical methods perform image blending or patch stitching directly in pixel space, permanently baking illumination effects and cast shadows into the texture; Dante-w explicitly extracts illumination-free diffuse albedo via generative distillation and neural inverse rendering.
- vs RGB↔X / DiffusionRenderer (Generative intrinsic decomposition): Pretrained diffusion models suffer from multi-view flickering and loss of micro-textures due to latent compression; Dante-w lifts these predictions onto a shared 3D-consistent neural manifold and recovers photographic details via raw image reconstruction.
- vs LightSwitch / TensoIR (Neural inverse rendering): Optimization-only neural inverse rendering methods suffer from severe local minima and material ambiguities in complex outdoor environments; Dante-w leverages web-scale diffusion priors as low-frequency anchors to ensure robust convergence.
Rating¶
- Novelty: ⭐⭐⭐⭐⭐ Elegant formulation that resolves the ill-posed inverse rendering problem by pairing neural hash resolution limits with diffusion distillation.
- Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous validation across real-scale in-the-wild captures, high-resolution 4K synthetic benchmarks, and legacy datasets with comprehensive ablations.
- Writing Quality: ⭐⭐⭐⭐⭐ Clear, structured narrative with precise mathematical formulations and intuitive architectural illustrations.
- Value: ⭐⭐⭐⭐⭐ Seamlessly integrates with industrial graphics pipelines while accelerating texturing by over 7× with negligible memory footprint.