Monocular Avatar Reconstruction via Cascaded Diffusion Priors and UV-Space Differentiable Shading¶

Conference: ECCV 2026
arXiv: 2606.28144
Code: https://marcus-avatar.github.io (Project page, code is promised to be released after acceptance)
Area: 3D Vision
Keywords: 3D Avatar Reconstruction, PBR Material Estimation, Diffusion Priors, UV-space Differentiable Rendering, Cascaded LoRA

TL;DR¶

Using a unified pretrained diffusion backbone to pipeline three sub-tasks: "texture completion \(\rightarrow\) illumination homogenization \(\rightarrow\) material decomposition", with each sub-task attaching only a single LoRA, combined with a UV-space differentiable BRDF rendering loss, high-fidelity and relightable 3D facial PBR avatars at 4K resolution can be reconstructed from a single in-the-wild image using fewer than 100 real 3D scans.

Background & Motivation¶

Reconstructing relightable, high-fidelity 3D facial avatars from a single in-the-wild photo is a classically ill-posed problem. While the industry gold standard, the Light Stage system, yields the highest-quality geometry and materials, such equipment is prohibitively expensive and relies on complex studio environments, rendering it inaccessible for widespread use. Consequently, academia has turned to monocular reconstruction. Traditional 3DMMs (3D Morphable Models) represent faces within a compact, linear subspace, which is robust but inherently fails to capture high-frequency details such as wrinkles and pores. Subsequent neural texture completion methods (e.g., UV-GAN, GANFit, UV-IDM, FreeUV) can hallucinate invisible regions but "bake" illumination into the textures, making the reconstructed assets incompatible with relighting in standard PBR pipelines. To achieve true decoupling of illumination and material, generative networks (GANs or diffusion models) must be employed to predict non-linear material maps such as albedo, normals, and roughness.

However, this path faces a dilemma between data efficiency and generalization. On one hand, high-quality PBR training data is extremely scarce: synthetic data lacks realism, while real Light Stage scans are costly and difficult to scale. On the other hand, the methods themselves have bottlenecks: GANs are prone to mode collapse and over-smoothing (e.g., MoSAR suffers from a waxy appearance and pale skin tones), whereas fine-tuning diffusion models on small datasets (e.g., UltraAvatar) often erases the model's valuable prior knowledge, leading to generalization failure on in-the-wild images. More critically, most frameworks rely on differentiable rasterization combined with 2D screen-space supervision. This optimization paradigm is highly sensitive to occlusions such as hair and glasses in the input image, often baking these occlusions and artifacts into the reconstructed textures.

The Key Insight of this work is: Rather than collecting custom large-scale datasets or training a fragile generator from scratch, it is better to leverage the strong priors of large-scale pretrained diffusion models. Modern rendering engines (such as Blender) can be utilized to "amplify" fewer than 100 high-quality scans into massive synthetic paired datasets to adapt the diffusion model, while directly imposing physical constraints on the UV-space materials to bypass the sensitivity of screen-space supervision to occlusions. Core Idea: Centered around a shared, pretrained diffusion backbone, cascaded LoRAs sequentially adapt it into a texture completer, an illumination homogenizer, and a material estimator. In the UV space, a Cross-Intrinsic Attention mechanism collaboratively generates various PBR maps, and a differentiable GGX rendering loss enforces physical consistency. This achieves high-fidelity and generalizable relightable avatar reconstruction under extremely scarce real-scan data constraints.

Method¶

Overall Architecture¶

Given a single in-the-wild image, the pipeline first estimates the 3D facial geometry and camera parameters, projecting the image back to the UV space to obtain an "incomplete" texture map. Then, a frozen pretrained diffusion backbone (LongCat-Image-Edit DiT), outfitted with different LoRAs, sequentially performs three tasks: completing the occluded/missing textures, flattening baked-in illumination to a uniform lighting baseline (illumination homogenization), and executing multi-branch material estimation on this clean texture to predict five PBR maps (albedo, normal, roughness, specular, and displacement) at once. The predicted material maps are finally upsampled to 4K via a super-resolution network. At the core of the training phase is a data-generation pipeline: the geometry of nearly 100,000 in-the-wild faces from FFHQ / CelebAMask-HQ is reconstructed to compute visibility masks. Afterward, PBR materials from fewer than 100 professional scans are randomly mapped onto these geometries and rendered + baked in Blender under 2,041 HDRI environment lights, assembling paired supervisions of "illuminated textures / homogenized textures / incomplete textures".

The entire process is a clear multi-stage serial pipeline, illustrated in the flowchart below:

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["In-the-wild Image"] --> B["MSSF Geometry Reconstruction<br/>ConvNeXt V2 + DINOv3<br/>→ Geometry + Incomplete Texture"]
    B --> C["Texture Completion LoRA<br/>flow-matching to fill occluded gaps"]
    C --> D["Illumination Homogenization LoRA<br/>flatten baked-in illumination"]
    D --> E["Cross-Intrinsic Attention<br/>Three-branch LoRA joint PBR estimation"]
    E -->|re-render decoded clean latents| F["UV-Space Differentiable GGX Rendering Loss<br/>constrain physical consistency"]
    E --> G["4K Super-Resolution → Relightable Rendering"]

Key Designs¶

1. Cascaded LoRA Reusing a Single Diffusion Backbone: Turning Three Sub-tasks into Three "Adapters" of the Same Model

Limitations of Prior Work: Texture completion, de-lighting, and material estimation are traditionally modeled with three separate and disconnected networks. Each network must be trained independently and fed with separate data, which is data-expensive and compromises stylistic consistency. Ours: This work proposes to share a single frozen pretrained diffusion Transformer (a modern image-editing DiT) among all three tasks. The three tasks differ only in terms of "conditional inputs + supervision targets + attached LoRA", with the backbone parameters entirely frozen and only the LoRAs optimized. Each LoRA (rank 32) comprises approximately 91M parameters, representing only 0.7% of the backbone, and is injected into the projection layers of the attention and feed-forward blocks. Both completion and homogenization are formulated as conditional flow-matching diffusion: given noise level \(\sigma\in(0,1)\) and noisy latent \(z_\sigma=(1-\sigma)z_0+\sigma\epsilon\), the model learns to predict the velocity field \((\epsilon-z_0)\). The completion task conditions on the incomplete texture \(T_{\text{inc}}\) hoping to predict the complete texture; the homogenization task conditions on the illuminated texture \(T_{\text{env}}\) with the target set to \(T_{\text{hom}}\) under uniform lighting. This sharing scheme allows the expensive generative prior to be reused across all three tasks; consequently, the small dataset only needs to learn "how to guide the prior toward the specific sub-task" rather than learning a generator from scratch, allowing the pipeline to succeed with fewer than 100 scans.

\[\mathcal{L}_{\text{FM}}=\mathbb{E}_{\sigma,\epsilon}\left[\left\|f(z_\sigma,\,c,\,\sigma)-(\epsilon-z_0)\right\|_2^2\right]\]

2. Illumination Homogenization: Flattening Illumination Before Material Decomposition

Even when texture completion yields a full UV map, illumination remains scene-dependent—each face carries its own set of highlights and shadows. Attempting material estimation directly on such illuminated textures presents an exceedingly difficult, one-to-many inverse rendering problem: the network must simultaneously hallucinate materials while implicitly decomposing unknown, complex lighting from a single texture, yielding an immense solution space that leads to optimization failure. Therefore, this work inserts an explicit illumination homogenization stage, utilizing the same diffusion backbone to map the "illuminated texture" into a canonical, normalized lighting domain (supervised by \(T_{\text{hom}}\) baked under white ambient light). Ablations show that this step is vital for "stable training": removing it causes the material estimation to degrade into a degenerate solution, where high-contrast shadows are baked into the albedo and misconstrued by normal/displacement maps as high-frequency geometric noise, yielding physically implausible results even after 40k training steps. It serves as a normalization step that significantly narrows the search space, transforming the subsequent decomposition from "ill-posed" to "well-posed".

3. Cross-Intrinsic Attention: Aligning Different Material Maps Rather Than Predicting Independently

Training independent LoRAs for albedo, normals, and roughness leads to modality-specific overfitting: individual branches tend to misinterpret high-frequency local signals (e.g., skin textures, stubble) as geometric structures, causing pores on the normal map to be reconstructed as sharp geometric spikes, and smooth crow's feet to disintegrate into fractured scratches. This work allows three material branches (albedo, normals, and reflectance containing roughness/specular/displacement) to share the backbone while attaching separate LoRAs, stacking the three modalities along the batch dimension during training. Inside each Transformer block, standard self-attention is replaced with Cross-Intrinsic Attention. While each modality calculates its own query, keys and values are concatenated from the features of all material branches: \(K=[k_{\text{alb}},k_{\text{nrm}},k_{\text{rsd}}]\) and \(V=[v_{\text{alb}},v_{\text{nrm}},v_{\text{rsd}}]\). This enables each branch to directly "observe" complementary geometric and reflectance cues from the other branches. This design preserves the modality-specific customization of the LoRAs while forcing spatial alignment and coherent structural details via cross-modality information exchange, suppressing hallucinations such as surface pigmentation masquerading as geometry.

4. UV-Space Differentiable GGX Rendering Loss: Binding the Rendering Equation Directly to Materials, Bypassing Screen-Space Supervision

While joint diffusion guarantees spatial alignment, it does not ensure that the decomposition is physically meaningful—the network might predict visually plausible but physically incorrect materials. To combat this, a UV-space differentiable BRDF rendering loss is introduced to explicitly couple material prediction with the imaging process. During training, the clean latent variable is first reconstructed from the noisy state \(z_\sigma\) and the predicted velocity field as \(\hat z_0=z_\sigma-\sigma\,\hat v_\theta\), and then decoded into the predicted albedo, normal, and reflectance maps. The core design is to decouple texture decoding from "individual geometry": by fixing a template face, its world-coordinate position map, geometric normal map, and geometric tangent map are pre-computed in Blender to serve as geometric priors. This allows lighting to be accurately computed for 3D light transport directly within the 2D UV domain without relying on rasterization. Shading is performed via a GGX microfacet shader \(\mathcal{S}_{\text{GGX}}\) under sampled light/view directions to produce \(\hat T_{\text{shaded}}\), which is then compared against the ground-truth illuminated texture using L2 + LPIPS losses. UV-space differentiable shading is preferred over traditional 2D rasterization supervision because the latter is highly sensitive to occlusions like hair and glasses, which bakes artifacts into the textures. Using a fixed template geometry with visibility mask weighting in the UV space allows physical constraints to be cleanly applied while avoiding occlusion contamination. For numerical stability, a simplified GGX shader is implemented (omitting the classical normalization denominator, bypassing tangent-space re-orthonormalization, and adding a low-intensity ambient term to prevent gradient explosion in fully occluded areas); details are provided in the appendix.

\[\hat T_{\text{shaded}}=\mathcal{S}_{\text{GGX}}\big(\hat T_{\text{alb}},\hat T_{\text{nrm}},\hat T_{\text{rough}},\hat T_{\text{spec}},\hat T_{\text{disp}};\,\tilde T_{\text{pos}},\tilde T^{\text{geo}}_{\text{nrm}},\tilde T^{\text{geo}}_{\text{tan}},L,V\big)\]

Loss & Training¶

Geometry reconstruction (MSSF encoder: trainable ConvNeXt V2 + frozen DINOv3, regressing Hifi3D++ parameters) utilizes a self-supervised objective \(\mathcal{L}=\lambda_{\text{pho}}\mathcal{L}_{\text{pho}}+\lambda_{\text{lan}}\mathcal{L}_{\text{lan}}+\lambda_{\text{per}}\mathcal{L}_{\text{per}}+\lambda_{\text{reg}}\mathcal{L}_{\text{reg}}\), where the photometric loss handles occlusions via a skin attention mask, and the standard 68-point landmarks are upgraded to an 88-point hybrid configuration (stable facial contour points + robust MediaPipe mouth points) to resolve instability in perioral detection. For material estimation, a hybrid objective combining diffusion supervision and rendering constraints is adopted: \(\mathcal{L}=\mathcal{L}_{\text{diff}}+\lambda_{\text{img}}\|\hat T_{\text{shaded}}-T_{\text{shaded}}\|_2^2+\lambda_{\text{lpips}}\text{LPIPS}(\hat T_{\text{shaded}},T_{\text{shaded}})\) (with \(\lambda_{\text{img}}=0.5\), \(\lambda_{\text{lpips}}=0.1\)). All components are trained in BF16 format using 8×H100 GPUs. The completion and homogenization LoRAs are trained for 20k steps each, while the material LoRAs are trained in a two-stage manner: 10k steps of independent training followed by 10k steps of joint training.

Key Experimental Results¶

Main Results¶

Geometry is evaluated on the REALY benchmark (100 subjects reporting NMSE across four regions: nose, mouth, forehead, and cheek); texture completion and illumination homogenization are quantitatively compared using PSNR, SSIM, LPIPS, and CSIM.

Task	Metric	Ours	Compared Methods	Description
Geometry Reconstruction (REALY All, mm↓)	NMSE	1.490	MoSAR 1.500 / HiFace 1.275	Third overall, yet outperforming non-linear MoSAR using a linear 3DMM
Texture Completion	PSNR↑	22.44	UV-IDM 19.39 / HRN 15.70	Leads in all metrics
Texture Completion	LPIPS↓	0.0621	UV-IDM 0.1674 / HRN 0.2982	Significantly better detail fidelity
Texture Completion	CSIM↑	0.540	UV-IDM 0.269 / HRN 0.441	Highest identity consistency
Illumination Homogenization	CSIM↑ / BS↓	0.4667 / 3.963	FFHQ-UV 0.1340 / 5.738	More uniform lighting and better identity preservation

Since MoSAR, FitMe, and Relightify are not open-source, their official test samples are utilized for qualitative comparison and a user study. Across three dimensions—geometric detail, texture realism, and relighting quality—the user preference rate for Ours is 100% against Relightify, 93.3%~96.7% against FitMe, and 83.3% for texture realism / 80.0% for relighting against MoSAR. Ours is also shown to surpass the commercial closed-source ChatAvatar in terms of detail and material completeness (the latter only provides albedo, normals, and specular maps, while Ours predicts the complete PBR set including roughness and displacement).

Ablation Study¶

Configuration	Key Phenomenon	Description
Full model	REALY All 1.490	Full model
w/o Enhanced \(\mathcal{L}_{\text{lan}}\)	All drops to 1.457 (mouth area degrades)	Removing the 88-point hybrid landmark loss
w/o MSSF	All drops to 1.806	Removing the DINOv3 semantic prior; all metrics deteriorate
w/o Illumination Homogenization	No reasonable material maps obtained even after 40k steps	Shadows are baked into albedo, normals/displacement exhibit noise
Independent LoRA (w/o Cross-Intrinsic Attention)	Normal map fractures, and pores turn into geometric spikes	Each modality overfits to high frequencies; pigmentation is misconstrued as geometry

Key Findings¶

Illumination homogenization is the primary contributor to stable training: removing it degrades material estimation from a well-posed inverse problem to an ill-posed one-to-many problem, collapsing into degenerate solutions. This is the most compelling ablation in the paper.
The DINOv3 semantic prior in MSSF contributes significantly: removing it causes all REALY metrics to deteriorate (All drops from 1.490 to 1.806). This prior is precisely what enables the linear 3DMM to outperform the non-linear MoSAR.
The value of Cross-Intrinsic Attention lies in geometric coherence: independent branches misinterpret skin pigmentation or stubble as geometry, leading to fractured normals. Joint training coupled with differentiable BRDF constraints ensures continuous crow's feet and smooth skin.
The cost is slow inference: on a single H100, completion takes 30s, homogenization takes 30s, and joint material estimation takes ~3 minutes, totaling ~4 minutes. This makes it more suitable for high-quality offline asset production rather than real-time applications.

Highlights & Insights¶

The paradigm of "a single frozen backbone + cascaded LoRAs tackling multiple sub-tasks" achieves extreme data efficiency: each LoRA uses only 0.7% of the backbone's parameters, yet fewer than 100 scans are sufficient to support completion, de-lighting, and material estimation tasks. This paradigm of "sharing generative priors and only learning task adaptation" can be easily ported to other data-scarce inverse rendering or intrinsic decomposition tasks.
Using Blender to "amplify" <100 scans into 100k synthetic pairs (mapping materials to in-the-wild geometries + rendering and baking under 2,041 HDRIs + generating gaps via visibility masks) represents a highly practical data engineering pipeline. It clearly demonstrates the methodology of "using scarce ground truths as physical anchors and relying on rendering engines to supply diversity".
Shifting physical constraints from screen space to UV space is a crucial insight: using fixed template geometry combined with visibility mask weighting enables physical rendering equation constraints while naturally preventing occlusions from baking artifacts into textures. This directly addresses the long-standing issue of screen-space rasterization supervision being highly sensitive to occlusions.
The Cross-Intrinsic Attention design utilizing "shared K/V, independent Q + individual LoRAs" is clever: it enforces alignment between different material maps while preserving modality-specific adaptations. This is a design highly worth reusing in other multi-task generative architectures.

Limitations & Future Work¶

The authors acknowledge focusing only on facial/head PBR assets, omitting full-body, dynamic hair, clothing, or accessories. Although treating hair/glasses/hands as occlusions aids facial material recovery, it loses identity details behind the occlusions and fails under extreme facial expressions.
Geometry relies on the linear Hifi3D++ basis, failing to capture extreme non-rigid deformations (e.g., blinking or asymmetric squinting causes rendering misalignment). Further, removing semi-transparent occlusions (like glasses) causes the perioral or periocular regions to over-smooth due to a lack of high-frequency information.
Inference latency is high (~4 minutes per image). The authors suggest utilizing consistency, adversarial, or distribution matching distillation to accelerate inference.
Fine-tuning LoRAs on small-scale professional scans diminishes the base model's open-domain text-editing capability. Striking a balance between physically accurate material estimation and flexible text-driven appearance editing remains an open question.
Personal Addendum: The geometry metrics only rank third on REALY. The authors primarily defend their performance by demonstrating superiority under identical training conditions alongside extensive qualitative comparisons and user studies. There is a lack of a unified quantitative material benchmark due to competitors not being open-source. Finally, the 4K super-resolution is a post-processing step powered by a fine-tuned Real-ESRGAN rather than being end-to-end; thus, the final detail quality is partially bottlenecked by the super-resolution module.

vs MoSAR: Both perform monocular relightable avatar reconstruction. MoSAR relies on a GAN backbone and differentiable shading, which is prone to mode collapse, a waxy appearance, and pale skin tones, while its screen-space rasterization struggles with occlusions. This work substitutes that with a diffusion prior and UV-space physical constraints, yielding more accurate skin tones and cleaner occlusion handling, while also outperforming MoSAR on REALY despite using linear geometry.
vs UltraAvatar: Both attempt to utilize diffusion models for avatar reconstruction. UltraAvatar fine-tunes an earlier diffusion model on a small dataset, which limits final quality and loses in-the-wild generalization. This work freezes the backbone while only training LoRAs, utilizing Blender to generate massive synthetic pairs to preserve both the generative prior and generalization capability.
vs UV-IDM / FreeUV (Texture Completion Methods): These methods bake illumination into textures, precluding relighting. This work explicitly performs illumination homogenization and PBR material decomposition, outputting relightable assets.
vs NextFace (Differentiable Ray Tracing to Explicitly Model Self-Occlusion): Shared philosophy (both aim to bypass screen-space supervision sensitivity to occlusions). However, NextFace is computationally expensive and generalizes poorly. This work runs differentiable shading in the UV space based on a fixed template geometry, which is more efficient and generalizable.
vs HiFace / 3DDFA-V3 (Geometry SOTA): This work ranks slightly lower quantitatively in geometry. However, HiFace relies on large-scale synthetic images and real meshes, while 3DDFA-V3 uses segmentation-clustered landmarks but is prone to overfitting and generating geometric bulges. Under equivalent in-the-wild training settings, this work proves more stable and reconstructs sharper high-frequency details courtesy of the predicted normal and displacement maps.

Rating¶

Novelty: ⭐⭐⭐⭐ The combination of "reusing a diffusion backbone via cascaded LoRAs + Cross-Intrinsic Attention + UV-space differentiable BRDF loss" provides a clean and effective mechanism for handling data efficiency and occlusion robustness, although individual components mostly represent the clever integration of existing technologies.
Experimental Thoroughness: ⭐⭐⭐⭐ Includes quantitative evaluations on REALY for geometry, multi-metric comparisons for textures, and thorough ablations explaining the impact of illumination homogenization and joint attention. However, due to competitors being closed-source, material evaluation relies solely on qualitative evaluations and user studies, lacking a unified quantitative benchmark.
Writing Quality: ⭐⭐⭐⭐ The logic of motivation is very clear, the figures match the text well, and the ablations are interpreted with solid physical intuition. The appendix details the differentiable shader equations thoroughly.
Value: ⭐⭐⭐⭐ Reconstructing 4K relightable avatars using <100 scans significantly lowers the barrier to high-quality digital human creation. It holds practical value for film, gaming, and virtual production. The authors also responsibly discuss deepfake misuse and mitigation strategies.