title: >- [Paper Note] Generative Relightable Avatars description: >- [ECCV 2026][3D Vision][Relightable Avatars] Combines explicit microfacet rendering and UV-space RelightNet with a flow-matching video diffusion model to achieve photorealistic, temporally coherent full-body human relighting under arbitrary illumination. tags: - ECCV 2026 - 3D Vision - Relightable Avatars - Video Generation - Microfacet BRDF date: 2026-09-19 content_hash: 1439f73275ec8eab

Generative Relightable Avatars¶

Conference: ECCV 2026
Paper: ECCV 2026 Official
Area: 3D Vision
Keywords: Full Body Avatars, Relighting, Generative Diffusion Models, Microfacet BRDF, Flow Matching

TL;DR¶

Generative Relightable Avatars (GRA) decomposes full-body human relighting into explicit physics-grounded coarse rendering and flow-matching-based video generative refinement, achieving photorealistic, temporally coherent free-view avatars with rich dynamic wrinkle patterns under arbitrary novel illuminations.

Background & Motivation¶

Synthesizing photorealistic full-body digital human avatars that can be faithfully relit under arbitrary novel illumination environments and free-view camera trajectories is fundamental for AR/VR telepresence, virtual production, and interactive media. Dynamic human bodies draped in loose garments pose extreme modeling difficulties: fine-grained appearance details such as dynamic clothing wrinkles, cast self-shadows, and view-dependent micro-specular highlights are essential visual cues for realism. Traditional approaches to relightable avatars predominantly rely on deterministic inverse rendering over neural implicit fields or 3D Gaussian representations. However, because skeletal motion is a sparse driving signal that physically maps to a multitude of feasible wrinkle patterns in a one-to-many relationship, deterministic regressions under single-bounce analytic BRDF assumptions inevitably regress to the statistical mean, resulting in over-smoothed textures and muted lighting responses.

Conversely, generative diffusion models operating directly in 2D image or video space demonstrate an impressive capacity to hallucinate intricate lighting and fine garment details. Nevertheless, because they function primarily on image planes without explicit 3D skeletal or surface geometry anchoring, they lack multi-view consistency and precise free-viewpoint 3D controllability. Attempting to train an unconstrained monolithic generative diffusion model to simultaneously solve for body motion, deforming surface geometry, camera viewpoint, and lighting distribution causes severe multi-modal training collapse and significant spatial drifting over time.

This paper navigates this trade-off by adopting a divide-and-conquer strategy: rather than demanding that a single generative model solve everything from scratch, it decouples the problem into physics-grounded coarse intermediate rendering and generative fine-scale refinement. The core idea is to structure relightable avatar generation as a hybrid pipeline combining explicit UV-space microfacet rendering and a feed-forward RelightNet for physical 3D grounding, followed by a flow-matching video diffusion model with atemporal lighting cross-attention and error recycling for high-frequency perceptual refinement.

Method¶

Overall Architecture¶

The input to GRA comprises a dynamic skeletal motion sequence $\boldsymbol{\theta}$, target virtual camera viewpoints $\boldsymbol{\kappa}$, and arbitrary target HDR environment maps $\boldsymbol{E}$. The framework produces temporally coherent, photorealistic video renderings of the character via three sequential stages. First, a tracked person-specific parametric mesh is leveraged to optimize spatially-varying albedo and roughness maps in UV space under an analytic microfacet model, generating a coarse physics-based texture. Second, a lightweight feed-forward convolutional network, RelightNet, processes multi-frame normal histories and per-texel viewing directions in UV space to approximate higher-order shading and pose-dependent dynamics, rasterizing into coarse viewpoint renderings $\boldsymbol{I}_t^\kappa$. Finally, a finetuned video-to-video diffusion model (WAN2.1-VACE) probabilistically synthesizes high-frequency clothing wrinkles and specular nuances conditioned on the coarse renderings and per-frame environment map latents.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Inputs: Skeletal Pose θ + Viewpoint κ + Envmap E"] --> B["Physics-based Texture Estimation<br/>Optimize UV albedo a and roughness r"]
    B --> C["RelightNet Pose-dependent Relighting<br/>Condition on normal history and view directions"]
    C --> D["Virtual Camera Rasterization<br/>Produce coarse renderings I_κ"]
    D --> E["Generative Video Refinement<br/>WAN2.1-VACE backbone + atemporal cross-attention"]
    E --> F["Error Recycling Stabilization<br/>Re-inject reconstruction residuals to prevent drift"]
    F --> G["Final Photorealistic Relit Avatar Video"]

Key Designs¶

1. Physics-based Texture Estimation: Anchoring the Generative Model with Explicit Physics

Monolithic video generators lack intrinsic geometric awareness and struggle to maintain correct shadow orientation across novel viewpoints. To supply downstream generative refinement with a pixel-aligned, physically consistent foundation, GRA initializes human appearance using a person-specific Deep Dynamic Characters (DDC) tracked mesh $M_t$. The outgoing radiance is formulated via a simplified Cook-Torrance microfacet BRDF: $$f_r(\boldsymbol{x}, \boldsymbol{\omega}_i, \boldsymbol{\omega}_o) = \mathrm{BRDF}(\boldsymbol{\omega}_i, \boldsymbol{\omega}_o; \boldsymbol{a}(\boldsymbol{x}), \boldsymbol{r}(\boldsymbol{x}))$$ where $\boldsymbol{a}(\boldsymbol{x})$ and $\boldsymbol{r}(\boldsymbol{x})$ represent spatially-varying albedo and roughness maps. Assuming single-bounce environment illumination, the static UV maps are optimized across training frames by minimizing the L1 photometric discrepancy against multi-view ground-truth images: $$\boldsymbol{a}^*, \boldsymbol{r}^* = \arg\min_{\boldsymbol{a}, \boldsymbol{r}} \sum_t \left\| \mathcal{P}(M_t, \boldsymbol{\kappa}_t, \boldsymbol{E}_t; \boldsymbol{a}, \boldsymbol{r}) - \boldsymbol{I}_t^{\mathrm{gt}} \right\|_1$$ The resulting coarse UV texture $\boldsymbol{I}^{\mathrm{coarse}}_t$ provides robust geometric low-frequency shading dependencies that constrain subsequent neural models from generating physically invalid hallucinations.

2. RelightNet Pose-dependent Relighting: Bridging Simplified BRDFs and Dynamic Shading

Static UV textures on an imperfectly tracked mesh lead to over-smoothed dynamic appearance, and the single-bounce microfacet assumption cannot capture secondary scattering, ambient occlusions, or complex body self-shadows. To bridge this gap, RelightNet $\mathcal{R}$ is introduced as an efficient feed-forward UNet operating in UV space. It maps the coarse texture alongside a temporal window of surface normal maps $\boldsymbol{N}_{t-k:t}$ and per-texel camera viewing directions $\boldsymbol{C}_t$ to a refined relit texture: $$\boldsymbol{I}_t = \mathcal{R}(\boldsymbol{E}_t, \boldsymbol{I}^{\mathrm{coarse}}_t, \boldsymbol{N}_{t-k:t}, \boldsymbol{C}_t)$$ By incorporating normal map dynamics and cross-attending to the target environment map $\boldsymbol{E}_t$, RelightNet captures complex dynamic shadows cast by limb motion, producing coarse renderings $\boldsymbol{I}_t^\kappa$ upon camera projection.

3. Atemporal Lighting Conditioning: Preventing Inter-frame Illumination Blurring

Standard video diffusion architectures utilize temporal cross-attention to enforce temporal continuity across frames. However, in relighting datasets such as RHC, the training sequences feature distinct, dynamic illumination conditions at every single frame. Applying temporal mixing directly across illumination features leads to temporal blur and dulls rapid lighting transitions. GRA projects camera-space environment maps $\boldsymbol{E}_t^\kappa$ through the WAN2.1 3DVAE encoder into latent tokens $\boldsymbol{\ell} \in \mathbb{R}^{T' \times H' \times W' \times d}$ and injects them into the DiT backbone via dedicated atemporal spatial cross-attention: $$\tilde{\boldsymbol{f}} = \mathrm{Concat}_i\left(\mathrm{CrossAttn}(\boldsymbol{f}^i, \boldsymbol{\ell}^i)\right)$$ Restricting cross-attention strictly within each individual frame enforces an inductive bias that anchors per-frame lighting responses, preserving sharp shadow boundaries and enabling zero-shot generalization to one-light-at-a-time (OLAT) strobe illumination.

4. Error Recycling: Mitigating Autoregressive Temporal Drift in Long Sequences

The generative stage deploys a Video Control Unit (VCU) to ingest coarse renderings as reactive frames while utilizing an identity reference image under neutral illumination to preserve subject likeness. In long-horizon autoregressive video rollout, slight synthesis discrepancies inevitably compound into severe temporal drift and character distortion. To resolve this, GRA incorporates Error Recycling during training: reconstruction residuals are computed from first-order latent velocity regressions: $$\boldsymbol{e}_0 = \hat{\boldsymbol{z}}_0 - \boldsymbol{z}_0^{\mathrm{rcy}}, \quad \boldsymbol{e}_1 = \hat{\boldsymbol{z}}_1 - \boldsymbol{z}_1^{\mathrm{rcy}}$$ These residuals are cached and re-injected as stochastic perturbations into clean latents $\tilde{\boldsymbol{z}}_1$, noisy latents $\tilde{\boldsymbol{z}}_0$, and inactive context latents $\tilde{\boldsymbol{z}}_{\mathrm{inac}}$ with probabilities $0.01$, $0.9$, and $0.9$. By explicitly training the flow-matching model on self-corrupted trajectories, error recycling regularizes the network against error accumulation and stabilizes long-form generation beyond 128 frames.

Loss & Training¶

The framework is optimized through a decoupled training paradigm: 1. Material Inverse Rendering: Static albedo $\boldsymbol{a}$ and roughness $\boldsymbol{r}$ are optimized in UV space via differentiable rendering using an L1 loss against ground-truth images. 2. RelightNet Supervision: Supervised end-to-end in UV texture space using an L1 loss between projected renderings and ground-truth frames. 3. Flow Matching with Error Recycling: Conditioned on coarse renderings and environment map tokens, the DiT backbone is trained under a latent flow matching velocity objective: $$\mathcal{L}_{\mathrm{ER}} = \mathbb{E}_s \left\| \boldsymbol{v}(\tilde{\boldsymbol{z}}_s, s, \boldsymbol{c}) - (\boldsymbol{z}_1 - \tilde{\boldsymbol{z}}_0) \right\|_2^2$$ To cultivate temporal roll-out capability, the first 5 frames are replaced with ground-truth frames with a $40\%$ probability and marked as inactive context anchors ($M_{1:5} = 0$), guiding the model to perform seamless temporal continuation.

Key Experimental Results¶

Main Results¶

Quantitative evaluations are conducted on the multi-view Relightable Holoported Characters (RHC) benchmark across four subjects (S1-S4). The training uses 1,015 HDR environment maps from the Laval Indoor dataset and tests on 8 held-out novel illuminations across 3 unseen camera viewpoints. Baselines include Relighting4D (R4D, upgraded with DDC tracking), IntrinsicAvatar (IA), MeshAvatar (MA), and DNF-Avatar (DNFA).

Subject & Method	PSNR (dB) ↑	LPIPS (×10⁻²) ↓	SSIM (%) ↑	FVD ↓
Subject 1 (S1)
R4D (Chen et al. 2022)	29.89	10.31	87.15	247
IA (Wang et al. 2024)	27.25	18.25	81.39	408
MA (Chen et al. 2024)	29.71	9.09	84.69	77
DNFA (Jiang et al. 2025)	26.32	11.57	82.50	212
Ours (GRA)	30.49	6.42	87.52	49
Subject 2 (S2)
R4D	31.13	8.04	87.08	161
IA	28.87	15.43	82.07	345
MA	30.36	8.03	84.99	74
DNFA	28.57	9.40	82.92	131
Ours (GRA)	31.15	5.64	87.50	48
Subject 3 (S3)
R4D	29.32	10.62	87.67	186
IA	26.14	22.52	79.07	621
MA	28.35	10.16	84.60	64
DNFA	26.37	12.04	82.26	150
Ours (GRA)	29.11	7.40	87.16	52
Subject 4 (S4)
R4D	31.98	10.07	87.90	168
IA	29.50	18.46	82.91	354
MA	31.13	8.50	84.59	73
DNFA	29.31	10.82	84.12	124
Ours (GRA)	32.09	6.55	87.35	63

Ablation Study¶

Ablation experiments conducted on Subject 1 evaluate the impact of individual architectural components and training mechanisms.

Configuration	PSNR (dB) ↑	LPIPS (×10⁻²) ↓	SSIM (%) ↑	FVD ↓	Note
Microfacet Rendering	29.34	12.26	86.85	178.42	Explicit physics-only; lacks high-frequency clothing detail
+ RelightNet	30.90	8.92	89.25	118.24	Adds UV feed-forward neural refinement; boosts PSNR
+ Generative Refinement (Ours)	30.49	6.42	87.52	48.79	Full model; achieves superior perceptual and temporal fidelity
Ours w/ Temporal Lighting Conditioning	30.78	6.53	88.00	58.18	Temporal cross-attention blurs per-frame lighting changes
Ours w/o Atemporal Lighting Conditioning	30.53	6.48	87.76	58.21	Omits explicit envmap tokens; relies solely on coarse guidance
Ours w/o VACE	30.35	6.93	87.12	61.84	Removes VACE spatio-temporal conditioning adapters
Ours w/o RelightNet	29.56	6.66	87.26	63.81	Feeds raw microfacet rendering directly to video model
Ours w/o VACE Finetuning	24.92	22.43	70.76	230.94	Frozen adapters fail to align coarse geometry; model collapses
Ours w/o Video Prior	30.02	9.60	86.67	394.91	Training DiT from scratch fails temporal consistency (FVD 394.91)
Ours w/ Neural Gaffer	29.33	9.25	86.62	102.10	Frame-by-frame 2D diffusion suffers severe temporal flickering
Ours w/ Few-shot Adaptation	29.82	7.01	87.30	65.25	Transfers to a new identity from only 170 frames (~6 seconds)

Key Findings¶

Generative Refinement Unlocks Perceptual Realism: Incorporating the generative refinement stage decreases LPIPS from $8.92 \times 10^{-2}$ to $6.42 \times 10^{-2}$ ($28.0\%$ relative improvement) and slashes FVD from $118.24$ to $48.79$ ($58.7\%$ reduction). Deterministic regressions inevitably blur out stochastic garment creases, whereas the generative prior hallucinates photorealistic high-frequency wrinkles.
Pretrained Video Priors Are Critical for Temporal Stability: Training the DiT architecture from scratch without pretrained video priors causes FVD to skyrocket to $394.91$. Similarly, substituting the video backbone with the 2D image relighting model Neural Gaffer results in substantial inter-frame flickering ($\text{FVD} = 102.10$), proving that joint spatio-temporal video priors are essential for avatar synthesis.
Out-of-Distribution Generalization to OLAT: Grounding the generative model in explicit microfacet normals and shadow computations enables zero-shot generalization to unseen one-light-at-a-time (OLAT) environments and localized near-field point lights, generating crisp, physically consistent cast shadows.

Highlights & Insights¶

Physics-Grounded Generative Hybridization: Rather than forcing a generative model to solve geometry, tracking, and rendering simultaneously, GRA delegates low-frequency physical shading and 3D camera correspondence to classical graphics models, dedicating diffusion capacity entirely to stochastic high-frequency appearance synthesis.
Atemporal Cross-Attention for Dynamic Illumination: While video synthesis typically favors temporal cross-attention, GRA recognizes that rapid, per-frame illumination variations require uncorrupted frame-wise conditioning, employing strictly atemporal spatial cross-attention to maintain crisp lighting transients.
Direct Texture Controllability via UV Albedo Editing: Because explicit material parameters are recovered in UV space, users can directly edit the 2D albedo map (e.g., painting customized patterns or changing garment colors). The video refinement stage automatically respects the modified coarse render without requiring retraining.

Limitations & Future Work¶

Limited Facial and Hand Articulation: The system relies on whole-body parametric DDC meshes and lacks dedicated high-resolution geometric tracking for subtle facial expressions, gaze shifts, and delicate hand-finger articulation, leading to minor blurriness in extreme close-up views.
Heuristic Near-Field Light Handling: Near-field point light relighting currently bypasses RelightNet via a fallback configuration ("w/o RelightNet"), as the feed-forward UV network is primarily tailored to far-field HDR environment maps.
Inference Latency of Diffusion Sampling: Multi-step ODE integration during flow matching imposes non-trivial computational latency, currently precluding real-time interactive rendering at 60 FPS without aggressive distillation or caching techniques.

vs Relighting4D (R4D): R4D fits a non-rigid neural radiance field under spherical harmonic illumination. Even when enhanced with the same high-precision DDC mesh, R4D remains limited by deterministic regression, achieving an LPIPS of $10.31 \times 10^{-2}$ on S1 compared to GRA's $6.42 \times 10^{-2}$, with noticeable blur around fast-moving folds.
vs IntrinsicAvatar (IA) & MeshAvatar (MA): IA suffers from SMPL tracking inaccuracies that compound through ray-tracing, yielding an FVD of 408 on S1. MA improves sharpness via a hybrid representation but lacks generative modeling for cloth dynamics. GRA outperforms MA by a wide margin in video motion consistency, reducing FVD from 77 to 49 on S1.
vs 2D Diffusion Relighting (Neural Gaffer, DiffRelight): 2D image-space relighting approaches lack consistent 3D geometric constraints across novel camera views and display prominent frame-to-frame temporal jitter ($\text{FVD} = 102.10$), whereas GRA enforces strict 3D view control and temporal coherence ($\text{FVD} = 48.79$).

Rating¶

Novelty: ⭐⭐⭐⭐ [Compelling integration of explicit microfacet inverse rendering and video latent flow matching for 3D relightable avatars]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive multi-subject benchmarking, comprehensive ablations, and zero-shot OOD evaluations on OLAT and near-field lighting]
Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical formulation, cohesive narrative bridging graphics and generative vision, and well-structured diagrams]
Value: ⭐⭐⭐⭐ [Offers a practical and robust paradigm for next-generation telepresence, virtual production, and interactive human avatars]