Skip to content

One-Shot Feed-Forward 360° Animatable Avatar via Inpainted UV-Space Gaussian Modeling

Conference: ECCV 2026
Paper: CVF Open Access
Project: https://shaelynz.github.io/fhavatar/
Area: 3D Vision
Keywords: 3D Gaussian Splatting, Animatable Head Avatar, One-shot 3D Reconstruction, UV-Space Modeling, Feed-Forward Inference

TL;DR

FHAvatar is a one-shot feed-forward 360° animatable full-head avatar creation framework that leverages 3D GAN priors and symmetric UV-space inpainting to regress rigged 3D Gaussians on parametric meshes, achieving high-fidelity 360° rendering and real-time animation at 246 FPS.

Background & Motivation

Reconstructing 3D animatable head avatars from a single 2D image is fundamental for teleconferencing, virtual reality, and augmented reality. While optimization-based methods using multi-view or monocular video capture yield impressive visual realism, their reliance on per-identity, frame-by-frame optimization spanning thousands of frames renders them impractical for interactive or instant deployment. To circumvent this efficiency bottleneck, feed-forward single-image reconstruction pipelines utilizing Neural Radiance Fields (NeRF) or 3D Gaussian Splatting (3DGS) have emerged, offering instantaneous avatar generation in a single inference pass.

However, existing one-shot feed-forward head avatar models suffer from severe quality degradation when rendering viewpoints deviate significantly from the frontal input pose. Because these models are predominantly trained on large-scale monocular facial video datasets such as VFHQ and CelebV-HQ—which overwhelmingly feature near-frontal camera motions—they inherently lack observability of the lateral and posterior regions of the head. When forced to synthesize side or back views, prior approaches inevitably succumb to geometric collapse, texture stretching, or facial distortion due to the unconstrained ambiguity of unseen surfaces, compromising avatar immersion.

The underlying challenge stems from a dual trade-off: a single input portrait provides rich, high-frequency details strictly for frontally visible regions, while 3D GAN inversion offers 360° full-head geometric priors but suffers from limited reconstruction fidelity due to constrained latent code capacity and low tri-plane resolution. To reconcile this tension, the authors formulate head modeling on the UV parameterization of the FLAME parametric face mesh, extracting coarse 360° priors from a pretrained 3D GAN while exploiting the intrinsic facial and UV symmetry to inpaint incomplete, high-resolution source image features. Core idea: Inpaint incomplete local UV features from a single portrait using global full-head priors from a pretrained 3D GAN within a parametric face model's UV space, leveraging facial and UV symmetry for dual-sided feature querying and detail enhancement to reconstruct 360° animatable full-head 3D Gaussians.

Method

Overall Architecture

FHAvatar takes a single source portrait \(I_s\) and target driving facial expressions and poses (tracked via FLAME from a driving frame \(I_d\)), directly predicting 3D Gaussian primitives embedded on the FLAME mesh surface in canonical space. The pipeline operates across three primary stages: first, the UV-Space Feature Extraction module derives coarse full-head tri-plane representations via 3D GAN inversion, unrolls multi-scale 2D image features into incomplete local UV feature maps, and applies a lightweight 2D UNet to predict geometric offsets that rectify hair shape discrepancies; second, the Symmetric UV-Space Inpainting module leverages Transformer cross-attention and horizontal flipping across multiple feature scales to transfer high-fidelity details from visible patches and symmetric counter-parts into the global UV representation; third, a UV Map Decoder translates the inpainted feature maps into 14-channel UV Gaussian attribute maps, from which 3D Gaussians are sampled, deformed with driving expressions, and rendered via 3DGS under a specialized 3D total variation regularization.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Input Source Image Is & FLAME Canonical Mesh"] --> B["UV Shape Refinement & Dual-Path Feature Extraction<br/>3D GAN Inversion + Multi-Scale 2D Projection"]
    B --> C["Symmetric UV-Space Inpainting<br/>Transformer Window Query + Flipped Detail Enhancement"]
    C --> D["UV Gaussian Attribute Map Generation<br/>Decode Position / Rotation / Scale / Opacity / Color"]
    D --> E["3D Gaussian Primitive Rigging & 360° Animation<br/>Expression-Driven Deformation with 3D TV Regularization"]

Key Designs

1. UV Shape Refinement & Dual-Path Feature Extraction: Decoupling Global Geometry from Local Details

Because a single 2D image \(I_s\) cannot provide geometry or appearance for the back of the head, FHAvatar deploys the pretrained 3D full-head GAN PanoHead and its feed-forward inversion network to map \(I_s\) into a canonical full-head tri-plane \(\mathbf{T}\). However, parametric FLAME meshes lack non-facial volumetric structures like hair, causing 3D-position-based tri-plane sampling on raw mesh coordinates to misalign. To resolve this, a 2D UNet \(\mathcal{F}_{refine}\) consumes the initial sampled tri-plane feature \(\mathbf{F}_{\mathbf{p}}^{\mathbf{T}}\) concatenated with the base UV 3D position map \(\mathbf{p}\) to predict a spatial displacement map \(\Delta\mathbf{p}_{uv}\): $\(\Delta\mathbf{p}_{uv} = \mathcal{F}_{refine}([\mathbf{F}_{\mathbf{p}}^{\mathbf{T}}, \mathbf{p}])\)$ Sampling \(\mathbf{T}\) at the refined coordinates \(\mathbf{p}_r = \mathbf{p} + \Delta\mathbf{p}_{uv}\) produces a topologically complete, coarse global full-head UV feature map \(\mathbf{F}_g\). Concurrently, a 2D CNN extracts multi-scale appearance features \(\{\mathbf{F}_s^i\}_{i=1}^N\) from \(I_s\), which are projected into UV space using \(\mathbf{p}_r\) and camera poses. A z-buffer UV visibility mask \(\mathbf{M}_v\) filters out occluded surface texels, yielding fine-grained but partially observed local UV feature maps \(\{\mathbf{F}_l^i\}_{i=1}^N\).

2. Symmetric UV-Space Inpainting: Symmetry-Guided Cross-View Detail Transfer

While \(\mathbf{F}_g\) covers the full head, it lacks fine appearance details; conversely, \(\{\mathbf{F}_l^i\}\) preserves rich local textures but suffers from severe view-dependent missing regions. Direct interpolation introduces seams and projection misalignment. FHAvatar introduces a two-tier symmetric inpainting mechanism exploiting facial bilateral symmetry. In the coarse inpainting stage, a Transformer cross-attention module treats patches from the encoded global feature \(\mathbf{F}_g^i\) as Queries, which attend to Keys and Values extracted from a local window of size \(w \times w\) centered at the corresponding position in \(\mathbf{F}_l^i\). Crucially, to handle side-view source images where half the face is occluded in \(\mathbf{F}_l^i\), tokens also attend to Keys and Values situated in the horizontally symmetric local window of \(\mathbf{F}_l^i\). In the local detail enhancement stage, to counteract resolution degradation and block artifacts from patchification, the model predicts an occlusion confidence mask \(\mathbf{M}_o\) from the Transformer output \(\mathbf{F}_c^i\) and \(\mathbf{F}_l^i\): $\(\mathbf{F}_{l,m}^i = \mathbf{M}_o \odot \mathbf{F}_l^i\)$ $\(\mathbf{F}_m^i = \mathbf{F}_{l,m}^i + \mathcal{F}_{\leftrightarrow}(\mathbf{F}_{l,m}^i) \odot (1 - \mathbf{M}_v \odot \mathbf{M}_o)\)$ where \(\mathcal{F}_{\leftrightarrow}\) denotes horizontal flipping. Convolving the concatenation \([\mathbf{F}_c^i, \mathbf{F}_m^i]\) outputs the final feature \(\mathbf{F}_f^i\), injecting crisp symmetric local details into the global representation.

3. 3D Total Variation Regularization: Suppressing Discretization Holes and Cross-Surface Artifacts

Binding 3D Gaussian primitives to a UV grid accelerates 2D convolutional decoding and expression skinning, but discrete primitive spacing on complex curved geometries can yield surface voids. These holes allow back-head Gaussians to leak visually into frontal views. Prior art addressed this using 2D total variation on rendered UV coordinate buffers (\(\mathcal{L}_{uv}\)), but UV seam discontinuities around the eyelids, lips, and eyeballs caused unnatural facial gluing and distortion. FHAvatar introduces a 3D total variation loss \(\mathcal{L}_{3d}\) executed directly on the rendered 3D coordinate map \(\mathbf{I}_{3d}\), formed by temporarily substituting Gaussian color channels with their 3D world coordinates \(\mathbf{G}_{position}\): $\(\mathcal{L}_{3d} = \mathrm{TV}(\mathbf{I}_{3d} - (1 - \mathbf{I}_\alpha))\)$ where \(\mathbf{I}_\alpha\) is the rendered alpha silhouette. This constraint enforces spatial continuity between adjacent screen pixels in world space, eliminating see-through holes while respecting natural topological discontinuities in the UV mapping.

Loss & Training

The framework is trained end-to-end, holding the 3DMM tracker and 3D GAN inversion models frozen. To prevent multi-view hallucination drift between video frames, training alternates between Animation Mode (reconstructing driving frame \(I_d\) from source \(I_s\)) and 3D Reconstruction Mode (reconstructing novel pseudo-views of \(I_d\) using \(I_d\) as source). The total objective combines multi-view reconstruction losses and geometric regularizers: $\(\mathcal{L} = \mathcal{L}_{re} + \mathcal{L}_\alpha + \lambda_{id}\mathcal{L}_{id} + \lambda_1 (\mathcal{L}_{lpips}^1 + \mathcal{L}_\alpha^1) + \mathcal{L}_{reg}\)$ where \(\mathcal{L}_{re} = \mathcal{L}_1 + \mathcal{L}_{lpips}\), \(\mathcal{L}_{id}\) is ArcFace cosine similarity, and \(\mathcal{L}_{reg} = \lambda_{3d}\mathcal{L}_{3d} + \lambda_{eye}\mathcal{L}_{eye} + \lambda_{pos}\mathcal{L}_{pos} + \lambda_{shape}\mathcal{L}_{shape} + \lambda_{shape}^{tv}\mathcal{L}_{shape}^{tv}\).

Key Experimental Results

Main Results

The model was benchmarked on the VFHQ test set and HDTF (20 unseen videos) against 2D talking-head SOTA StyleHEAT, as well as 3D NeRF and 3DGS methods: Real3DPortrait, Portrait4D, Portrait4D-v2, GAGAvatar, and LAM. Quantitative evaluations span self-reenactment and cross-identity reenactment.

Dataset Method PSNR ↑ SSIM ↑ LPIPS ↓ CSIM ↑ AKD ↓ AED ↓ APD ↓ Speed (FPS) ↑
VFHQ (Self) StyleHEAT 18.96 0.7505 0.3821 0.1695 20.78 0.7236 4.1380 2.19
VFHQ (Self) Real3DPortrait 21.00 0.7572 0.2915 0.7696 3.934 0.3918 1.1890 11.02
VFHQ (Self) Portrait4D-v2 20.66 0.7421 0.2719 0.8075 5.367 0.3539 1.6181 18.39
VFHQ (Self) GAGAvatar 21.60 0.7745 0.2249 0.8459 2.954 0.3125 0.8898 58.11
VFHQ (Self) LAM 21.67 0.7756 0.2716 0.6846 3.676 0.4535 1.4286 231.74
VFHQ (Self) Ours 23.24 0.7995 0.2384 0.8012 2.798 0.3634 0.8660 246.00
HDTF (Self) StyleHEAT 21.91 0.8102 0.2979 0.5096 6.564 0.4860 1.1539 2.19
HDTF (Self) Real3DPortrait 23.39 0.8110 0.2362 0.8570 3.104 0.3413 0.7633 11.02
HDTF (Self) GAGAvatar 23.72 0.8177 0.2089 0.8894 2.679 0.3034 0.5977 58.11
HDTF (Self) LAM 23.55 0.8167 0.2362 0.7542 3.386 0.4180 0.8911 231.74
HDTF (Self) Ours 26.61 0.8642 0.1900 0.8622 2.287 0.3318 0.5286 246.00

In cross-identity reenactment on VFHQ, the method achieved a CSIM of 0.6757 (outperforming GAGAvatar's 0.6681 and LAM's 0.5562). On HDTF cross-reenactment, CSIM reached 0.8568 compared to LAM's 0.7656.

Ablation Study

Ablation experiments conducted on HDTF and the multi-view MEAD dataset validate each algorithmic component:

Config HDTF PSNR ↑ HDTF SSIM ↑ HDTF CSIM ↑ HDTF AKD ↓ MEAD PSNR ↑ MEAD CSIM ↑ MEAD AKD ↓ MEAD APD ↓ Note
Ours (Full model) 26.61 0.8642 0.8622 2.287 17.20 0.6765 4.844 1.9484 Full design performs best
w/o \(\Delta\mathbf{p}_{uv}\) 26.15 0.8596 0.8618 2.338 17.09 0.6678 5.887 2.6566 Removing mesh refinement causes hair blur
w/o \(\{\mathbf{F}_l^i\}_{i=1}^N\) 23.82 0.8186 0.5230 2.732 17.09 0.3932 4.849 1.9517 Relying solely on GAN tri-plane destroys identity
w/o \(\mathcal{T}\) (Transformer) 25.79 0.8579 0.8461 2.335 17.05 0.6500 4.792 2.1080 Removing cross-attention produces eye/teeth misalignment
w/o Sym. (Symmetry) 26.60 0.8651 0.8646 2.307 17.18 0.6631 5.103 2.2145 Significant drop under large multi-view pose shifts
Replacing with \(\mathcal{L}_{uv}\) 26.40 0.8622 0.8567 2.324 17.08 0.6663 5.058 1.9627 Conventional UV TV causes boundary tearing around eyes/mouth

Key Findings

  • Local Features Dictate Identity Retention: Discarding local UV feature maps \(\{\mathbf{F}_l^i\}\) causes catastrophic drops in identity similarity (CSIM plunged from 0.8622 to 0.5230 on HDTF, and from 0.6765 to 0.3932 on MEAD), establishing that GAN inversion tri-planes alone cannot capture personal likeness.
  • Symmetric Prior Is Essential for Multi-View Generalization: While the symmetric module shows minor differences on frontal HDTF videos, removing it on multi-view MEAD noticeably degraded tracking metrics (AKD worsened from 4.844 to 5.103, APD from 1.9484 to 2.2145), demonstrating its critical role when only a profile is exposed.
  • Superiority of 3D TV over UV TV: Replacing \(\mathcal{L}_{3d}\) with \(\mathcal{L}_{uv}\) degraded PSNR on HDTF (26.40 vs 26.61) and generated severe blending artifacts between eyelids and eyeballs, validating the need for spatial continuity constraints in world space rather than texture space.

Highlights & Insights

  • Synergizing 3D GAN Full-Head Priors with Explicit 3DGS: Instead of performing costly test-time GAN latent optimization, FHAvatar leverages feed-forward inversion as an initial 3D scaffold, bridging the gap between complete 360° geometry and feed-forward real-time 3D Gaussian decoding at 246 FPS.
  • Symmetry-Aware Inpainting in Unwrapped Face UV Space: Transforming bilateral facial geometry into symmetric 2D coordinate spaces provides an elegant solution to the single-view occlusion problem without resorting to slow generative diffusion inpainting.
  • World-Space Coordinate TV for Surface Continuity: Applying total variation on rendered 3D coordinates rather than texture space coordinates effectively resolves the topological tearing caused by UV seams around the facial orifices.

Limitations & Future Work

  • Dependency on Upstream FLAME Tracking: Driving fidelity (AED/APD) is bounded by the precision of the monocular FLAME tracker; tracking jitter directly impacts the animated Gaussian positions.
  • Complex Hair and Accessory Generalization: Because Gaussian offsets are anchored to a smooth facial mesh topology, voluminous hairstyles, dynamic curls, or heavy eyeglasses exceeding the displacement capacity of \(\Delta\mathbf{p}_{uv}\) may exhibit minor stretching artifacts.
  • Future Directions: Decoupling hair and accessories into separate layered Gaussian primitives or conditioning the UV inpainting network on multi-modal geometry diffusion priors could enhance high-frequency structures.
  • vs GAGAvatar (NeurIPS 2024): GAGAvatar employs separate expression/appearance Gaussians with a neural refiner but deteriorates under wide off-axis angles; FHAvatar integrates 3D GAN global priors to provide consistent 360° full-head coverage.
  • vs LAM (SIGGRAPH 2025): LAM queries features on FLAME vertices using Transformers but lacks back-head observation; FHAvatar achieves superior PSNR (HDTF 26.61 vs 23.55) and higher rendering throughput (246 FPS vs 231 FPS) through UV parameterization and tri-plane feature completion.
  • vs GGHead (SIGGRAPH Asia 2024): GGHead relies on UV-space TV regularization (\(\mathcal{L}_{uv}\)), which induces hole-filling artifacts across facial seams; FHAvatar resolves this via the novel 3D TV loss \(\mathcal{L}_{3d}\).

Rating

  • Novelty: ⭐⭐⭐⭐ [Pioneers one-shot feed-forward 360° animatable avatar generation via symmetric UV inpainting and 3D TV loss]
  • Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Rigorous self and cross-identity benchmarks across VFHQ, HDTF, and MEAD with detailed ablation]
  • Writing Quality: ⭐⭐⭐⭐⭐ [Clear mathematical formulations, well-structured figures, and logically sound narrative]
  • Value: ⭐⭐⭐⭐⭐ [Enables real-time 246 FPS 360° avatar animation from a single portrait, with substantial industrial utility]