FFAvatar: Feed-Forward 4D Head Avatar Reconstruction from Arbitrary Images

Conference: ECCV 2026
Paper: ECCV 2026 Poster
Code: https://jj-yao.github.io/ffavatar/
Area: 3D Vision
Keywords: 4D head avatar reconstruction, feed-forward Gaussian splatting, FLAME prior, sparse-to-dense, motion-aware refinement

TL;DR

FFAvatar presents a feed-forward 4D head avatar reconstruction framework that incrementally reconstructs high-fidelity, controllable 3D Gaussian avatars from one or multiple arbitrary portrait images using alternating attention for identity disentanglement, a sparse-to-dense canonical alignment strategy, and a motion-aware dynamic refinement module.

Background & Motivation

High-fidelity 4D head avatar modeling plays a foundational role in virtual reality, digital humans, telepresence, and immersive content creation. An ideal 4D avatar system must preserve personal identity across diverse head poses and expressions while maintaining temporal smoothness and geometric fidelity. Traditional neural head modeling pipelines predominantly rely on per-subject optimization over dense multi-view captures. When deployed under sparse or in-the-wild observations, these methods suffer from severe performance degradation, slow convergence, and poor generalization to unseen expressions.

Feed-forward 3D reconstruction offers an attractive alternative to bypass expensive per-instance optimization. However, existing feed-forward approaches face several structural trade-offs. First, single-reference models suffer from incomplete geometry and ambiguous textures when rotated to unobserved angles, whereas recent few-shot methods typically construct multiple view-dependent canonical subspaces. Such multi-canonical formulations result in linear proliferation of 3D Gaussian primitives and GPU memory as the number of input views increases. Second, facial appearance and dynamic motion remain tightly coupled in image representations, leading to identity drift, ghosting artifacts, and blurred expressions during novel animation. Third, optimizing dense UV representations directly incurs excessive computational costs, whereas relying solely on coarse parametric templates fails to reconstruct high-frequency details such as hair strands and wrinkles.

To resolve these conflicts, this work proposes a progressive feed-forward paradigm grounded on a unified global canonical field. Core idea: disentangle identity appearance from expression and pose variations across arbitrary reference images via an alternating attention mechanism, build a compact global canonical 3D Gaussian field using sparse-to-dense cross-modal alignment on the FLAME template, and introduce a motion-aware refinement network to restore fine-grained, identity-dependent dynamic deformation.

Method

Overall Architecture

The FFAvatar framework operates in two primary stages: Static Appearance Canonical Field Generation and 3D Head Avatar Animation. In the first stage, given one or more portrait images, a frozen DINOv3 backbone extracts dense multi-scale feature maps. These features are augmented with camera pose and expression encodings and processed by an alternating attention module that interleaves intra-image and inter-image attention to synthesize a global appearance representation invariant to viewpoints and expressions. In the canonical space, sparse FLAME template vertices and rasterized dense UV coordinates undergo two-stage sparse-to-dense cross-modal alignment. A feed-forward decoder then outputs the complete attributes of \((M+5023)\) canonical 3D Gaussians. In the animation stage, facial motion is driven by linear blend skinning (LBS) on FLAME, followed by the Motion-Aware Refinement Module (MARM), which predicts residual Gaussian attribute offsets conditioned on facial expression, head pose, and camera parameters to capture non-linear muscle dynamics.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Arbitrary Reference Portraits<br/>Single or multiple images"] --> B["DINOv3 Feature Extraction<br/>Concatenate camera and expression tokens"]
    B --> C["Alternating Attention Mechanism<br/>Disentangle identity from pose/expression"]
    C --> D["Sparse-to-Dense Cross-Modal Alignment<br/>FLAME vertices and UV rasterization"]
    D --> E["Canonical 3D Gaussian Field Decoder<br/>Outputs (M+5023) Gaussian primitives"]
    E --> F["FLAME Coarse Deformation<br/>Linear blend skinning LBS"]
    F --> G["Motion-Aware Refinement Module MARM<br/>Predict non-linear motion residuals"]
    G --> H["High-Fidelity 4D Avatar Rendering<br/>Real-time 3D Gaussian splatting"]

Key Designs

1. Alternating Attention Aggregation: Disentangling Identity from Expression and Viewpoint

Extracting a clean, identity-specific appearance representation from an arbitrary number of unconstrained reference images requires isolating static features from lighting variations, transient expressions, and perspective distortions. FFAvatar augments visual features \(I_i^{\text{feat}} \in \mathbb{R}^{H \times W \times C}\) from DINOv3 with camera pose embeddings \(\mathbf{Z}_{\text{cam}}\) and expression embeddings \(\mathbf{Z}_{\text{exp}}\), passing them through an alternating attention architecture. This structure alternates between intra-image self-attention (to preserve spatial semantics) and inter-image cross-attention (to match shared identity cues across disparate frames). The module effectively aggregates multi-view identity features into a unified appearance representation \(\mathbf{Z}^{\text{app}} \in \mathbb{R}^{NHW \times C}\) without constructing view-dependent canonical subspaces, ensuring linear scalability as reference images increase from 1 to 32.

2. Sparse-to-Dense Hierarchical Alignment: Anchoring Global Canonical 3D Gaussians

Directly regressing dense 3D Gaussians across the entire head causes severe optimization instability, excessive memory usage, and GPU out-of-memory errors during training. Conversely, anchoring Gaussians only to sparse template vertices misses fine geometric and textural details. FFAvatar solves this with a two-stage sparse-to-dense cross-modal alignment strategy. First, 5,023 canonical FLAME template vertices \(\mathbf{V}_0 \in \mathbb{R}^{5023 \times 3}\) act as coarse geometric anchors, querying \(\mathbf{Z}^{\text{app}}\) via cross-attention to produce sparse structural features \(\mathbf{T}_s\). Second, the mesh is unwrapped into an \(uv\_size \times uv\_size\) UV grid, interpolating \(\mathbf{T}_s\) across valid UV coordinates via barycentric coordinates \((\alpha, \beta, \gamma)\) of intersecting triangles to yield dense UV features \(\mathbf{T}_{\text{uv}} \in \mathbb{R}^{M \times C}\). Concatenating sparse vertex tokens and dense UV tokens, a second-stage cross-attention alignment refines the multi-resolution representation:

\[\mathbf{T}_d = \text{CrossAtt}([\mathbf{T}_s; \mathbf{T}_{\text{uv}}], \mathbf{Z}^{\text{app}})\]

The feed-forward static decoder \(\mathcal{D}_{\text{static}}\) then directly predicts the complete attribute set of \((M+5023)\) Gaussian primitives: offset position \(\mathbf{f}_n \in \mathbb{R}^3\), opacity \(o_n \in \mathbb{R}\), spherical harmonic color \(\mathbf{c}_n \in \mathbb{R}^3\), anisotropic scale \(\mathbf{s}_n \in \mathbb{R}^3\), and rotation quaternion \(\mathbf{r}_n \in \mathbb{R}^4\).

3. Motion-Aware Refinement Module (MARM): Non-Linear Subject-Specific Dynamic Personalization

While FLAME linear blend skinning (LBS) and corrective blendshapes provide reliable rigid head motion and coarse facial articulation, parametric blendshapes cannot faithfully capture personalized non-linear deformation patterns, such as subtle skin folds, dimples, and eyelid micro-movements. FFAvatar introduces MARM (\(\mathcal{R}_{\text{motion}}\)), a lightweight Transformer network taking canonical FLAME vertex encodings \(\text{PE}(\mathbf{V}_0)\), target motion parameters \(\boldsymbol{\theta} = \{\boldsymbol{\theta}_{\text{pose}}, \boldsymbol{\theta}_{\text{exp}}\}\), and camera parameters \(\boldsymbol{\theta}_{\text{cam}}\) as inputs. Instead of predicting full Gaussian parameters per frame, it predicts compact residual updates \(\Delta \mathbf{g}\):

\[\Delta \mathbf{g} = \mathcal{R}_{\text{motion}}(\text{PE}(\mathbf{V}_0), \boldsymbol{\theta}, \boldsymbol{\theta}_{\text{cam}})\]

The final animated field is obtained via attribute-wise addition: \(G(\boldsymbol{\theta}) = G' \oplus \Delta \mathbf{g}\), where \(G'\) denotes the FLAME-deformed coarse field. This residual design prevents identity degradation, accurately renders dynamic facial details, and can be bypassed in high-throughput environments to achieve an ultra-fast rendering speed of 468 FPS.

Loss & Training

The framework is implemented in PyTorch and optimized with the Adam optimizer at a learning rate of \(4.0 \times 10^{-5}\) across 300,000 iterations. The DINOv3 backbone remains frozen, while all attention, alignment, decoding, and motion modules are optimized end-to-end. Training batches randomly sample between 1 and 8 frames from monocular video sequences. Head regions are tracked, cropped, and resized to \(512 \times 512\) using the GAGAvatar pipeline, with automated background removal applied.

Key Experimental Results

Main Results

Quantitative novel expression synthesis evaluation on the VFHQ monocular test set comparing against one-shot and few-shot state-of-the-art baselines:

Method	Setting	PSNR ↑	SSIM ↑	LPIPS ↓	CSIM ↑	AED ↓	APD ↓	Creation Time ↓	Rendering FPS ↑
Real3D-Portrait	One-shot	20.88	0.780	0.154	0.750	0.150	0.268	3.5s	15
Portrait4D-v2	One-shot	21.34	0.794	0.144	0.717	0.117	0.187	2.9s	11
GAGAvatar	One-shot	21.83	0.818	0.128	0.816	0.111	0.135	1.6s	63
LAM	One-shot	22.65	0.829	0.109	0.822	0.102	0.134	1.1s	219
FastAvatar	One-shot	17.85	0.813	0.167	0.679	0.136	0.328	2.6s	339
FFAvatar (Ours)	One-shot	21.82	0.843	0.108	0.817	0.109	0.149	1.3s	31
GPAvatar	Few-shot	22.91	0.795	0.154	0.765	0.138	0.189	0.7s	5
FastAvatar	Few-shot	18.12	0.819	0.153	0.781	0.116	0.321	12.2s	97
FFAvatar (fast)	Few-shot	23.20	0.862	0.088	0.852	0.084	0.117	2.1s	468
FFAvatar (Full)	Few-shot	23.35	0.864	0.081	0.861	0.079	0.114	2.1s	31

Cross-dataset quantitative generalization results on the NeRSemble multi-view dataset under novel-view (8 input views, 8 unseen views) and novel-expression settings:

Method	Setting	Novel-View LPIPS ↓	Novel-View SSIM ↑	Novel-View PSNR ↑	Novel-Expr LPIPS ↓	Novel-Expr SSIM ↑	Novel-Expr PSNR ↑
Portrait4D-v2	One-shot	0.172	0.797	16.81	0.152	0.814	18.24
GAGAvatar	One-shot	0.129	0.833	22.52	0.095	0.857	25.87
LAM-20K	One-shot	0.175	0.819	16.43	0.122	0.834	20.55
FFAvatar (Ours)	One-shot	0.121	0.839	19.18	0.106	0.851	20.23
FlashAvatar	Per-subject	0.209	0.785	17.84	0.221	0.764	16.94
GaussianAvatars	Per-subject	0.164	0.813	17.99	0.178	0.822	17.56
GPAvatar	Few-shot	0.163	0.822	22.26	0.154	0.829	22.58
FastAvatar	Few-shot	0.158	0.824	20.11	0.135	0.845	22.49
FFAvatar (Full)	Few-shot	0.098	0.858	21.95	0.075	0.881	24.08

Ablation Study

Sparse-to-Dense Canonical Generation Strategy (evaluated over 1,000 training iterations under a 40 GB GPU memory cap):

Config	UV Resolution	PSNR ↑	SSIM ↑	LPIPS ↓	CSIM ↑	AED ↓	1,000-iter Time ↓	GPU Memory ↓	Note
S-5K	-	19.69	0.837	0.174	0.689	0.131	40 min	22.5 GB	Coarse 5,023 vertices only; severe loss of textures
D-20K	128	23.41	0.869	0.097	0.858	0.087	120 min	38.2 GB	Direct dense point cloud optimization; unstable convergence
S2D-20K (Default)	128	23.35	0.864	0.081	0.861	0.079	47 min	25.8 GB	Balanced fidelity, stable deformation, low memory
S2D-64K	256	23.42	0.871	0.077	0.866	0.078	72 min	37.1 GB	Higher UV resolution with sharper high-frequency details
D-64K	256	-	-	-	-	-	-	OOM	Direct dense optimization overflows 40 GB GPU memory

Impact of Incremental Reference Frame Count:

Input Frame Count	PSNR ↑	SSIM ↑	LPIPS ↓	CSIM ↑	AED ↓	Creation Time ↓	Note
1 View (One-shot)	21.82	0.843	0.108	0.817	0.109	1.3s	Plausible appearance with minor occlusion artifacts
4 Views	22.75	0.858	0.091	0.852	0.094	1.7s	Significant gain in identity and structural stability
8 Views (Default)	23.35	0.864	0.081	0.861	0.079	2.1s	Sweet spot for quality and latency
16 Views	23.36	0.866	0.080	0.872	0.078	4.3s	Smoother surface normal and view transition
32 Views	23.38	0.867	0.077	0.874	0.078	11.6s	Incremental convergence with minimal overhead

Key Findings

• Seamless Incremental Scaling: Increasing input observations from 1 to 8 frames yields the most pronounced performance jump (PSNR rises from 21.82 to 23.35 and LPIPS drops from 0.108 to 0.081), verifying the power of the alternating attention module in aggregating multi-view cues without inflating Gaussian primitive counts.
• Overcoming the Memory Barrier: The S2D paradigm eliminates the OOM bottlenecks seen in direct dense point optimization. Under \(256 \times 256\) UV resolution, D-64K crashes due to OOM, while S2D-64K consumes only 37.1 GB memory. Furthermore, S2D-20K cuts 1,000-iteration training time from 120 minutes down to 47 minutes.
• Plug-and-Play Dynamics: Removing MARM boosts rendering throughput to 468 FPS (FFAvatar fast) while preserving primary identity geometry, offering practical flexibility between real-time mobile deployment and full-fidelity dynamic synthesis.

Highlights & Insights

• Unified Global Canonical Field: Eliminates the redundant multi-canonical representations common in prior few-shot methods, keeping the overall 3D Gaussian count constant regardless of the number of reference images.
• Hierarchical Sparse-to-Dense Feature Injection: Anchoring to FLAME vertices ensures rigid topological stability, while subsequent UV densification restores high-frequency texture and hair details, striking an optimal trade-off between fidelity and training efficiency.
• Synergy with Diffusion Priors: When only a single image or text prompt is provided, FFAvatar can be coupled with multi-view diffusion models (e.g., MMDM) to synthesize multi-angle reference views and reconstruct complete 4D avatar assets.

Limitations & Future Work

• Extreme Angle Artifacts: Because training is conducted on monocular video (VFHQ) where viewpoints are predominantly frontal, large profile angles and extreme head pitches can exhibit slight geometric blurring or texture stretching around the jawline and neck.
• Tracking Dependency: The feed-forward pipeline depends on external tracking for camera poses and FLAME coefficients. Tracking jitter or inaccurate parameter fitting directly propagates into rendering artifacts.
• Future Directions: Integrating pose tracking and expression regression directly into the end-to-end Transformer architecture would eliminate external dependencies and provide unified end-to-end optimization.

Related Work & Insights

• vs GAGAvatar / LAM-20k: Existing one-shot Gaussian avatar methods lack incremental expansion capabilities, causing severe artifacts when synthesizing unobserved novel angles. FFAvatar seamlessly accommodates 1 to 32 frames to progressively refine visual fidelity.
• vs GPAvatar / FastAvatar: GPAvatar and FastAvatar suffer from linearly increasing Gaussian counts and plummeting inference FPS as more views are added. FFAvatar maintains a fixed Gaussian primitive count (\((M+5023)\)), sustaining 31 FPS with full motion refinement and up to 468 FPS in fast mode.

Rating

• Novelty: ⭐⭐⭐⭐ [The combination of alternating attention for multi-image appearance aggregation and S2D canonical Gaussian alignment effectively solves the primitive bloating problem in few-shot reconstruction.]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Extensive evaluations across VFHQ and NeRSemble benchmarks, covering novel-view and novel-expression synthesis, resolution ablations, and frame scaling studies.]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear structural organization, rigorous motivation, well-defined mathematical formulations, and thorough qualitative analysis.]
• Value: ⭐⭐⭐⭐⭐ [Provides a practical, scalable, and high-efficiency paradigm for 4D digital human generation from arbitrary portrait inputs.]