Face Anything: 4D Face Reconstruction from Any Image Sequence¶

Conference: ECCV2026
arXiv: 2604.19702
Code: None
Area: 3D Vision
Keywords: 4D face reconstruction, canonical coordinate prediction, dense tracking, feed-forward network, correspondence estimation

TL;DR¶

The paper proposes Face Anything, unifying 4D face reconstruction and tracking as canonical coordinate prediction—the network jointly estimates depth and canonical face coordinates from arbitrary image sequences, and dense correspondences are obtained directly through nearest-neighbor search in canonical space without explicitly modeling inter-frame motion. It outperforms existing methods across depth estimation (16% lower RMSE), correspondence accuracy (3× improvement), and inference efficiency (32× speedup).

Background & Motivation¶

Reconstructing dynamic facial geometry and establishing dense inter-frame correspondences from arbitrary image sequences are fundamental to digital humans, telepresence, and facial animation. This task comprises two coupled subproblems: recovering high-precision facial geometry for each frame, including details such as wrinkles, hair strands, and the mouth interior, and establishing pixel-level correspondences across frames for tracking. Recent feed-forward 3D reconstruction models, such as DA3, VGGT, and Pi3, can accurately predict depth and ray maps from single or multiple images. However, they do not output explicit correspondences and therefore cannot directly support tracking. Parametric face methods such as P3DMM provide correspondences but are constrained by fixed topologies such as FLAME and cannot represent structures outside that topology, including hair, glasses, and the mouth interior. Meanwhile, motion-field prediction methods exemplified by V-DPM model motion by predicting inter-frame point-cloud deformations. They require multiple forward passes per inference, incur substantial computation, and tend to lose stable tracking under large motions.

The key challenge is that geometry reconstruction and correspondence estimation are treated as separate problems, each with different representations and network architectures, making accuracy and efficiency difficult to achieve together. More importantly, learning motion fields (deformation fields) is fundamentally difficult: deformation targets vary drastically with viewpoint, expression magnitude, and motion complexity, while longer sequences yield sparser learning signals. The face domain, however, has a distinctive property: faces across identities, poses, and expressions share a highly structured canonical space. Mapping each pixel into this shared space would allow inter-frame correspondences to follow naturally from geometric proximity between canonical coordinates, without explicitly estimating motion trajectories.

The paper's approach is to turn tracking into reconstruction in canonical space, since canonical facial geometry remains stable under pose and expression changes. Core Idea: replace inter-frame motion prediction with canonical map prediction—the network outputs both a depth map and a 3D canonical face coordinate map for each frame, and dense correspondences are obtained directly through nearest-neighbor search between the canonical coordinates of two frames. This paradigm simplifies dynamic tracking into canonical-space reconstruction: a single forward pass yields both spatiotemporally consistent geometry and dense tracking, substantially outperforming motion-field methods in accuracy and efficiency.

Method¶

Overall Architecture¶

Face Anything uses a unified feed-forward Transformer to jointly predict three per-pixel maps from image sequences of arbitrary length (1-16 frames): a depth map \(D_i\), a ray map \(R_i\), and a canonical coordinate map \(C_i\) containing a 3D canonical-space coordinate for each pixel. Its backbone inherits DA3's Transformer architecture, with an added DPT-style prediction head for joint outputs. At inference time, dense correspondences between any two frames are obtained through KD-Tree nearest-neighbor search over their canonical coordinate maps, without additional motion modeling or per-frame optimization. For training, existing datasets lack canonical dense correspondence annotations, so the authors construct a dataset from NeRSemble multi-view videos: COLMAP multi-view reconstruction provides high-precision geometry, and FLAME parametric tracking aligns the reconstruction to a shared canonical space, yielding per-pixel supervision for canonical coordinates. Training proceeds in two stages: the backbone is first pretrained on the DAViD monocular face dataset to learn facial geometry priors, followed by joint fine-tuning of the depth and canonical coordinate heads on NeRSemble.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Image sequence<br/>(1-16 frames, monocular/multi-view)"] --> B["Transformer backbone<br/>(DAViD-pretrained weights)"]
    B --> C["DPT joint prediction head<br/>(independent at each time step)"]
    C --> D["Per-frame outputs:<br/>Depth D_i + rays R_i<br/>+ canonical coordinates C_i (3D)"]
    D --> E["Inter-frame canonical coordinate maps<br/>KD-Tree nearest-neighbor search"]
    E --> F["Dense correspondences<br/>(pixel-level tracking trajectories)"]
    D --> G["4D visible-surface reconstruction<br/>(spatiotemporally consistent dynamic geometry)"]
    A --> H["Training data construction pipeline"]
    H --> I["NeRSemble multi-view videos<br/>(16 cameras, 414 subjects)"]
    I --> J["MediaPipe FPS sampling<br/>(50 diverse poses/expressions)"]
    J --> K["COLMAP MVS reconstruction<br/>+ FLAME parametric tracking"]
    K --> L["Deformation transfer: COLMAP points→<br/>FLAME canonical space"]
    L --> M["Canonical coordinate maps (GT)<br/>+ depth maps (GT)"]
    M --> C

Key Designs¶

1. Canonical coordinate maps: turning dense tracking into canonical-space reconstruction

This is the paper's central paradigm shift. Conventional methods formulate dense tracking as motion-field prediction, learning a function \(T: p_i \to p_j\) that maps pixels in frame \(i\) to corresponding pixels in frame \(j\). This mapping changes drastically with facial pose and expression, making it difficult to learn and limiting generalization. The paper takes a different approach: predict a pose- and expression-independent canonical-space coordinate \(C_i(p) \in \mathbb{R}^3\) for each pixel, defined in FLAME's neutral-pose, neutral-expression, metric-scale coordinate system. Inter-frame correspondence reduces to:

\[ \mathbf{q} = \arg\min_{\mathbf{q}' \in \Omega_j} \|C_i(\mathbf{p}) - C_j(\mathbf{q}')\|_2 \]

In other words, KD-Tree nearest-neighbor search between two canonical coordinate maps suffices. This design has three strengths. First, the learning target in canonical space is static: canonical coordinates are similar for any face in any pose, unlike motion fields that vary with the temporal gap. Second, correspondences are implicitly defined, eliminating explicit modeling of motion dynamics; even long-range matching across dozens of frames requires only one query. Third, canonical coordinates encode both semantic correspondence—the nose tip always matches the nose tip—and geometric position, with 3D coordinates directly back-projectable into a 3D point cloud. They therefore provide a natural alignment basis for downstream tasks such as animation and deformation transfer. Experiments show that this design achieves an EPE approximately half that of motion-field prediction.

2. Joint depth–canonical coordinate prediction architecture

The network uses a 1.2B-parameter Transformer backbone extended from DA3. The main modification is to the prediction heads: a canonical coordinate head is added alongside the existing depth head, and both share a DPT-style network structure, independently decoding depth maps \(D_i\), ray maps \(R_i\), and canonical coordinate maps \(C_i\) from the same feature maps. The key motivation for joint prediction is the complementarity between depth and canonical coordinates: if two pixels are close in a 2D image but have different depths, their canonical coordinates must also differ. Depth signals therefore provide important geometric constraints for learning canonical coordinates. Conversely, inter-frame consistency of canonical coordinates supplies a temporal smoothness prior for depth estimation.

Training follows a two-stage strategy. Stage one pretrains the backbone and depth head on DAViD (approximately 100,000 monocular face images) to learn general facial geometry priors and depth perception. Stage two freezes part of the backbone, adds the canonical coordinate head, and jointly fine-tunes all output heads on NeRSemble. An alternating sampling strategy feeds the network multi-view images from the same timestamp to learn multi-view geometric consistency, alternating with images from multiple timestamps of the same camera to learn temporal correspondences. This design effectively narrows the domain gap between multi-view capture data and real monocular videos, allowing the model to perform both reconstruction and tracking well.

3. COLMAP+FLAME-based dataset construction and canonical annotations

The primary obstacle to learning canonical coordinates is the lack of annotated data: no existing dataset provides both per-pixel depth and canonical-space correspondences. The paper builds a complete pipeline for generating canonical coordinate supervision from multi-view videos. The data source is NeRSemble, with synchronized recordings from 16 cameras of 414 subjects covering diverse identities and expressions. Across all sequences of each subject, MediaPipe first estimates facial blendshape parameters and head pose. Farthest point sampling (FPS) then selects the 50 most diverse timestamps, with 40 selected by blendshape differences and 10 by pose differences, ensuring balanced coverage of expression and pose variations.

For each selected timestamp, COLMAP multi-view stereo reconstruction runs on all 16 views to produce high-precision depth maps and dense point clouds; PatchMatchStereo parameters are adjusted to achieve 80.3% facial coverage. FLAME parametric tracking is then performed for each subject to obtain a fitted FLAME mesh for each frame. The key step preserves the fine-grained geometry of the COLMAP point cloud while aligning it to FLAME canonical space: for each reconstructed 3D point, the deformation vector of its nearest FLAME vertex is used to deform it from the current pose into FLAME's neutral-pose, neutral-expression canonical coordinates. The resulting canonical point cloud preserves COLMAP's high-frequency geometric details, including hair strands, glasses, and the mouth interior, while providing semantically consistent correspondences across identities. Finally, back-projecting the canonical point cloud onto the image plane yields per-pixel canonical coordinate supervision. Notably, although FLAME tracking has small errors, the network compensates for this alignment noise during training and learns richer structures outside the FLAME topology than the parametric model can represent.

Loss & Training¶

For depth \(D\), rays \(R\), and canonical coordinates \(C\), each prediction \(X\) and its ground truth \(X^*\) are supervised with a combination of three losses:

Regression loss \(\mathcal{L}_{\text{reg}}^X = \frac{1}{|\Omega|}\sum_{\mathbf{p}\in\Omega} |X(\mathbf{p})-X^*(\mathbf{p})|\): L1-driven learning
Confidence-weighted loss \(\mathcal{L}_{\text{conf}}^X = \frac{1}{|\Omega|}\sum_{\mathbf{p}\in\Omega} (\gamma|X(\mathbf{p})-X^*(\mathbf{p})|W_X(\mathbf{p}) - \alpha\log W_X(\mathbf{p}))\): the network also predicts per-pixel confidence \(W_X\) to adaptively downweight uncertain regions
Gradient loss \(\mathcal{L}_{\text{grad}}^X = \frac{1}{|\Omega|}\sum_{\mathbf{p}\in\Omega} (|\nabla_x E_X(\mathbf{p})|+|\nabla_y E_X(\mathbf{p})|), E_X = X-X^*\): enforces spatial smoothness of predictions

The total loss is \(\mathcal{L} = \lambda_C(\mathcal{L}_{\text{reg}}^C + \mathcal{L}_{\text{conf}}^C + \mathcal{L}_{\text{grad}}^C) + \sum_{X\in\{D,R\}}(\mathcal{L}_{\text{reg}}^X + \mathcal{L}_{\text{conf}}^X + \mathcal{L}_{\text{grad}}^X)\), where \(\lambda_C=5\) gives the canonical coordinate head greater weight because training it from scratch requires more gradient signal. Training uses AdamW, with cosine annealing of the learning rate from \(2\times10^{-5}\) to \(1\times10^{-8}\), bfloat16 precision and gradient checkpointing to reduce memory usage, and 90 epochs of 800 batches each, with up to 48 images per batch.

Key Experimental Results¶

Main Results¶

Task	Dataset	Metric	Ours	Prev. SOTA	Gain
Monocular depth estimation(AbsRel×10)	NeRSemble	Image/video	0.040/0.038	Sapiens-2B 0.048/0.048	17-21%
Monocular depth estimation(AbsRel×10)	Ava-256	Image/video	0.048/0.048	Pi3 0.066/0.071	27-32%
2D dense correspondence(EPE↓,without hair)	NeRSemble	Margin=2	1.719	P3DMM 3.089	44%
2D dense correspondence(EPE↓,with hair)	NeRSemble	Margin=2	1.838	P3DMM 5.550	67%
Cycle consistency(CCE_mean↓)	VFHQ	Margin=5	0.398	V-DPM 0.797	50%
Cycle consistency(CCE_mean↓)	VFHQ	Margin=20	0.774	V-DPM 1.348	43%
3D tracking EPE↓	NeRSemble	Margin=1-10	0.005	V-DPM 0.015	67%
Inference efficiency	40 images	Runtime(s)↓	5	V-DPM 160	32×
Peak inference memory	Single GPU	Memory(GB)↓	19	V-DPM 40	53%

Ablation Study¶

Config	NeRSemble AbsRel(monocular)↓	EPE(2D)↓	NeRSemble AbsRel(16 views)↓	Camera Rot↓
DA3 baseline (without canonical head)	0.085	-	0.076	-
+DAViD-pretrained backbone	0.061	-	0.033	-
Monocular training (fixed view, varying time)	0.054	3.031	0.064	0.038
Static training (fixed time, varying views)	0.050	3.864	0.015	2.102
Motion-field prediction instead of canonical coordinates	-	6.210	-	-
Ours: full design	0.053	3.271	0.015	0.054

Key Findings¶

The fundamental gap between canonical coordinates and motion fields: in the ablation, the variant explicitly predicting inter-frame motion fields (Ours Motion Pred) reaches an EPE of 6.21, nearly twice that of canonical coordinate prediction (3.27), directly supporting the central argument that canonical-space reconstruction is easier to learn than motion-field prediction.
Joint optimization of monocular and multi-view depth: monocular training, which varies timestamps with a fixed view, performs well on monocular depth (AbsRel 0.054) but poorly on multi-view depth (0.064). Static training, which varies views at a fixed timestamp, does the opposite: good multi-view depth (0.015) but poor monocular depth (0.050). The final design alternates the two sampling modes, trading a small sacrifice in monocular depth for balanced optimal performance across all metrics.
The importance of canonical coordinate weighting: with \(\lambda_C=1\), giving it the same weight as the depth head, EPE reaches 4.03; increasing it to \(\lambda_C=5\) reduces EPE to 3.27. Raising it further to 10 slightly lowers EPE but harms depth accuracy, indicating that \(\lambda_C=5\) is the best trade-off.
A major inference efficiency advantage: on 40 images, V-DPM requires 160 seconds+40GB of memory, whereas this method needs only 5 seconds+19GB, delivering 32× faster inference and 53% memory savings. Its maximum batch size on a single GPU reaches 470 images, compared with only 74 for V-DPM. This advantage comes from avoiding the repeated forward passes for frame-by-frame tracking required by motion-field methods.
Spatiotemporal consistency for smooth tracking: for long-range correspondence (Margin=20), the method achieves a CCE_median of only 0.069, versus V-DPM's 1.054 and P3DMM's 1.146. This indicates highly consistent semantic correspondences over long timescales, with almost no cycle inconsistency.

Highlights & Insights¶

The simplicity of the "tracking as reconstruction" paradigm: reducing tracking to reconstruction is the most elegant aspect. No motion modeling, multi-step propagation, or test-time optimization is needed: after one feed-forward pass produces canonical coordinates, nearest-neighbor search completes tracking. Using an invariant representation to handle dynamic variation is particularly valuable in dynamic scenes.
Combining COLMAP detail preservation with FLAME cross-identity alignment: COLMAP alone recovers high-frequency geometry but cannot establish cross-frame or cross-identity correspondences; FLAME alone provides alignment but loses details outside its topology. Deformation transfer combines both, retaining COLMAP's hair, glasses, and mouth details while obtaining a semantically aligned canonical space. This data construction strategy itself has substantial transfer potential.
Implicit long-range consistency of canonical coordinates: even across a gap of 20 frames, nearest-neighbor matching of canonical coordinates remains highly accurate (CCE_median=0.069). The learned canonical space is therefore inherently robust to large expression changes and long-term drift, which motion-field methods struggle to achieve.
Lightweight, efficient engineering: a 1.2B-parameter model with a DPT head takes 5 seconds to process 40 frames at inference time, with KD-Tree matching taking less than 0.2 seconds per pair and memory usage of only 19GB. The model design addresses both academic metrics and practical deployment needs.

Limitations & Future Work¶

Domain restriction: the method is specifically designed for faces and relies on facial geometry priors, so it cannot generalize to nonfacial scenes. Canonical coordinates for nonfacial objects near the face, such as microphones, hands, and glasses, are unreliable and can cause incorrect matches.
Degradation under extreme conditions: reconstruction quality declines under severe occlusion, extreme viewpoints such as side views where the face is barely visible, and large hair motions; canonical coordinate predictions become unstable.
Insufficient fine-grained mouth tracking: although the method covers the mouth interior, it cannot yet track individual teeth and the tongue, which would require higher-resolution supervision of internal structures.
Future directions: the paper proposes two promising extensions: generalizing canonical coordinate prediction to broader nonrigid categories such as humans and animals, and combining canonical coordinates with generative models such as conditional diffusion and neural rendering to directly drive controllable facial animation and digital human generation from monocular videos.

vs DA3 (Depth Anything 3): DA3 provides the backbone, predicting depth+ray maps but not canonical coordinates and therefore not supporting tracking. The paper adds a canonical coordinate head and joint training strategy to DA3, enabling the same network to output both geometry and correspondences.
vs V-DPM (Dynamic Point Maps): V-DPM performs dynamic reconstruction and tracking by predicting inter-frame point-cloud deformation fields. It needs multiple forward passes, one per frame pair, resulting in high computation and unstable long-range tracking. Canonical coordinates require only one forward pass, reducing tracking to nearest-neighbor search and outperforming V-DPM in EPE (67% reduction), inference speed (32×), and memory usage (53% reduction).
vs P3DMM (Pixel3DMM): P3DMM predicts 2D UV coordinates on the FLAME surface. Its fixed topology cannot represent non-FLAME regions such as hair, glasses, and the mouth interior. This method predicts nonparametric 3D canonical coordinates without topology constraints, reducing EPE in regions including hair from 5.55 to 1.84 (67% lower).
vs DAViD / Sapiens: these methods provide high-quality single-frame facial geometry estimates but process frames independently, without an inter-frame consistency mechanism. The shared canonical coordinate space naturally ensures temporal consistency without additional temporal post-processing.
vs 3DGS head avatar methods: unlike methods such as GaussianAvatars that require hours of per-sequence optimization, this method performs tracking without test-time optimization (27 seconds vs ∼5-12 hours), while achieving lower EPE (5.20 vs 7.01-7.21).

Rating¶

Novelty: ⭐⭐⭐⭐⭐ [Explicitly turns tracking into canonical-space reconstruction, a clear paradigm shift and the first feed-forward unified reconstruction+tracking approach in 4D faces]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Across four datasets—NeRSemble, Ava-256, VFHQ, and CelebV-HQ—evaluates depth, 2D tracking, 3D tracking, efficiency, and ablations in 9 experimental tables; ablations cover key dimensions including canonical coordinates vs motion fields, backbone freezing, and loss weights]
Writing Quality: ⭐⭐⭐⭐ [Clear structure, explicitly identified comparison methods, and progressively developed motivation; paradigm naming and baseline comparisons in the ablations could be more direct]
Value: ⭐⭐⭐⭐⭐ [The two-stage training pipeline+dataset construction+SOTA results offer substantial engineering and academic value; the canonical coordinate paradigm could extend to general 4D, and its high inference efficiency supports practical deployment]