FlexiAvatar: Unified 3D Gaussian Human Avatars Under Arbitrary Body Visibility

Conference: ECCV 2026
Paper: ECCV 2026 Official
Area: 3D Vision
Keywords: 3D Gaussian Splatting, Human Avatar Reconstruction, Visibility-Aware Optimization, Arbitrary Body Visibility, SMPL-X

TL;DR

Addressing the severe geometric distortion and texture drift caused by the full-body visibility assumption in existing monocular human avatar methods, FlexiAvatar introduces the first unified 3D Gaussian avatar reconstruction framework across arbitrary body visibility, combining rasterization visibility-aware pruning to block spurious gradient leakage with occlusion-robust SMPL-X tracking, part-specific residual refinement, and diffusion-based 360° rotation completion to outperform both dedicated and generalist baselines across full-body, upper-body, and head-only settings.

Background & Motivation

Reconstructing high-fidelity, animatable 3D human avatars from monocular video is a cornerstone technology for AR/VR, virtual production, telepresence, and digital entertainment. Recently, 3D Gaussian Splatting (3DGS) has rapidly emerged as the primary representation for human avatar modeling, offering real-time differentiable rendering and explicit geometric control that surpass the computational bottlenecks of neural radiance fields (NeRF). However, representative state-of-the-art 3DGS avatar frameworks—such as GART, GaussianAvatar, ExAvatar, and Vid2AvatarPro—uniformly operate under a rigid assumption: the training video must maintain complete and continuous full-body visibility. In unconstrained real-world settings such as video conferencing, talk shows, news broadcasts, and mobile social media, inputs span an arbitrary visibility spectrum featuring extreme cropping, upper-body framing, and head-only close-ups.

When confronted with partial-body visibility or severe occlusions, existing methodologies suffer from a fundamental architectural breakdown. Full-body methods blindly bind 3D Gaussians to all vertices of a parametric template (e.g., SMPL-X) and optimize all body parts simultaneously; in the absence of visual evidence for unobserved limbs (such as legs and feet), this optimization introduces severe ghost artifacts and geometric hallucinations. Worse still, because Gaussian attributes are decoded from a shared triplane representation, gradients from observed regions propagate through the shared latent space into unobserved Gaussians, indirectly contaminating visible torso and facial textures. Conversely, specialized partial-body approaches—such as head-only models (RGBAvatar, InsTaG, FATE) or upper-body pipelines (GUAVA)—are architecturally confined to fixed camera framings, incapable of extending to full-body animation or handling dynamic zoom. This fragmentation forces practitioners to maintain separate, purpose-built pipelines for distinct visibility regimes.

The core tension lies in how to ground Gaussian optimization strictly within regions supported by empirical visual evidence using a single, unified parametric foundation, while completing unobserved viewpoints without inducing hallucination or corrupting observed geometry. Core idea: FlexiAvatar proposes a unified 3D Gaussian human avatar framework governed by visibility-aware optimization, which restricts updates strictly to observed Gaussians via rasterization evidence statistics and Otsu adaptive thresholding, integrates occlusion-robust confidence-masked SMPL-X tracking, incorporates part-specific spatiotemporal residual refinement, and synthesizes 360° rotation auxiliary videos via video diffusion to achieve state-of-the-art avatar reconstruction across full-body, upper-body, and head-only settings within a single pipeline.

Method

Overall Architecture

FlexiAvatar reconstructs an animatable 3D Gaussian human avatar from monocular video under arbitrary visibility. The framework first performs confidence-masked SMPL-X tracking and face geometry alignment to extract a robust personalized canonical template, while employing a pose-guided video diffusion model (MimicMotion) to generate 360° rotating auxiliary videos that complete textures for consistently unobserved regions (such as the back). Canonical mesh vertices are associated with 3D Gaussians and mapped via orthogonal projection into a learnable triplane feature field to decode static properties and pose-dependent deformations. Following a warm-up phase, visibility-aware optimization computes per-Gaussian visibility rates from rasterization passes, automatically selecting an Otsu threshold to prune unobserved Gaussians and mask their contribution to reconstruction and regularization losses. Meanwhile, part-specific spatiotemporal residual MLPs are activated for the face and hands to capture high-frequency dynamic details. Finally, deformed Gaussians are posed via SMPL-X linear blend skinning (LBS) and rendered into novel views using 3DGS.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Monocular Video Input<br/>(Arbitrary Visibility)"] --> B["Occlusion-Robust SMPL-X Tracking<br/>Confidence-masked keypoints & FLAME face alignment"]
    A --> C["Diffusion-Driven Video Generation<br/>MimicMotion generates 360° rotation for unseen textures"]
    B --> D["Canonical Hybrid Mesh-Gaussian Representation<br/>Triplane orthogonal projection decodes static & dynamic offsets"]
    C --> E["Visibility-Aware Gaussian Optimization<br/>Post-warmup Otsu adaptive pruning & dual loss masking"]
    D --> E
    E --> F["Part-Specific Residual Refinement<br/>Specialized lightweight MLPs compensate high-frequency details"]
    F --> G["SMPL-X Linear Blend Skinning & 3DGS Rendering<br/>Outputs high-fidelity animatable human avatar"]

Key Designs

1. Occlusion-Robust SMPL-X Tracking: Eliminating Out-of-Frame Topological Corruption In partial-body videos, off-the-shelf whole-body pose estimators generate erroneous predictions for out-of-frame joints (e.g., occluded lower limbs). Direct supervision using all keypoints distorts the underlying parametric skeleton and degrades the canonical template. To prevent this, FlexiAvatar incorporates a per-joint binary visibility mask \(\mathbf{m}_j \in \{0, 1\}\) based on detector confidence scores with a threshold \(\tau = 0.4\), restricting the 2D keypoint reprojection loss strictly to well-localized, visible joints: $$ \mathcal{L}{\text{kpt}} = \sum_j \right|_1 $$ where } \mathbf{m}_j \left| \Pi\big(V_j(\boldsymbol{\theta}, \boldsymbol{\beta}, \Delta \mathbf{J})\big) - \mathbf{K\(\Pi\) denotes camera perspective projection, \(V_j\) is the 3D joint position derived from pose \(\boldsymbol{\theta}\), shape \(\boldsymbol{\beta}\), and learnable joint offsets \(\Delta \mathbf{J}\), and \(\mathbf{K}_j\) is the target 2D keypoint. To faithfully capture fine-grained identity, the SMPL-X facial region is aligned with a pre-fitted FLAME template through a composite geometric objective \(\mathcal{L}_{\text{sface}}\) (comprising vertex L1 distance, mesh Laplacian curvature smoothness, and edge length conservation), regularized by shape and symmetry terms \(\mathcal{L}_{\text{sreg}}\). By employing SMPL-X as the universal parametric body model across all visibility regimes, including head-only captures, FlexiAvatar eliminates the need to maintain separate facial models (e.g., FLAME) in head-only scenarios.

2. Visibility-Aware Gaussian Optimization: Blocking Spurious Gradient Leakage Across Shared Features In hybrid triplane-Gaussian representations, all Gaussian properties are decoded from shared continuous feature planes. When unobserved body parts participate in forward rasterization or backward propagation, gradients originating from visible pixels flow through the shared triplane into unobserved Gaussians, leaking spurious gradients that contaminate neighboring visible regions. To resolve this, FlexiAvatar tracks empirical visibility evidence by computing the per-Gaussian visibility rate: $$ v_i = \frac{1}{F} \sum_{f=1}^{F} v_i^f $$ where \(v_i^f\) is the binary rasterization visibility flag for Gaussian \(i\) at frame \(f\), and \(F\) is the total number of frames. During an initial 2,000-iteration warm-up phase, all Gaussians are optimized without filtering to stabilize coarse geometry. After warm-up, Otsu's method models the distribution \(\{v_i\}\) as a bimodal histogram, automatically selecting the optimal threshold \(\tau^*\) that minimizes intra-class variance between visible and invisible Gaussians. Gaussians with \(v_i < \tau^*\) are excluded from forward rasterization, and their contributions to reconstruction and Laplacian smoothness regularization losses are zeroed out. This design eliminates gradient leakage at its root, while reducing the active Gaussian count by 13.8% to 49.7% in upper-body and head-only settings, cutting memory footprint and boosting rendering frame rates.

3. Part-Specific Residual Refinement: Spatiotemporal Appearance Compensation for Expressive Regions Because global triplane feature interpolation tends to over-smooth high-frequency dynamic appearance, such as subtle facial expressions and finger crease variations, FlexiAvatar introduces a localized implicit residual refinement module. Three lightweight MLPs are allocated specifically to the face, left hand, and right hand. Given a Gaussian center \(\boldsymbol{\mu} \in \mathbb{R}^3\), timestamp \(t\), and local skeletal pose \(\boldsymbol{\theta}\), spatiotemporal features are constructed via positional encoding \(\gamma(\cdot)\). Each specialized MLP is activated only when its corresponding part is visible according to the visibility mask, predicting dynamic color offsets \(\Delta \mathbf{C}_{\text{res}}(\gamma(\boldsymbol{\mu}), \gamma(t), \boldsymbol{\theta})\). These residual MLPs are engaged after 2,000 iterations once coarse geometry has converged, yielding the final color: $$ \mathbf{C}{\text{final}} = \mathbf{C}}} + \Delta \mathbf{C{\text{pose}} + \Delta \mathbf{C} $$ This decouples global pose articulation from localized high-frequency texture variation, allowing the avatar to render crisp, expressive facial details and articulated hands.}

4. Diffusion-Driven Generative Video Completion: Synthesizing Omnidirectional Appearance Without Hallucination Monocular captures inherently suffer from unobserved viewpoints, particularly the subject's back. Applying 2D diffusion distillation (such as SDS) directly within the 3D optimization loop often causes severe multi-face artifacts and geometric collapse on partial-body inputs. Instead, FlexiAvatar employs a pre-trained pose-guided video diffusion model (MimicMotion) conditioned on a single source frame to synthesize an auxiliary video of the subject executing a complete 360° rotation. This synthetic sequence provides explicit visual evidence for previously unobserved viewpoints. During training, captured frames and synthetic frames are sampled with equal probability, and the reconstruction loss on synthetic frames is down-weighted to prevent generation artifacts from compromising observed fidelity.

Loss & Training

The framework is trained end-to-end on a single NVIDIA RTX A6000 GPU for 30,000 iterations using the Adam optimizer with a batch size of 1 and a base learning rate of \(1 \times 10^{-3}\). The total training objective is formulated as: $$ \mathcal{L} = \lambda_{\text{L1}} \mathcal{L}1 + \lambda}} \mathcal{L{\text{ssim}} + \lambda}} \mathcal{L{\text{lpips}} + \lambda}} \mathcal{L{\text{face}} + \lambda $$ Here, }} \mathcal{L}_{\text{reg}\(\mathcal{L}_1\), \(\mathcal{L}_{\text{ssim}}\), and \(\mathcal{L}_{\text{lpips}}\) supervise cropped foreground human regions to penalize pixel-level deviation, structural distortion, and perceptual blur. A dedicated facial consistency loss \(\mathcal{L}_{\text{face}} = \|I_{\text{mesh}} - I_{\text{gt}}\|_1\) renders an average UV texture via a differentiable mesh renderer to regularize facial Gaussian stability across varying expressions. The regularization term \(\mathcal{L}_{\text{reg}}\) enforces Laplacian smoothness on Gaussian positions, scales, and colors, masked by the binary visibility flag \(\mathbf{m}_i\) to ensure that smoothing is strictly applied to well-observed primitives.

Key Experimental Results

Main Results

FlexiAvatar was evaluated across three distinct body visibility regimes: full-body (NeuMan, ZJU-MoCap, WildAvatar), upper-body (TalkShow), and head-only (INSTA). Across all benchmarks, FlexiAvatar consistently achieves state-of-the-art quantitative reconstruction fidelity.

Table 1: Full-Body Avatar Reconstruction on the NeuMan Dataset

Method	Representation	PSNR ↑	SSIM ↑	LPIPS (×100) ↓
HumanNeRF	Implicit NeRF	27.06	0.967	1.90
InstantAvatar	Implicit Fast NeRF	28.47	0.972	2.80
NeuMan	Neural Radiance Field	29.32	0.972	1.40
Vid2Avatar	Unmasked Decomposition	30.70	0.980	1.40
3DGS-Avatar	Deformable 3DGS	28.99	0.974	1.60
GaussianAvatar	Pose-driven 3DGS	29.94	0.980	1.20
Vid2AvatarPro	Prior-guided 3DGS	32.71	0.983	1.19
ExAvatar	Expressive 3DGS	34.80	0.984	0.90
FlexiAvatar (Ours)	Visibility-Aware 3DGS	35.77	0.987	0.83

Table 2: Upper-Body Avatar Reconstruction on the TalkShow Dataset

Subject	Method	PSNR ↑	SSIM ↑	LPIPS (×100) ↓
Oliver	GART	24.14	0.927	7.76
	GUAVA (Upper-body specific)	26.76	0.890	12.17
	ExAvatar (Full-body general)	29.13	0.938	2.29
	Ours	29.80	0.952	1.81
Conan	GART	26.89	0.974	5.56
	GUAVA (Upper-body specific)	29.70	0.928	7.69
	ExAvatar (Full-body general)	35.66	0.980	3.61
	Ours	36.73	0.984	2.82
Chemistry	GART	22.28	0.914	11.89
	GUAVA (Upper-body specific)	26.85	0.920	7.15
	ExAvatar (Full-body general)	25.67	0.922	10.29
	Ours	27.80	0.935	9.67

Table 3: Head-Only Avatar Reconstruction on the INSTA Dataset

Method	Type	PSNR ↑	SSIM ↑	LPIPS (×100) ↓
FlashAvatar	Embedded 3DGS	27.44	0.912	11.05
FATE	Full-head Editable 3DGS	27.85	0.942	5.68
MonoGaussianAvatar	Point-based 3DGS	28.91	0.945	7.43
SplattingAvatar	Mesh-embedded 3DGS	29.03	0.932	10.35
GaussianAvatars	Rigged 3DGS	29.10	0.945	8.61
GaussianBlendShapes	Blendshape 3DGS	30.01	0.947	9.17
RGBAvatar	Online Compact 3DGS (Specialized SOTA)	32.72	0.953	6.04
FlexiAvatar (Ours)	Unified Framework	33.04	0.953	5.63

Ablation Study

Table 4: Component Ablations on NeuMan (Full-Body) and INSTA (Head-Only)

Benchmark	Configuration / Ablation	PSNR ↑	SSIM ↑	LPIPS (×100) ↓	Note
NeuMan (Full-body)	Full Model	35.77	0.987	0.83	Complete proposed pipeline
	w/o refinement	34.90	0.985	0.86	Drops 0.87 dB; degrades fine facial expressions and hand sharpness
	w/o gen. data	35.29	0.986	0.92	Unseen back regions exhibit missing texture artifacts under novel views
INSTA (Head-only)	Full Model	33.04	0.953	5.63	Complete proposed pipeline
	w/o visibility-aware opt.	32.15	0.949	5.80	Drops 0.89 dB; spurious gradients from unseen lower limbs corrupt head details
	w/o SMPL-X opt.	31.07	0.934	7.60	Largest single degradation (-1.97 dB); unmasked occluded joints distort canonical mesh

Table 5: Computational Footprint and Rendering Efficiency

Setting	Method	Gaussians (#G)	Asset Size (MB)	Anim (ms)	Rend (ms)	FPS ↑
Head (INSTA)	ExAvatar	167,390	8.94	32.175	2.098	29.18
	Ours	84,095 (-49.7%)	4.49 (-49.8%)	24.135 (-25.0%)	1.410 (-32.8%)	39.15 (+34.2%)
Upper Body (TalkShow)	ExAvatar	167,390	8.94	30.239	1.966	31.06
	Ours	144,350 (-13.8%)	7.71 (-13.8%)	28.127 (-7.0%)	1.408 (-28.4%)	33.87 (+9.1%)

Key Findings

• Confidence-masked SMPL-X tracking is critical for partial views: Removing keypoint confidence masking (w/o SMPL-X opt.) incurs a catastrophic drop on INSTA (PSNR falls by 1.97 dB, LPIPS degrades from 5.63 to 7.60). Without masking, noisy 2D detections on out-of-frame joints corrupt the skeletal configuration, misaligning Gaussian attachment anchors.
• Visibility pruning delivers dual gains in fidelity and efficiency: Otsu-based visibility filtering not only eliminates spurious gradient contamination across the triplane (boosting PSNR by nearly 0.9 dB), but also eliminates 49.7% of inactive Gaussians in head-only captures. This halves the asset memory footprint (from 8.94 MB to 4.49 MB) and increases frame rate from 29.18 to 39.15 FPS (+34.2%).
• Localized residual refinement recovers lost high frequencies: The dedicated face and hand residual MLPs contribute a 0.87 dB PSNR gain on NeuMan with negligible overhead, successfully recovering fine facial expressions, eyelid folds, and articulated finger creases.

Highlights & Insights

• Adaptive Evidence-Driven Pruning: By leveraging rasterization visibility rates and Otsu's adaptive bimodal thresholding, the model automatically discovers the exact boundary of visual evidence without manual tuning across arbitrary cropping ratios.
• Breaking the Fragmentation of Avatar Architectures: Rather than fragmenting the domain into FLAME-based head models, single-view upper-body models, and full-body rigs, FlexiAvatar establishes that a unified SMPL-X 3DGS foundation can surpass all specialized architectures when paired with principled visibility masking.
• Asymmetric Diffusion Video Prior: Instead of employing fragile and compute-heavy score distillation sampling (SDS) during 3D optimization, FlexiAvatar uses a video diffusion model (MimicMotion) to generate 360° rotating sequences as an auxiliary training set, elegantly solving back-view texture completion with low computational overhead.

Limitations & Future Work

• Non-Rigid Dynamics in Loose Clothing: As with most SMPL-X LBS-driven Gaussian representations, FlexiAvatar struggles to accurately simulate complex non-rigid topological deformations of loose garments (such as long dresses, fluttering scarves, or open coats) that decouple from skeletal motion.
• Cross-View Lighting and Texture Discrepancies in Diffusion Views: While down-weighting the loss on diffusion-generated frames prevents structural corruption, synthetic auxiliary views may still introduce minor illumination or texture inconsistencies under extreme novel viewpoints.
• Future Directions: Extending the visibility-aware mask into a deformation-aware formulation could dynamically segment regions undergoing severe non-rigid clothing movement, integrating physical simulation constraints into the 3DGS optimization pipeline.

Related Work & Insights

• vs ExAvatar: ExAvatar represents the baseline for expressive whole-body 3DGS avatars, but it forces full-body Gaussian optimization on partial-body videos, resulting in severe ghost artifacts and triplane gradient leakage into the torso. FlexiAvatar prunes unobserved Gaussians to eliminate spurious gradients, outperforming ExAvatar in quality while halving memory footprint in head-only captures.
• vs GUAVA: GUAVA reconstructs upper-body avatars from single images but lacks multi-view temporal consistency and cannot generalize to full-body or head-only inputs. FlexiAvatar leverages multi-frame dynamics and diffusion-assisted completion, outperforming GUAVA by 3.06 to 7.03 dB on TalkShow clips.
• vs RGBAvatar & FATE: RGBAvatar and FATE are specialized head avatar methods tied to facial blendshapes or strict head boundaries. FlexiAvatar proves that a unified whole-body SMPL-X formulation, augmented with FLAME facial alignment and residual MLPs, can surpass dedicated head models (33.04 dB vs. 32.72 dB PSNR on INSTA).

Rating

• Novelty: ⭐⭐⭐⭐⭐ Formulates the first unified 3DGS human avatar reconstruction framework across arbitrary visibility; the adaptive Otsu visibility pruning and decoupled spatiotemporal residual modules are principled and elegant.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Rigorous evaluations across five datasets covering full-body, upper-body, and head-only settings, accompanied by comprehensive ablations on visual quality, memory footprint, and runtime.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear motivation targeting a pervasive yet overlooked assumption in 3DGS avatar modeling, backed by detailed mathematical formulations and qualitative figures.
• Value: ⭐⭐⭐⭐⭐ Solves the practical dilemma of maintaining fragmented pipelines for different camera viewports, providing a high-efficiency, single-pipeline solution for AR/VR, video telepresence, and digital avatars.