PHOSA: Photorealistic 3D Sign Avatar Modeling and Benchmark

Conference: ECCV 2026
Paper: ECCV Official
Project: https://naaapi.github.io/PHOSA
Area: Human Understanding
Keywords: Sign Language Avatar, 3D Gaussian Splatting, SMPL-X Fitting, MVSign Benchmark, Kinematic Decoupling

TL;DR

Addressing the lack of photorealism and fine-grained hand/facial fidelity in mesh-based sign language avatars, this paper introduces MVSign—the first 16-view Chinese sign language benchmark co-designed with Deaf experts—coupled with a hybrid multi-model SMPL-X fitting pipeline and a body-head-hand decoupled 3D Gaussian representation featuring partial hand kinematic decoupling.

Background & Motivation

Sign Language Production (SLP), which converts spoken or written text into continuous sign sequences, is fundamental for enabling accessible communication between hearing individuals and the Deaf community. While recent generative models have substantially advanced motion and pose synthesis, existing SLP systems predominantly represent signers using parametric body meshes such as SMPL-X. These synthetic meshes lack photorealistic visual appearance, realistic cloth deformation, and fine-grained expressiveness, failing to convey the natural, nuanced signing experience that Deaf individuals strongly prefer and rely on for accurate semantic interpretation.

Developing animatable, photorealistic sign avatars driven by parametric body models poses two severe bottlenecks. First, suitable multi-view benchmarks are scarce: existing multi-view human datasets (e.g., DNA-Rendering, HumanRF, MVHumanNet) focus almost exclusively on coarse torso and limb motions, leaving out the delicate hand configurations and subtle facial expressions critical for signing; conversely, existing sign language datasets (e.g., PHOENIX14T, How2Sign) are designed for recognition or coarse motion generation, lacking multi-view imagery and calibrated 3D annotations necessary for neural human rendering. Second, modeling fine-grained articulations remains inherently difficult: modern 3D Gaussian avatar pipelines commonly model whole-body dynamics jointly, causing the network capacity to be heavily dominated by large-scale trunk motion while treating rapid, self-occluding hand gestures as blurred background noise.

To overcome these barriers, the authors establish an end-to-end paradigm spanning data collection, accurate parameter annotation, and representation decoupling. Collaborating directly with Deaf experts, they build a 16-view synchronous capture setup covering both fundamental sign shapes and daily conversational sentences, alongside a multi-model hybrid SMPL-X fitting pipeline. Core idea: establish MVSign as the first 16-view expert-curated sign benchmark, develop a hybrid fitting pipeline fusing specialized pose/hand/face priors, and design a decoupled 3D Gaussian representation with partial hand kinematic decoupling that isolates hand mobility from trunk motion to achieve high-fidelity sign avatar synthesis.

Method

Overall Architecture

The PHOSA framework comprises the MVSign benchmark pipeline, a motion-aware data sampling strategy, and a decoupled 3D Gaussian avatar rendering network. Given driving parameters \(\Theta\), the pipeline deforms canonical templates using Linear Blend Skinning (LBS) and renders posed coordinate maps for body, head, and hands. Three specialized StyleUNet networks process these posed maps to predict canonical Gaussian offsets and appearance attributes. Finally, canonical Gaussians are warped to observation space via LBS and rendered using differentiable Gaussian splatting rasterization.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-view Sign Videos<br/>16 synchronized RGB 2048×2448"] --> B["Hybrid SMPL-X Fitting Pipeline<br/>DWPose+HaMeR triangulation+INFERNO face"]
    B --> C["Motion-aware Data Sampling<br/>blur filtering + pose distance balancing"]
    C --> D["Decoupled Sign Avatar Representation<br/>body/head/hands split + partial hand kinematics"]
    D --> E["Three-branch StyleUNet Attribute Prediction<br/>canonical Gaussian offsets and appearances"]
    E --> F["LBS Observation Warping & Rasterization<br/>anatomical focus losses for high-fidelity rendering"]

Key Designs

1. Hybrid SMPL-X Fitting Pipeline: Fusing Multi-Model Priors for Expressive Body-Hand-Face Alignment Complex hand self-occlusions, rapid finger crossings, and subtle facial articulations cause single end-to-end whole-body estimators to fail. PHOSA develops a modular hybrid fitting pipeline combining specialized models: it first detects initial 2D keypoints via DWPose across all 16 views, and then invokes HaMeR on cropped hand regions to obtain accurate MANO hand parameters and 2D keypoints, directly overriding noisy hand detections from DWPose. Calibrated multi-view constraints are then enforced via multi-view triangulation optimization to obtain robust 3D keypoints \(\hat{J}^{3D}\). Body pose \(\theta\) and shape \(\beta\) parameters are solved by minimizing a Geman-McClure robust error objective: $$ \min_{\theta, \beta, \psi} \sum_{i=1}^N \sum_{j \in \mathcal{J}} \rho\left( R_\theta(J(\beta))j - \hat{J}^{3D} \right) + \gamma |\psi|^2 $$ Because skeletal keypoints are too sparse to capture subtle mouth movements and facial expressions, the pipeline integrates INFERNO to extract dense, speech-aware 3D facial expression parameters \(\psi\), replacing the initial skeletal face parameters. Frame-to-frame warm-start initialization and SmoothNet filtering are subsequently applied to guarantee smooth temporal consistency.

2. Motion-aware Data Sampling Strategy: Eliminating Motion Blur and Gesture Imbalance Sign language features fast hand movements that introduce motion blur even at elevated frame rates, as well as a large proportion of resting or hanging hand poses that skew the dataset. Training directly on uniformly sampled frames causes the avatar representation to overfit to static poses and underperform on expressive gestures. PHOSA designs a two-stage motion-aware sampling strategy: first, it identifies blurred frames by jointly gating three complementary metrics—DWPose detection confidence, Laplacian gradient magnitude over cropped hands, and hand velocity computed from SMPL-X parameter trajectories. Second, it computes pairwise pose distances in SMPL-X space, clusters gestures into semantic categories, and removes redundant resting frames to produce a balanced gesture distribution for training.

3. Decoupled Sign Avatar Representation with Partial Hand Kinematic Decoupling Standard holistic 3D Gaussian avatars bind the entire body to a single 2D pose map, allowing trunk motion to dilute high-frequency hand gradients. PHOSA explicitly decomposes the avatar into three anatomical branches: BodyNet, HeadNet, and HandNet. Each branch utilizes a dedicated StyleUNet to predict pose-dependent canonical Gaussian attributes (position offset \(\Delta P\), rotation, scale, opacity, and color). Crucially, the authors propose Partial Hand Kinematics (PHK): when rendering the hand pose position map \(P_{\text{hand}}\), all body joints are frozen in canonical rest pose, preserving only wrist and finger joint mobility. The two hands are rendered from orthogonal top and bottom views and concatenated into a compact \(256 \times 256\) representation. This decouples local hand gesture deformations from global arm and torso trajectories, drastically improving gesture generalization on unseen test poses.

Loss & Training

To ensure tight alignment with human anatomy, the network predicts canonical position offsets \(\Delta P\) relative to SMPL-X vertices rather than unconstrained absolute coordinates. The composite objective balances global photometric accuracy, region-specific anatomical emphasis, and mesh regularization: $$ \mathcal{L} = \mathcal{L}1 + \lambda}} \mathcal{L{\text{SSIM}} + \lambda}} \mathcal{L{\text{LPIPS}} + \lambda}} \mathcal{L{\text{hand}} + \lambda}} \mathcal{L{\text{head}} + \lambda |\Delta P|_2^2 $$ Here, }\(\mathcal{L}_{\text{hand}}\) and \(\mathcal{L}_{\text{head}}\) compute focused L1 losses over Sapiens+SAM segmented masks with weights \(\lambda_{\text{hand}} = 3\) and \(\lambda_{\text{head}} = 3\), forcing gradients onto fine-grained communicative regions; \(\lambda_{\text{offset}} = 0.005\) penalizes unnatural Gaussian drift. Resolution settings are \(1024 \times 512\) for body, \(256 \times 256\) for hands, and \(256 \times 128\) for head. All models train stably on a single NVIDIA RTX 3090 GPU.

Key Experimental Results

Main Results

Quantitative evaluations were performed on both the 16-view MVSign dataset and monocular in-the-wild sign language videos collected from the web. Baselines include SplattingAvatar, GaussianAvatar, AnimatableGaussians, EVA, and Mmlphuman, all adapted to SMPL-X driving.

Table 1: Main comparative evaluation on MVSign and in-the-wild web sign videos (Table 10 in original paper)

Setting & Method	Full PSNR↑	Full SSIM↑	Full LPIPS↓	Hand PSNR↑	Hand SSIM↑	Hand LPIPS↓	Face PSNR↑	Face SSIM↑	Face LPIPS↓
Multi-view: MVSign
SplattingAvatar [47]	23.71	0.9625	0.0494	15.64	0.6762	0.3884	16.90	0.7633	0.2423
GaussianAvatar [20]	24.23	0.9633	0.0428	16.30	0.6833	0.3082	17.78	0.7737	0.2086
AnimatableGS [30]	25.09	0.9647	0.0465	16.95	0.7002	0.3135	18.63	0.7842	0.2230
EVA [19]	25.56	0.9667	0.0436	17.43	0.7154	0.2869	19.21	0.8000	0.1964
Mmlphuman [57]	25.91	0.9652	0.0460	17.41	0.7084	0.2675	19.38	0.8025	0.1869
PHOSA (Ours)	27.03	0.9689	0.0393	18.55	0.7325	0.2568	20.60	0.8189	0.1757
Monocular: In-the-wild Web
SplattingAvatar [47]	18.71	0.9198	0.1223	16.37	0.6572	0.3114	16.97	0.7243	0.2751
GaussianAvatar [20]	18.90	0.9261	0.0918	16.30	0.6668	0.2797	17.29	0.7178	0.2157
AnimatableGS [30]	19.34	0.9252	0.0954	17.42	0.6951	0.2799	17.65	0.7277	0.2561
EVA [19]	19.22	0.9250	0.0966	17.49	0.6958	0.2588	17.75	0.7381	0.2132
Mmlphuman [57]	19.61	0.9239	0.1023	16.90	0.6645	0.2607	17.88	0.7284	0.2374
PHOSA (Ours)	20.39	0.9334	0.0773	18.70	0.7189	0.2289	18.84	0.7502	0.1864

Ablation Study

The impact of architectural decoupling, kinematic isolation, and the hybrid SMPL-X fitting pipeline are isolated below.

Table 2: Ablation of decoupled representation and partial hand kinematics (Table 7 in original paper)

Configuration	Full PSNR↑	Full SSIM↑	Full LPIPS↓	Hand PSNR↑	Hand SSIM↑	Hand LPIPS↓	Face PSNR↑	Face SSIM↑	Face LPIPS↓	Description
Baseline (single map)	25.24	0.9570	0.0490	16.65	0.6912	0.3759	18.47	0.7948	0.2167	Monolithic whole-body representation
+ DR (Decoupled Rep.)	26.28	0.9607	0.0430	17.53	0.7174	0.3188	20.53	0.8099	0.1731	Independent body/head/hand branches
+ PHK (Partial Hand Kinematics)	27.93	0.9665	0.0388	18.74	0.7322	0.2666	21.50	0.8253	0.1595	Freezing torso joints during hand rendering

Table 3: Ablation of hybrid SMPL-X fitting strategy on avatar modeling (Table 2 in original paper)

Fitting Strategy	Full PSNR↑	Full SSIM↑	Full LPIPS↓	Hand PSNR↑	Hand SSIM↑	Hand LPIPS↓	Face PSNR↑	Face SSIM↑	Face LPIPS↓	Description
w/o Hybrid Fitting	23.58	0.9637	0.0618	16.82	0.7371	0.2987	16.62	0.7923	0.2435	DWPose skeletal fitting alone
w Hybrid Fitting	26.90	0.9722	0.0370	18.85	0.7704	0.2301	20.51	0.8499	0.1665	HaMeR + Triangulation + INFERNO

Key Findings

• Partial Hand Kinematic decoupling unlocks clean hand synthesis: While isolating body parts (+DR) yields modest perceptual improvements, adding partial hand kinematics (+PHK) drives hand LPIPS down from 0.3188 to 0.2666 and improves hand PSNR by 1.21 dB. Removing body pose variance from the hand maps prevents the network from learning spurious correlations.
• Accurate hybrid fitting is foundational for neural avatar rendering: Table 3 shows that fitting parameters directly from 2D keypoints degrades downstream rendering quality to 23.58 dB PSNR; integrating HaMeR and INFERNO elevates full avatar PSNR to 26.90 dB, with hand and facial LPIPS improving by 23.0% and 31.6% respectively.
• Deaf community evaluation confirms substantial real-world superiority: In the user study with 20 native Deaf participants (Table 9 in original paper), PHOSA was favored by 56.8% for comprehensibility (vs. 20.8% for Mmlphuman), 53.0% for clarity, 62.4% for temporal consistency, and 51.2% for aesthetic appeal. Furthermore, 82.5% of Deaf participants reported that photorealistic avatars significantly improved their understanding over raw SMPL-X meshes.

Highlights & Insights

• First high-resolution multi-view benchmark co-designed with Deaf experts: MVSign establishes a gold-standard dataset recorded across 16 synchronized 2048×2448 cameras with dedicated head portrait views and extensive lexicon coverage (109 basic signs + 50 situational sentences), bridging a major empirical gap in sign language research.
• Kinematic isolation as a general design principle for dexterous avatars: The partial kinematic decoupling trick cleanly separates local articulation from global pose hierarchies, an architectural insight that transfers directly to other dexterous tasks like musical instrument playing and fine hand-object manipulation.
• Pragmatic hybrid mocap engineering: Rather than relying on a single end-to-end whole-body estimator, combining specialized state-of-the-art estimators (DWPose, HaMeR, INFERNO) with multi-view geometric triangulation establishes a highly robust annotation standard.

Limitations & Future Work

• Inference latency from multi-branch StyleUNets: Forward inference across three separate StyleUNet backbones incurs notable computational overhead, currently preventing low-power or mobile real-time interactive deployment.
• Subject-specific optimization: Avatars are trained per subject and do not yet support zero-shot feed-forward generation from a single uncalibrated snapshot.
• Future directions: Exploring compact network distillations (e.g., lightweight MLPs or tensor factorization) to accelerate inference, and integrating explicit sign grammar physics constraints to prevent hand-to-body interpenetration.

Related Work & Insights

• vs AnimatableGaussians [30]: AnimatableGaussians parameterizes canonical Gaussians via a single whole-body pose map, which causes rapid finger movements to blur under coarse trunk movement; PHOSA introduces three-way anatomical decoupling and partial hand kinematics to resolve fine-grained gesture details.
• vs EVA [19]: EVA addresses expressive monocular avatars with adaptive density control, but lacks multi-view supervision and kinematic decoupling; under fast sign sequences it suffers from hand-body entanglement, whereas PHOSA maintains sharp finger boundaries across novel views and poses.
• vs SGNify [15] / NSA [2]: SGNify and NSA focus primarily on mesh-level motion estimation and generative sign synthesis; PHOSA tackles the complementary photorealistic rendering challenge, converting parametric driving parameters into photorealistic 3D Gaussian avatars.

Rating

• Novelty: ⭐⭐⭐⭐☆ (First multi-view sign Gaussian avatar benchmark; elegant kinematic decoupling and hybrid fitting pipeline)
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ (Evaluated across multi-view benchmark and in-the-wild web video, extensive ablations, and Deaf community user study)
• Writing Quality: ⭐⭐⭐⭐⭐ (Rigorous problem formulation, transparent ablation setups, and well-structured presentation)
• Value: ⭐⭐⭐⭐⭐ (Substantially advances accessible communication technology for the Deaf community; releases high-value benchmark data)

Related Papers

• [ECCV 2026] Mind-to-Face: Neural-Driven Photorealistic Avatar Synthesis via EEG Decoding
• [ECCV 2026] BackTranslation2.0 -- A Linguistically Motivated Metric to Assess Sign Language Production
• [ECCV 2026] SIGNER: Temporally Grounded Sign Language Generation via Time-Resolved Conditioning
• [ECCV 2026] SIGNET: Motion-Level Knowledge Transfer for Cross-Language Sign Language Translation
• [ECCV 2026] SignNet-1M: Large-Scale Multilingual Sign Language Video Dataset with Downstream Benchmarks