LUNA: Learning Universal 3D Human Animation Beyond Skinning¶

Conference: ECCV 2026
arXiv: 2606.31981
Code: None
Area: 3D Vision
Keywords: Human Animation, 3D Gaussian Splatting, LBS-Free, Implicit 2D Driving, Cross-Identity Generalization

TL;DR¶

LUNA proposes the first end-to-end 3D human animation framework that completely abandons Linear Blend Skinning (LBS). It directly maps 2D driving signals (RGB images, keypoints, hand-drawn sketches, or even unseen character images) to 3D Gaussian deformations via a decoupled Transformer animator. During training, LBS distillation is utilized to provide structural priors to prevent geometric collapse. During inference, it completely bypasses LBS constraints, achieving zero-shot cross-identity and cross-modal 3D human animation.

Background & Motivation¶

From monocular input, creating high-fidelity, drivable 3D digital human avatars is a long-standing goal in computer vision and graphics, with massive demand in film production, VR/AR, and gaming. However, current monocular human animation methods rely almost entirely on pre-defined parametric human models like SMPL/SMPL-X and Linear Blend Skinning to drive deformations. While this pipeline provides convenient shape priors, it also introduces fundamental limitations. On one hand, the topology of parametric human models is fixed, and skeletal weights cannot express complex non-rigid dynamics such as loose clothing sliding or fluttering drapery. On the other hand, LBS is highly dependent on accurate 3D pose estimation. Yet, monocular 3D pose estimation is inherently an ill-posed problem, where tiny joint errors can escalate into temporal jitter and geometric artifacts in rendering. Recently, feed-forward methods represented by LHM and IDOL have achieved impressive generalization performance via large-scale Transformers. However, they still anchor animations to parametric models—utilizing LBS to map the canonical space to the pose space during training, and requiring explicit monocular 3D poses as driving signals during inference—thus failing to escape the aforementioned bottlenecks.

The key challenge of this dilemma lies in: the structural prior provided by LBS is crucial for geometric stability, but its rigid skeletal topology inherently limits the expressible range of deformations. If LBS is completely abandoned, the model is prone to collapsing into flat structures due to the lack of explicit human shape constraints; if LBS is retained, it cannot capture non-rigid dynamics like fabric sliding. The key insight of this paper is: LBS can serve as a soft prior during the training phase to stabilize geometry, while the inference phase can completely break free from it—allowing the model to directly learn the end-to-end mapping from 2D signals to 3D deformations without requiring explicit 3D pose estimation as an intermediate representation.

Core Idea: The paper proposes LUNA, the first completely LBS-free end-to-end 3D human animation framework. It directly maps 2D driving signals to 3D Gaussian deformations via a decoupled Transformer neural animator. During training, LBS teacher distillation is utilized to provide a structural soft prior to prevent geometric collapse, while learning rich non-rigid dynamics from limited annotated data and massive unannotated videos using an adaptive weighted mixed-execution supervision. During inference, it achieves zero-shot cross-identity and cross-modal implicit 3D driving.

Method¶

Overall Architecture¶

The inputs to LUNA consist of a set of unaligned multi-view identity images (4 views) and a single 2D driving image, and the output is a high-fidelity 3D human avatar in the pose corresponding to the driving image. The entire pipeline is divided into two stages: the Identity Encoder lifts the multi-view identity images into a set of 3D Gaussians in canonical space and their corresponding semantic tokens; the Implicit Neural Animator extracts motion cues from the 2D driving signal to predict decoupled global rigid transformations (rotation $R$ + translation $T$) and per-Gaussian local deformations (position offsets, rotation residuals, color residuals), mapping the canonical Gaussians into the pose space. Finally, the posed Gaussians are rendered via differentiable rasterization, trained end-to-end with a mixed supervision strategy.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multi-view Identity Images<br/>(4 Unaligned Views)"] --> B["Sapiens Encoder"]
    B --> C["Image Tokens"]
    C --> D["MM-Transformer Fusion<br/>+ Semantic Query Tokens"]
    D --> E["Canonical 3D Gaussians<br/>+ Identity Tokens"]

    F["2D Driving Signals<br/>(RGB/Keypoints/Sketches)"] --> G["DINOv3 Encoder"]
    G --> H["Motion Tokens"]

    E --> I["Implicit Neural Animator"]
    H --> I
    I --> J["Global Rigid Transform R,T<br/>+ Per-Gaussian Local Deformation"]
    J --> K["Posed 3D Gaussians"]
    K --> L["Differentiable Rasterization"]
    L --> M["Rendered Images & Mixed Supervision<br/>(Photometric + Distillation + Projection)"]

Key Designs¶

1. Decoupled Neural Animator: Separate Prediction of Global Rigid and Local Non-Rigid Components

Traditional LBS implicitly binds deformations to skeletal weights, making it difficult to express non-rigid dynamics. LUNA decomposes animation into two levels: global rigid transformation (rotation $R$ and translation $T$) for coarse-grained human spatial alignment, and per-Gaussian local deformation (position offset $\Delta\mu$, rotation residual $\Delta q$, color residual $\Delta c$) for capturing fine-grained non-rigid effects under pose changes. The global rotation is predicted in a continuous triangular form from global motion descriptors extracted by DINOv3 via an MLP head—the outputs $(s_i, c_i)$ for each axis are normalized by Tanh and combined into a valid rotation matrix, avoiding direct regression of Euler angles with their $\pm\pi$ discontinuities. Local deformations are decoded by a lightweight MLP after fusing identity tokens and motion tokens through a cross-modal Transformer. This decoupled design allows the global transformation to handle large rotations and displacements without affecting local details, while local deformations focus on non-rigid effects like fabric sliding and clothing fluttering, jointly achieving an expressiveness far exceeding LBS.

2. Implicit 2D Driving: Completely Bypassing Explicit Pose Estimation

The most striking design of LUNA is its ability to directly accept 2D images as driving inputs, rather than the explicit 3D pose parameters required by traditional methods. The driving image is processed by a DINOv3 encoder to generate motion tokens. DINOv3 is selected over a specialized human encoder like Sapiens because animation relies more on motion semantics than identity details, and DINOv3 generalizes better to out-of-domain signals (such as abstract sketches and cartoon characters). These motion tokens are fused with canonical identity tokens via a cross-modal Transformer, and the animator directly decodes the deformation parameters. To compel the model to extract only motion rather than appearance, strong data augmentations like grayscale conversion, Gaussian blur, and random foreground translation are applied to the driving images during training, forcing the model to ignore texture details. The revolutionary aspect of this design is that, after training, LUNA can be driven zero-shot by heterogeneous signals such as RGB images, 2D keypoints, hand-drawn sketches, or even game character images, completely eliminating monocular 3D pose estimation as a crucial bottleneck and significantly reducing error accumulation.

3. Mixed Supervision and LBS Distillation: Guaranteeing Geometric Stability without Skeletal Priors

An animator that completely abandons LBS and learns 3D deformations directly from 2D is severely under-constrained; relying solely on photometric rendering loss would cause the reconstruction to collapse into a flat structure. LUNA's solution exquisitely adopts "using yet discarding": it retains an LBS teacher model during training, which generates structured Gaussian attributes as soft distillation targets for each frame:

\[ \mathcal{L}_{distill} = \|\mu_p - \hat{\mu}\|_1 + \lambda_q \mathcal{L}_{rot}(q_p, \hat{q}) + \lambda_c \|c_p - \hat{c}\|_1 \]

More importantly, an adaptive weighting scheme is introduced: annotated data (with ground-truth LBS) is typically far scarcer than the massive unannotated web videos, and pure photometric gradients can overwhelm structured signals. LUNA adopts a fixed annotated-to-unannotated batch ratio (1:5) and scales the distillation loss weight inversely to the annotation ratio ($\lambda_{distill}=5$), preventing the structural prior from being diluted in mixed batches. Additionally, 3D supervision is not directly applied to global translation; instead, a 2D reprojection loss is used (projecting 3D Gaussian centers to 2D and aligning them with annotations) to prevent depth-axis errors from dominating the gradient. This mixed supervision allows the model to inherit the geometric stability anchored by LBS while learning complex non-rigid dynamics far beyond the expressive limits of LBS on large-scale unannotated data.

Loss & Training¶

Total loss function: $$ \mathcal{L}{total} = \mathcal{L}} + \mathcal{LR + \mathcal{L} $$} + \lambda_{distill} \cdot \mathcal{L}_{distill

where $\mathcal{L}_{render} = \mathcal{L}_1 + \mathcal{L}_{mask} + \mathcal{L}_{LPIPS}$ is the photometric rendering loss (pixel L1 + foreground mask + perceptual loss); $\mathcal{L}_R$ represents continuous-form supervision for global rotation; $\mathcal{L}_{proj}$ is the 2D reprojection loss for translation.

Training is performed in two stages: Stage 1 involves pre-training on monocular videos (Video35K + iPhone1K + Cloth10K, 64 A100 GPUs, 30K steps, lr=4e-4, using only rotation and projection losses in the first 1K steps for warmup); Stage 2 involves fine-tuning on the multi-view Dome dataset (16 A100 GPUs, 30K steps, lr=1e-4). Key configuration: number of Gaussians $K=8192$ (decoded to 65536 Gaussians), feature dimension $C=1024$, identity tokens containing 4096 body + 4096 face Gaussians.

Key Experimental Results¶

Main Results¶

Dataset	Metric	Vid2Avatar	ExAvatar	IDOL	LHM	UP2YOU	MV-LHM*	LUNA
Cloth10K	PSNR↑	18.97	19.53	17.43	19.20	19.74	20.12	22.07
(Non-rigid clothing)	L1↓	0.042	0.041	0.067	0.047	0.047	0.038	0.027
	LPIPS↓	0.157	0.153	0.213	0.167	0.149	0.158	0.131
NeuMan	PSNR↑	26.85	31.27	24.70	25.31	25.49	26.83	26.82
	L1↓	0.012	0.009	0.037	0.029	0.026	0.017	0.015
	LPIPS↓	0.017	0.009	0.051	0.039	0.041	0.023	0.023

Ablation Study¶

Configuration	PSNR↑(iPhone)	PSNR↑(Dome)	Key Findings
Full model	24.14	24.37	Full model
w/o Structural Distillation	21.19	21.71	LBS distillation is indispensable; without it, depth collapses and PSNR drops by ~3dB
w/o Global Rotation Decoupling	23.10	23.29	Severe drift and boundary aliasing occur during large movements without rotation separation
w/o Multi-view Fine-tuning	23.93	23.86	Pure monocular supervision leads to perspective distortion and floating Gaussians

Key Findings¶

Structural distillation is the most critical component: Removing it results in ~3dB drop in PSNR (24.14 $\to$ 21.19 on iPhone) along with severe depth collapse and geometric flattening, proving that photometric loss alone is insufficient to maintain geometric shapes without LBS priors.
Global rotation decoupling of large motions is paramount: Without rotation separation, the animator produces notable drift and boundary aliasing when processing large movements such as 180-degree turns, with iPhone PSNR declining from 24.14 to 23.10.
Animation smoothness vastly outperforms all LBS baselines: In terms of MSJ (Mean Squared Jerk, reflecting high-frequency jitter), LUNA achieves 0.0032, which is 4.5 times lower than the strongest baseline MV-LHM (0.0144). For MAE (Mean Acceleration Error), it achieves 0.0225 vs 0.0477 (a 2.1x reduction), indicating a massive improvement in temporal stability by bypassing pose estimation.
Cross-identity driving performance is close to self-driving levels: On NeuMan, cross-identity driving PSNR (26.80) is extremely close to self-driving (26.82), with keypoint error of only 17.3% (vs 26.7~28.7% for LBS methods), suggesting that the animator mainly extracts pure motion rather than appearance from driving signals.
Greatest superiority shown in loose clothing scenarios: On Cloth10K, PSNR is ~2dB higher than the best baseline (MV-LHM), while performance on NeuMan (tight clothing) is on par, demonstrating that the real value of being LBS-free lies in modeling complex non-rigid dynamics.

Highlights & Insights¶

The "use yet discard" LBS distillation strategy is the greatest architectural highlight: Instead of simply using an LBS teacher for distillation, it designs an adaptive weighted mixed supervision, allowing the model to initiate learning using LBS structural priors while capturing non-rigid dynamics far beyond the LBS limits on unannotated data—after training, the LBS teacher is entirely discarded with zero inference overhead.
The cross-modal generalization capability of implicit 2D driving is stunning: Despite never seeing sketches or cartoon characters during training, LUNA handles them zero-shot flawlessly—this is attributed to the out-of-domain generalization capability of DINOv3 and the aggressive data augmentations used during training, forcing the model to focus on motion rather than texture.
2D reprojection loss trick for translation: Instead of supervising 3D translation directly, it is projected and aligned in 2D, preventing depth-axis errors from dominating the gradient—this trick holds transfer value for any differentiable 3D deformation prediction task.
Continuous rotation representation in decoupled prediction: Representing each axis output $(s, c)$ via Tanh normalization into a rotation matrix avoids the continuous $\pm\pi$ discontinuity of direct Euler angle regression—a classic yet practical engineering optimization.

Limitations & Future Work¶

Limitations acknowledged by authors: Severe occlusion degrades the semantic quality of the 2D driving signals, occasionally introducing temporal instability; extreme poses or major body shape discrepancies degrade cross-identity animation quality, as motion and shape are not explicitly decoupled.
Data dependency: Although branded as LBS-free, training still depends on pseudo-labels generated by a large-scale LBS distillation teacher and multi-view annotated data (Dome dataset). The acquisition cost of such data is as high as traditional fitting data.
Rendering quality ceiling: The rendering fidelity of 3D Gaussians is slightly inferior to NeRF in detail preservation—on NeuMan, PSNR does not exceed the optimization-based baseline ExAvatar (26.82 vs 31.27), indicating room for improvement in feed-forward fidelity.
Future Directions: Introduce explicit temporal modeling (such as inter-frame motion constraints or video consistency loss) to handle occlusions; decouple body shape and motion into two separate networks to accommodate larger body shape variations; scale training to more diverse datasets.

vs LBS-based methods (LHM, IDOL): They cannot escape LBS-parameterized models in both training and inference, accumulating errors across pose estimation and skinning; LUNA only uses LBS distillation as a prior during training and completely detaches during inference, cutting off the error propagation chain.
vs optimization-based methods (ExAvatar, Vid2Avatar): They achieve high-fidelity through per-identity optimization but cannot generalize to new identities; LUNA is feed-forward, completing animation in a single forward pass, showing strong generalization but with a gap in fidelity.
vs 2D human animation methods (MagicAnimate, Champ): Based on 2D diffusion models, they are flexible but lack 3D geometric consistency, suffering from temporal flickering and view inconsistencies; LUNA operates in true 3D space, naturally maintaining view consistency, though with higher computation costs (multi-view Gaussian rasterization).

Rating¶

Novelty: ⭐⭐⭐⭐⭐ The first completely LBS-free end-to-end 3D human animation framework; its implicit 2D driving paradigm is highly original.
Experimental Thoroughness: ⭐⭐⭐⭐ Compared against both optimization-based and feed-forward baselines across multiple datasets with complete ablations; however, the cross-identity generalization experiment is relatively small in scale.
Writing Quality: ⭐⭐⭐⭐⭐ Clearly described methodology, rigorous structure, well-designed ablation studies, and appropriate illustrations.
Value: ⭐⭐⭐⭐⭐ Marks a paradigm shift in 3D digital human animation from "template-driven" to "implicit-driven".