GenLCA: 3D Diffusion for Full-Body Avatars from In-the-Wild Videos¶
Conference: ECCV2026
Paper: ECCV Paper
Project: GenLCA
Area: Human Understanding / 3D Vision
Keywords: animatable avatars, 3D diffusion, partial observability, visibility supervision, multimodal editing
TL;DR¶
GenLCA encodes real videos into structured 3D Gaussian tokens and supervises diffusion only in visible regions, learning text- and image-controlled full-body avatars from million-scale identity data while supporting generation, editing, and facial and body animation.
Background & Motivation¶
An animatable avatar must remain consistent when turning, changing pose, or changing expression, rather than merely looking convincing from one viewpoint. Traditional acquisition relies on synchronized multiview equipment or monocular videos with extensive body coverage, producing reliable supervision but limiting identity and clothing diversity. Synthetic data scales more readily, yet materials, facial details, and clothing appearance differ from real human imagery. SDS methods that optimize individual 3D humans using 2D diffusion priors can produce unnatural styles and require lengthy optimization for each asset. Native 3D generation therefore needs not just a stronger denoiser, but a scalable training representation that is realistic and animatable.
Real-world videos offer another source of scale, but their subjects often show only their upper bodies or continuously face the camera. A feedforward reconstruction model can convert these images into a complete-shaped token tensor without making every position in that tensor reliable. For example, an upper-body video can yield transparent legs, while frontal footage can yield a blurry reconstructed back. Treating the entire reconstruction as accurate 3D ground truth teaches the generator to reproduce these failures as part of the human distribution. The question shifts from completing every training example to retaining its trustworthy information without supervising the model with guessed regions.
GenLCA reuses LCA's animatable representation, aligns identities to a shared set of body query points, and evaluates supervision validity at each position. This spatial correspondence allows local observations across videos to support full-body distribution learning without requiring every video to capture a complete person. A smaller amount of well-covered captured data still participates, so the method neither relies exclusively on incomplete videos nor removes reconstruction priors. Core Idea: turn incomplete 2D videos into locally trustworthy 3D tokens, isolate invalid inputs with learnable placeholders, and learn the generative distribution only through visible-region supervision.
Method¶
Overall Architecture¶
Training inputs are video frames of the same identity; deployment conditions can instead be text, body-part images, or scribbles, and these roles must be distinguished. The pipeline uses structured avatar tokenization, point-preserving compression, visibility-aware training, and multimodal flow generation in that order. Its output is not a collection of inconsistent 2D views, but a 3D Gaussian representation that a shared decoder can animate and render. Tokenization and visibility computation prepare training data, the compressor is trained separately first, and diffusion then learns in the compressed latent space. Generation produces a complete latent representation from noise, decodes it into Gaussian tokens, and applies pose and expression conditions for animation.
%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
A["Training: video frames of one identity"] --> B["Structured Avatar Tokenization"]
B --> C["Point-Preserving Compression"]
B -->|Rendering contributions produce mask| D["Visibility-Aware Training"]
C --> D
D -.->|Placeholder inputs and masked supervision| E["Multimodal Flow Generation"]
F["Generation: noise and text,<br/>body-part images or scribbles"] --> E
E --> G["Latent and avatar decoding<br/>Pose- and expression-driven Gaussian rendering"]
Key Designs¶
1. Structured Avatar Tokenization: map video observations to shared body locations
LCA receives 4 body images and 4 face images of one identity, with Sapiens first extracting image features. Body frames are selected to maximize yaw differences and visible coverage; face inputs are crops from randomly sampled video frames. A fixed set of 8,192 query points is sampled on a template body mesh and shared across identities instead of generating a separate unordered point cloud for each person. A transformer jointly processes image features and query-point positional embeddings, producing a 1,024-dimensional Gaussian token for each point. Token indices consequently have consistent spatial meaning across identities, enabling corresponding masks and positional embeddings downstream. This structure does not establish accurate human ground truth; it gives reconstruction outputs a shared, usable spatial organization.
Each token is decoded by a lightweight MLP into 8 Gaussians, totaling 65,536 Gaussians; this total is derived by multiplying the paper's token count by its Gaussians per token. The avatar decoder also receives pose and expression parameters and outputs Gaussian attributes that vary with those driving conditions. Linear blend skinning (LBS) subsequently deforms the representation, and Gaussian Splatting renders it. The generator thus inherits body and facial animation capabilities without encoding animation control anew as diffusion conditions. It also inherits LCA's representational ceiling: a stronger generative prior does not automatically create pose-dependent details that the decoder cannot express. Page 6 and Figure 2 describe this interface; detailed internal LCA training is deferred to supplementary Appendix B.
2. Point-Preserving Compression: reduce channels without losing visibility correspondence
The original tokens have shape \(8{,}192\times1{,}024\), making direct modeling expensive and placing diffusion in a high-dimensional reconstruction feature space. The compressor maps them to \(8{,}192\times8\) latents, reducing channels without reducing the number of query points. The channel dimension shrinks by a factor of 128, but total computation or memory need not fall by a factor of 128 because attention still processes the same number of positions. Encoder and decoder alternate MLP channel transformations with self-attention feature fusion, continually incorporating positional embeddings of the same query points. Preserving point count keeps each visibility label aligned with a latent token rather than requiring occlusion labels to be projected into a different spatial organization.
The encoder's 7 blocks use channel dimensions [512, 256, 128, 64, 32, 16, 8]. The decoder's 5 blocks use [32, 64, 128, 512, 1024], restoring the original Gaussian tokens for LCA decoding. Training combines L1 token reconstruction error and KL divergence, preserving decodable information while constraining the latent distribution. The main text writes latent encoding as \(Z=E(T,X)\), with query points \(X\) participating in encoding. Equation (1) on cached page 7 is corrupted, and the adjacent sentence swaps the encoder and decoder letter definitions; this note follows the architecture and encoding relation above without reconstructing the authors' exact equation. Visibility-aware training is applied to diffusion after compression; this does not establish that the compressor itself uses the same masked objective.
3. Visibility-Aware Training: isolate invalid inputs and invalid supervision together
The mask is based on rendering contributions rather than merely checking whether template surface normals face the camera. The reconstructed Gaussians are rendered with the cameras and poses corresponding to the input body images, and gradients measure their contributions to the images. Low-contribution Gaussians are considered invisible; a token is valid when at least 2 of its 8 decoded Gaussians are visible in at least one input view. This traces image evidence back to canonical body locations, separating trustworthy frontal regions from backs or legs guessed by the reconstruction model. The main text does not specify the numerical low-contribution threshold or complete gradient implementation, so the rule alone is insufficient for full reproduction.
At the diffusion input, invalid latent tokens are replaced by learnable placeholder features shared across identities, preventing their blurry appearance from influencing valid regions through attention. At the loss output, weighted supervision is computed only over valid regions, avoiding pressure to fit transparent legs or erroneous backs. The operations are complementary: masking only the loss does not eliminate input contamination, while replacing inputs but retaining corrupted targets remains inappropriate. Placeholders are not ground truth for an identity's missing parts, nor are they the MAE objective of reconstructing deliberately removed image patches. The model learns a joint distribution from semantic relationships among visible tokens across samples, rather than receiving accurate invisible-region targets for every sample. Replacing learnable placeholders with fixed zero tokens produces unnatural colors in the authors' ablation, suggesting that invalidity also needs a suitable internal representation.
4. Multimodal Flow Generation: connect text and local appearance conditions to a shared 3D latent space
The generator uses double-stream MMDiT blocks, separately computing queries, keys, and values for latent and condition tokens before concatenating them for joint attention. Attention outputs are split back into their respective processing branches, with distinct branches also used for different conditioning modalities. Time steps provide modulation; body-point embeddings are added to latent queries and keys to preserve each token's spatial correspondence. The network contains 28 blocks with 1,024 channels, 16 attention heads, an FFN ratio of 4.0, and RMSNorm on queries and keys. These choices connect language semantics or image appearance to body locations while keeping generation in 3D latent space rather than independently synthesizing multiple views.
Text is encoded by MetaCLIP huge, while scribbles and body-part images use DINOv2 big with registers. The body-part branch receives 5 part images and concatenates their embeddings; examples named in the main text include hair, face, and upper clothing. Training uniformly selects among 3 condition combinations: text-only, image-only, and text-plus-image. When a combination contains images, it further uniformly selects between scribbles and body-part images, supporting different granularities of appearance control. The authors demonstrate sequential editing with text, images, and scribbles, but place the detailed editing algorithm in supplementary material not supplied here. This note therefore establishes the conditioning interfaces and demonstrated capability without inventing latent inversion, local freezing, or a particular resampling procedure.
A Worked Example¶
Consider footage that mainly shows a frontal upper body; this illustrates the paper's problem setting rather than introducing a new experiment. LCA converts 4 body frames and 4 face crops into the shared 8,192 tokens, with potentially reliable face and chest tokens but unreliable back and leg tokens. The compressor preserves those 8,192 positions, while the rendering-contribution mask identifies locations supported by observations. During denoiser training, untrustworthy positions receive placeholders and the loss explains only trustworthy positions; other videos can provide leg or back information. At generation time, a user can supply a face image and clothing description, sample complete latents from noise, restore the tokens, and render with new pose and expression parameters. The complete output comes from a learned prior, not recovery of the true clothing back that was never recorded in the source video.
Loss & Training¶
The complete token dataset contains 1,117,411 identities, comprising 1,113,476 in-the-wild videos, 2,737 studio-captured identities, and 1,198 identities captured through phone-recorded rotations. The authors select 1,000 high-quality captured identities for testing; the total dataset size should not be interpreted directly as the training count after test removal. Sapiens also performs body segmentation and background removal, and each input image has text, scribble, and body-part condition annotations. The compressor trains for 1 day on 32 A100 GPUs with batch size 256; the L1 weight is 1.0, while the KL weight rises from \(10^{-3}\) to \(10^{-2}\) over the first 10K iterations. Its learning rate warms up from \(4\times10^{-10}\) to \(4\times10^{-4}\) over the first 1K iterations. GenLCA uses a rectified-flow conditional flow matching objective with minimum noise parameter \(\sigma_{\min}=10^{-5}\) and supervision restricted by the valid-region mask. The generator trains for 4 days on 64 A100 GPUs with batch size 128, warming up from \(2\times10^{-10}\) to \(2\times10^{-4}\) over the first 1K iterations. Condition tokens are replaced with zeros with probability 0.25 for classifier-free guidance, which is distinct from using learnable placeholders at invalid 3D positions. Generation uses an explicit Euler solver with 50 sampling steps; Figure 5 uses CFG 5.0, which does not establish a shared guidance scale for every experiment.
Key Experimental Results¶
Main Results¶
The text-generation comparison in Table 2 on page 13 uses 50 prompts to generate 50 avatars per method, rendering each avatar from frontal, side, and back views for 3 evaluation images. BLIP-VQA and Text CLIP assess semantic alignment, CLIB-FIQA assesses facial quality, and HyperIQA assesses overall image quality; higher is better for all four. The main text describes Text CLIP as a CLIP-feature comparison between captions predicted from renderings and the original prompts; no further exact definition is available in the cache, so the table follows the source's metric direction. The following selection preserves quality, semantics, distribution distance, and timing; the FID reference consists of 2D diffusion images generated with the same prompts.
| Method | BLIP-VQA ↑ | Text CLIP ↑ | CLIB-FIQA ↑ | HyperIQA ↑ | 2D Diffusion Reference FID ↓ | Inference Time ↓ |
|---|---|---|---|---|---|---|
| TADA | 0.50 | 0.71 | 0.48 | 55.02 | 188.19 | 2.5 h |
| HumanGaussian | 0.62 | 0.73 | 0.39 | 33.61 | 239.33 | 1.2 h |
| DreamWaltz-G | 0.58 | 0.75 | 0.50 | 59.33 | 175.23 | 3.0 h |
| TeRA | 0.42 | 0.67 | 0.44 | 44.01 | 151.80 | 12 s |
| SIGMAN | 0.29 | 0.58 | 0.42 | 56.11 | 280.06 | 3 s |
| GenLCA | 0.64 | 0.76 | 0.55 | 63.05 | 160.91 | 12 s |
The image-conditioned evaluation in the same Table 2 uses 200 images from THuman 2.0 and 200 from HuGe100K, with corresponding ground-truth images as the FID reference. Only SIGMAN and GenLCA are compared because they support image conditioning; other methods are unevaluated rather than assigned zero performance.
| Image-Conditioned Method | THuman 2.0 FID ↓ | HuGe100K FID ↓ |
|---|---|---|
| SIGMAN | 121.40 | 160.48 |
| GenLCA | 96.03 | 76.50 |
Ablation Study¶
Figure 7 and Section 5.3 on page 14 provide qualitative ablations without verifiable per-configuration numerical scores; the table summarizes the authors' observations without invented drops.
| Config | Change | Result Reported in the Paper |
|---|---|---|
| Full model | Retain visibility supervision, learnable placeholders, and in-the-wild data | Illustrated reference for appearance and text alignment |
| Without visibility-aware training | Feed all valid and invalid tokens and compute losses on all of them | Noticeable blurriness and transparency in the back and lower body |
| Without learnable placeholders | Retain visibility handling but replace invalid regions with fixed zero tokens | Unnatural colors |
| Without in-the-wild data | Use only 3,000 indoor-captured identities | Overfitting and poor text-aligned generation |
Key Findings¶
GenLCA achieves HyperIQA 63.05 versus DreamWaltz-G's 59.33, and image-conditioned HuGe100K FID 76.50 versus SIGMAN's 160.48. However, TeRA has the better FID against 2D diffusion images, 151.80 versus GenLCA's 160.91; the authors attribute this to TeRA's closer alignment with the synthetic training distribution. The reported 12 s is generation time, not animation frame rate or a strict speedup measured on independently verified identical hardware; SIGMAN remains faster at 3 s. In Table 3 on page 13, 30 participants each evaluate 10 randomly assigned avatar cases using a 5-point scale while watching rotating videos. GenLCA scores 4.56, 4.68, 4.65, and 4.63 for semantic alignment, multiview consistency, visual quality, and geometric quality, respectively; the last is subjective quality rather than a geometric error measurement. The ablations support distinct roles for supervision reliability, placeholder representation, and data diversity, but do not numerically rank their contributions.
Highlights & Insights¶
- Treating incomplete 3D reconstructions as partially valid supervision better reflects real video acquisition than accepting every pseudo-label as ground truth. Data sources can scale without first repairing all missing regions.
- Shared body query points support decoding while connecting compression, positional embeddings, and visibility masks. Representation design therefore directly determines whether the supervision strategy is practical.
- Input placeholders and output masking address different contamination paths. A transferable lesson is to distinguish missing evidence from genuinely low-valued features.
Limitations & Future Work¶
- The authors explicitly identify inherited LBS constraints on animation; extreme poses, rare prompts, and insufficient coverage still cause failures, with oversmoothed or missing back details.
- The supplied cache contains the main paper and references, not the repeatedly cited supplementary material. Editing implementation, visibility thresholds, and some evaluation details cannot be fully reproduced from this cache.
- Reader assessment: text evaluation uses only 50 prompts, and image FID uses 200 inputs per dataset; these results do not establish equal reliability across all identity, clothing, and pose subgroups.
- Reader assessment: million-scale data and pretrained LCA jointly determine performance, so gains cannot all be attributed to diffusion architecture; video permissions, identity privacy, and data availability also require independent assessment.
- Future work could strengthen reconstruction and animation models and quantify failure rates by body-part coverage, testing visibility handling more directly than merely increasing average generation quality.
Related Work & Insights¶
- vs LCA: LCA reconstructs animatable tokens from observed images, whereas GenLCA learns a sampleable, condition-controlled distribution over that representation. Generation does not replace every reconstruction and animation component.
- vs TADA / HumanGaussian / DreamWaltz-G: These SDS approaches optimize individual assets using 2D priors, whereas this method samples native 3D latents after training; Table 2's generation-time advantage accompanies substantial upfront training cost.
- vs TeRA / SIGMAN: The difference concerns not only network architecture but whether partially observed in-the-wild data can enter training; FID values against different reference distributions require separate interpretation.
- vs MAE: MAE typically reconstructs deliberately hidden content, while these masks identify supervision that was unreliable from the outset. Learnable placeholders do not imply missing-region ground truth exists.
Rating¶
- Novelty: 4/5, combines animatable reconstruction tokens with partially observed generative training, with its main contribution in the training-data paradigm.
- Experimental Thoroughness: 4/5, covers automatic metrics, user studies, and central ablations, but qualitative ablations and limited prompt scale constrain the conclusions.
- Writing Quality: 4/5, presents a clear main argument while deferring editing and underlying reconstruction details to supplementary material.
- Value: 5/5, provides a concrete route to training animatable 3D humans from large-scale real videos, although reproduction requires substantial resources.