AHOY! Animatable Humans under Occlusion from YouTube Videos with Gaussian Splatting and Video Diffusion Priors¶

Conference: ECCV 2026
arXiv: 2603.17975
Code: To be confirmed
Area: Video Generation / 3D Vision / Human Reconstruction
Keywords: 3D Gaussian Avatar, Human Reconstruction, Video Diffusion Models, Occlusion Reconstruction, Animatable Avatar

TL;DR¶

AHOY reconstructs complete, animatable 3DGS avatars from heavily occluded monocular YouTube videos. It first leverages DensePose + FLUX + multi-view diffusion to build a coarse canonical avatar, uses a LoRA-fine-tuned Wan 2.2 video diffusion model via RF-Inversion to generate multi-view "hallucinated" supervisory videos, and finally employs map-pose/LBS-pose decoupling to absorb inconsistencies in the generated data, training a high-fidelity Gaussian avatar with pose-dependent deformations.

Background & Motivation¶

Reconstructing re-posable 3D human avatars from monocular video is a long-standing goal in computer vision and graphics. Recent works in NeRF and 3DGS are capable of generating high-fidelity avatars that can be driven to new poses in real time. However, these works share a common underlying prerequisite: the subject must be fully visible in the video, typically starting in a frontal canonical pose. This assumption excludes the vast majority of real-world videos—such as billions of daily activity videos on YouTube (cooking, sports, conversation) where the human body is heavily occluded by furniture, other people, or the scene itself, meaning large body areas never appear in the footage, and only a single view is available per frame. Being able to reconstruct complete avatars from such videos would unlock an unlimited source of human appearance data, bypassing the need for controlled capture environments, multi-view camera arrays, or the active cooperation of subjects.

However, reconstructing full avatars from occluded monocular videos faces three intertwined challenges. First, occluded regions are never observed and lack direct supervision signals; the model must "hallucinate" unseen body surfaces. Second, high-fidelity avatars require pose-dependent deformations (e.g., changes in clothing wrinkles during bending, unfolding underarms when raising hands), but occluded videos provide only a single view per pose, making it impossible to obtain the multi-view supervision required to train deformations. Third, even if video diffusion models are used to fill in unobserved regions, they sample and generate each video independently. Lacking strict geometric consistency across views and frames, direct training on these generated videos as multi-view supervision causes severe blurriness and artifacts.

The core idea of this work is to use an identity-fine-tuned video diffusion model as a "hallucination bridge": first embedding coarse avatar renderings into the latent space of the diffusion model and decoding them back, utilizing identity priors to fill in unobserved regions while preserving the body layout; then, through the decoupling of map-pose and LBS-pose, the "pose determining Gaussian appearance" and the "pose determining Gaussian deformation" are separated to absorb multi-view inconsistencies in the generated data, ultimately training a complete pose-dependent Gaussian avatar from the "hallucinated" videos.

Method¶

Overall Architecture¶

AHOY's pipeline is divided into four progressive stages, moving from partially observed, occluded videos to complete animatable avatars. Stage 1 establishes a coarse canonical avatar from partial observations: visible pixels are accumulated frame-by-frame onto a canonical texture atlas using DensePose UV mapping, FLUX completes unobserved regions, and a multi-view diffusion model generates 4 canonical-view RGB images from a single completed image, which are then used to train a time-invariant canonical Gaussian avatar (without modeling pose-dependent deformation). Stage 2 generates "hallucinated" supervisory videos: the Wan 2.2 video diffusion model is fine-tuned via LoRA using the identity information from the input video. The coarse avatar is then rendered into carefully designed structured motion sequences (360-degree rotation + specific movements like raising hands, sitting), and these coarse renderings are embedded into the diffusion latent space via RF-Inversion and decoded back. This yields high-quality hallucinated videos with removed artifacts and filled-in unobserved regions. Stage 3 upgrades this to a complete avatar: utilizing the "same-pose, multi-view" coverage in the hallucinated videos (where frames in rotation sequences approximate the same pose), a pose-dependent Gaussian avatar is trained. Meanwhile, map-pose/LBS-pose decoupling and frame-by-frame pose/camera fine-tuning corrections are introduced to absorb diffusion model inconsistencies, and a FLAME head branch is used to guarantee facial identity. Stage 4 drives the trained avatar using new poses constrained by PCA projections, synthesizing it into a 3DGS scene captured by a smartphone.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Occluded Monocular Video<br/>Frame Sequence + NLF Pose Estimation"] --> B1["Stage 1a: UV Texture Mapping<br/>DensePose Frame Accumulation → FLUX Completion<br/>→ Single Canonical Image"]
    B1 --> B2["Stage 1b: Multi-View Canonical Images<br/>Multi-View Diffusion Generates 4-View RGB<br/>(Front/Back/Left/Right)<br/>→ Canonical Gaussian Avatar<br/>StyleUNet Predicts Attribute Offsets<br/>LBS Deformation"]
    B2 --> C1["Stage 2a: Identity Fine-Tuning Diffusion<br/>Wan 2.2 + LoRA<br/>Fine-Tune Input Video Identity"]
    C1 --> C2["Stage 2b: Structured Motion Rendering<br/>Coarse Avatar → 360° Rotation + Static Action<br/>→ Coarse Rendered Video"]
    C2 --> C3["Stage 2c: RF-Inversion Refinement<br/>Coarse Rendering → Forward Rectified Flow ODE Encoding<br/>→ Backward Decoding → Hallucinated Video<br/>(Preserve Layout + Fill Textures)"]
    C3 --> D1["Stage 3a: Pose-Dependent Gaussian Avatar<br/>StyleUNet Input Upgraded From Canonical<br/>to Pose-Dependent Map"]
    D1 --> D2["Stage 3b: Map/LBS Pose Decoupling<br/>map-pose Determines Gaussian Appearance (Shared in Sequence)<br/>LBS-pose Determines Deformation (Independent per Frame)<br/>+Δθ/Δγ to Absorb Inconsistencies"]
    D2 --> D3["Stage 3c: Head-Body Separation<br/>Body Uses Hallucinated Video (With Δ Corrections)<br/>Head Uses CAP4D Face Diffusion<br/>+ Original Observation Frame Supervision + Reg"]
    D3 --> E["Stage 4: Animation & Scene Composition<br/>New Poses → PCA Projection Constraints<br/>→ LBS Deformation → 3DGS Scene Synthesis"]

Key Designs¶

1. Hallucination as Supervision: RF-Inversion Fills Unobserved Regions

The core challenge of reconstruction under occlusion is that occluded regions are never observed, leaving no direct supervision. The proposed approach is coarse-to-fine: based on partial observations, a coarse canonical avatar is trained that understands body layout and pose, but carries rough artifacts in unobserved areas. This coarse avatar is rendered under structured motion sequences (a 360-degree rotation paired with specific actions), where the renderings retain correct body poses and silhouettes. The key step is passing these coarse renderings into the identity-fine-tuned diffusion model for RF-Inversion—the forward rectified flow ODE encoding preserves spatial layout, while the backward decoding uses the LoRA-fine-tuned identity prior to fill in missing texture details. The resulting "hallucinated" videos cover previously unseen body surfaces (back, underarms, etc.), providing dense supervision from multiple views and poses. Ablation studies show that removing RF-Inversion causes the novel-view PSNR to drop from 24.12 to 21.10 (-3dB) and novel-pose PSNR to drop from 22.81 to 19.40, validating the indispensability of this step.

2. Two-Stage Progressive Architecture from Canonical to Pose-Dependent Gaussian Avatar

High-fidelity avatars must model pose-dependent deformations—wrinkles and contours of clothing change depending on whether a person bends, raises their hands, or sits down. However, occluded monocular videos cannot provide the multi-view coverage required to train such dependencies. The key insight is to temporarily simplify the problem: during the first stage, pose-dependent modeling is intentionally omitted, training only a canonical Gaussian avatar to memorize the human body's textures and structure. Once the second stage generates the hallucinated videos, particularly the designed "rotation" sequences, the problem is qualitatively transformed—the same action is observed from 360 degrees, which effectively provides multi-view supervision for that pose. In the rotation sequence, the map-pose remains unchanged (e.g., "hands raised"), while the LBS-pose handles only the deformation from the 360-degree rotation, thereby providing sufficient supervisory signals for the pose-dependent Gaussian avatar. In the ablation study, using only the coarse avatar (omitting the second stage and pose-dependent modeling) causes the novel-view PSNR to plummet from 24.12 to 16.10 (-8dB), proving the necessity of this upgrade.

3. Map-pose / LBS-pose Decoupling: Absorbing Multi-View Inconsistencies

Hallucinated videos are generated independently by the diffusion model, lacking strict geometric consistency across views and frames. Directly using them as ground-truth multi-view supervision would lead to rendering artifacts. The solution lies in separating the two roles of pose: the map-pose, which determines "what the Gaussian attributes look like," and the LBS-pose, which determines "how the Gaussians deform into the frame." The map-pose is fed as input into the StyleUNet and is shared within the same rotation sequence—for a "hands raised" motion, the map-pose remains constant, and StyleUNet outputs the same set of Gaussian attributes. Meanwhile, the LBS-pose varies per frame, handling deformations introduced by body rotations. Furthermore, fine-tunable small corrections \(\Delta\theta\) and \(\Delta\gamma\) are added to the LBS-pose and camera parameters of each frame to absorb diffusion model inconsistencies. Ablations indicate that removing this decoupling drops the novel-view PSNR by 1.5dB (24.12→22.60) and the novel-pose PSNR by 2.2dB, as deformation quality directly impacts novel-pose generalization.

4. Head-Body Separate Supervision: Dual Diffusion Protects Facial Identity

Heads generated by video diffusion models often suffer from cross-frame identity inconsistency—while LoRA fine-tuning preserves clothing and body shape, facial details remain unstable. The proposed solution divides the avatar Gaussians into head and body sections based on SMPL-to-FLAME correspondence. The body is supervised using the masked pixels of the hallucinated videos: \(\mathcal{L}_{body} = \sum\|(\tilde{\mathbf{I}} - \hat{\mathbf{I}})\odot(1-\mathbf{M}_{head})\|_1\). The head is supervised using identity-consistent frontal/profile images generated by the CAP4D multi-view face diffusion model, optimizing a dedicated FLAME deformation branch for head Gaussians. The motivation behind this separation is pragmatic: expecting a single video diffusion model to simultaneously handle body textures and facial identity is unreliable, whereas dividing the tasks between two specialized diffusion models is much more effective. Ablation shows that omitting head-body separation drops the novel-view PSNR by 0.5dB, though the visual change in identity consistency is far more pronounced than the numerical metric suggests.

Loss & Training¶

The complete training objective consists of five terms: \(\mathcal{L} = \lambda_{body}\mathcal{L}_{body} + \lambda_{obs}\mathcal{L}_{obs} + \lambda_{head}\mathcal{L}_{head} + \lambda_{per}\mathcal{L}_{per} + \lambda_{reg}\mathcal{L}_{reg}\). \(\mathcal{L}_{body}\) provides L1 supervision on the body region using the hallucinated videos, \(\mathcal{L}_{obs}\) supervises the visible regions of the original observed frames (masking out occluding objects), \(\mathcal{L}_{head}\) supervises head Gaussians using CAP4D face images, \(\mathcal{L}_{per}\) is the LPIPS perceptual loss, and \(\mathcal{L}_{reg}\) regularizes skinning weights toward the SMPL defaults. The \(\lambda\) coefficients serve as hyper-parameters to balance the weights.

Key Experimental Results¶

Main Results¶

Dataset	Setting	Metric	Ours	Prev. SOTA	Gain
BEHAVE	Novel View	PSNR↑	24.12	19.34 (SyncHuman)	+4.78
BEHAVE	Novel View	SSIM↑	0.905	0.831 (SyncHuman)	+0.074
BEHAVE	Novel View	LPIPS↓	0.090	0.172 (SyncHuman)	-0.082
BEHAVE	Novel Pose	PSNR↑	22.81	16.93 (LHM)	+5.88
BEHAVE	Novel Pose	SSIM↑	0.889	0.774 (LHM)	+0.115
BEHAVE	Novel Pose	LPIPS↓	0.110	0.238 (LHM)	-0.128
YouTube	Animation (Occluded Input)	PSNR↑	22.81	19.03 (LHM)	+3.78
YouTube	Animation (Canonical Input)	PSNR↑	22.83	20.17 (LHM)	+2.66
YouTube	Static (Occluded Input)	PSNR↑	22.01	19.82 (SyncHuman)	+2.19
YouTube	Static (Canonical Input)	PSNR↑	23.05	22.03 (SyncHuman)	+1.02

Ablation Study¶

Configuration	BEHAVE Novel View PSNR	BEHAVE Novel Pose PSNR	Description
AHOY (Full)	24.12	22.81	Full pipeline
(A) Coarse Avatar Only	16.10 (-8.02)	14.30 (-8.51)	Skip hallucination generation and full avatar stages
(B) W/o RF-Inversion	21.10 (-3.02)	19.40 (-3.41)	Directly use coarse rendering as supervision
(C) W/o Map/LBS Decoupling	22.60 (-1.52)	20.60 (-2.21)	No frame-by-frame pose correction optimization
(D) W/o Head-Body Separation	23.60 (-0.52)	22.20 (-0.61)	Head also supervised by Wan video

Key Findings¶

RF-Inversion is the largest contributor to performance: removing it drops the PSNR by more than 3dB, validating the necessity of the "coarse rendering followed by diffusion refinement" hallucination bridge design.
The contribution of map/LBS decoupling is more pronounced under the novel-pose setting than under the novel-view setting (-2.21 vs -1.52), since deformation quality directly affects novel-pose generalization, and the decoupling mechanism directly determines the quality of deformation corrections.
The quantitative contribution of head-body separation is relatively small (0.5dB), but it substantially improves visual identity fidelity—qualitative differences on faces are much more distinct than indicated by PSNR differences, serving as a reminder that avatar reconstruction evaluation should not rely solely on metrics.
Even when baseline methods are provided with unoccluded canonical images (the optimal input scenario), AHOY still significantly outperforms them, demonstrating that the full 3DGS avatar pipeline (pose-dependent Gaussian avatar + multi-stage progressive training + multi-view diffusion supervision) has inherent structural advantages.

Highlights & Insights¶

"Three-stage progressive training over single-step regression" is the best practice for reconstruction under severe occlusion—rather than forcing a model to infer a complete avatar in a single forward pass, the pipeline proceeds via canonical → coarse rendering → RF-Inversion → pose-dependent Gaussian avatar, ensuring clear and reasonable supervisory signals and constraints at every step.
RF-Inversion for avatar supervision is an elegant design reuse pattern—the coarse rendering provides correct body layouts (geometric priors) while the diffusion model provides plausible texture details (appearance priors). These two strong priors complement each other, achieving much more controllable and stable results than direct multi-view generation.
Map-pose/LBS-pose decoupling is an elegant solution to address multi-view inconsistencies in generated data—by separating the "appearance" and "deformation" roles of a pose, this design concept can be extended to other scenarios utilizing generative models for 3D supervision (such as NeRF-from-diffusion, etc.).
The head-body separation design reflects a pragmatic divide-and-conquer philosophy—instead of forcing a single video diffusion model to simultaneously handle body textures and facial identity (tasks of different difficulties and requirements), it assigns them to the most suitable specialized tools.

Limitations & Future Work¶

The authors acknowledge that identity-fine-tuned diffusion models may "hallucinate" plausible but incorrect details—for regions that are never observed, the generated results lack geometric constraints to guarantee truthfulness.
The pipeline requires optimization across multiple stages (LoRA fine-tuning + multi-view generation + structured motion rendering + RF-Inversion × N videos + full avatar training), making it far slower than feed-forward methods and hard to apply in online scenarios.
The final avatar quality is bounded by the fidelity of the video diffusion model—while advancements in Wan 2.2 / HunyuanVideo will naturally improve results, the reliability of current generative models remains a bottleneck.
The method requires a minimum number of visible frames of the target subject in the input video to support LoRA fine-tuning; if occlusion is extremely severe (e.g., the person appears for only a few frames), establishing an effective identity prior is difficult.

vs LHM / IDOL: Feed-forward single-image animatable avatar methods are fast but assume unoccluded inputs. AHOY reconstructs from occluded videos; though slower, it generates full avatars under severe occlusion. The two are complementary, and AHOY's outputs can serve as high-quality "de-occluded" training data for feed-forward methods.
vs SyncHuman / PSHuman: Static reconstruction methods that do not produce animatable avatars. AHOY's core advantage is that its output avatars can be driven to novel poses, which static methods cannot achieve.
vs Vid2Avatar / Vid2AvatarPro: Reconstructs avatars from unoccluded videos but cannot handle occlusions; AHOY fills this gap.
vs Dream-Lift-Animate / AdaHuman / MoGA: Also reconstructs avatars from incomplete inputs but lacks public code. AHOY offers a reproducible solution and provides thorough evaluations on real-world occluded YouTube videos.
vs VIDA (Nazarczuk et al.): Uses video diffusion as 3DGS supervision in sparse-view settings, sharing a similar line of thought but focusing on different problems. AHOY extends this to occluded monocular avatar reconstruction.

Rating¶

Novelty: ⭐⭐⭐⭐⭐ [Systematizes "diffusion model hallucination as avatar supervision" into a complete four-stage pipeline, with map-pose/LBS-pose decoupling stands as an elegant, original design, opening up a new direction for animatable avatar reconstruction from YouTube-level real-world occluded videos.]
Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Evaluated on 50 YouTube videos + BEHAVE multi-view quantitative evaluation + two input configurations (occluded vs. canonical) + four core ablations verified individually + comprehensive coverage across animation/static/novel-view/novel-pose settings, presenting a rigorous and complete experimental design.]
Writing Quality: ⭐⭐⭐⭐⭐ [The motivation progresses logically, the method unfolds progressively from coarse to fine, and the design motivation and role of each stage are explained clearly, making even a complex pipeline easy to comprehend.]
Value: ⭐⭐⭐⭐⭐ [Unlocks YouTube-level real-world human data as a source for 3D avatar assets, holding immense potential for games, VR, and film, with highly extensible methodological designs.]