Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Joints¶

Conference: ECCV2026
arXiv: 2603.11755
Code: https://zhangcyg.github.io/handcontrolvideo/
Area: Video Generation
Keywords: Egocentric Video Generation, Hand Pose Control, 3D Hand Joints, Occlusion-Aware, Diffusion Models

TL;DR¶

This paper proposes using sparse 3D hand joints as explicit control signals to drive egocentric video generation. Underpinned by three key designs—occlusion-removed context aggregation, 3D depth-weighted occlusion-aware propagation, and 3D geometric embedding—the proposed lightweight module (~20k parameters) built on WAN 2.1 achieves precise video control for fine-grained hand-object interactions, significantly outperforming existing 2D trajectory and implicit pose-based methods.

Background & Motivation¶

Controllable video generation is transitioning from "generating visually appealing content" towards "simulating the visual world model of the physical world." Under this trend, motion-controlled video generation has emerged as a core direction. Especially in egocentric scenarios that require understanding fine-grained hand-object interactions, models must simultaneously maintain high-frequency hand joint motion details, preserve geometric plausibility under frequent occlusions, and strictly follow the spatial consistency of motion trajectories. These three requirements pose fundamental challenges to existing paradigms.

Existing control paradigms can be categorized into two classes. Trajectory-based methods (such as WAN-Move, MotionStream) treat the motion stream as isolated 2D point trajectories, discarding the rigid structural integrity of the hand. They naturally lack depth perception—when fingers cross each other in 3D space, 2D projections cannot distinguish foreground/background layers, often resulting in unrealistic motion details. Pose-based methods (such as EgoControl, Mask2IV) either encode the hand pose into a compressed, low-frequency latent code (which smooths out fine joint details) or project it into 2D skeleton maps or masks (which become a visual mess when fingers overlap), failing to provide clear spatial guidance for the generator. The shared weakness of both approaches is: they lack the explicit 3D geometric understanding required to handle dense occlusions.

The core insight of this paper is straightforward: sparse 3D hand joints are a more suitable control signal for egocentric video generation than 2D trajectories and implicit poses because they possess three key advantages—explicit 3D geometry resolving occlusion ambiguity, a sparse and intuitive interface supporting interactive editing, and cross-morphology generalization (direct transferability to robotic hands). However, directly injecting 3D joints is non-trivial: deciding which joints in the source frame are occluded and should not contribute visual features, resolving feature attribution when foreground fingers cross background fingers in target frames, and preserving 3D structural information within a 2D latent space are all challenges that must be addressed. Core Idea: To build an occlusion-aware, geometry-preserving 3D hand control framework that splits sparse 3D joints into two injection streams—one for occlusion-aware motion feature aggregation and propagation, and another for explicit 3D geometric embedding—precisely driving egocentric video generation with an extremely lightweight module (~20k parameters).

Method¶

Overall Architecture¶

Based on the WAN 2.1 image-to-video (I2V) diffusion Transformer, this work overlays an extremely lightweight 3D hand control module onto its conditional injection channel. The core pipeline is divided into two parallel injection streams:

The first stream is occlusion-aware motion features: it performs local Gaussian context aggregation on the projected location of each 3D hand joint from the VAE latent code of the reference (source) frame, while utilizing a pairwise occlusion penalty to suppress unreliable texture features of occluded joints. These "clean" visual features are then propagated to target frames along 3D joint trajectories, resolving feature conflicts caused by intersecting occlusions through a 3D depth-weighted softmax mechanism.

The second stream is 3D geometric embedding: each joint is associated with both a sinusoidal position encoding of its 3D coordinates (2D position + depth) and a learnable semantic index embedding (instructing the model, e.g., "this is the thumb", "this is the index finger"). These components are concatenated, projected via an MLP, and then densified into geometric feature maps using Gaussian heatmaps. The outputs of both streams are concatenated and processed by a causal 3D convolutional head (ensuring temporal causality), finally being concatenated with the noisy latents and visual conditions to be fed into the LoRA-finetuned DiT blocks.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Source Frame VAE Latent"] --> B["Joint Projection<br/>Gaussian Heatmap"]
    A --> C["Occlusion Penalty<br/>Remove Occlusion Signal"]
    B --> D["Clean Contextual Features"]
    C --> D
    D --> E["Propagate along 3D Trajectory<br/>to Target Frames"]

    F["3D Joint Sequence"] --> G["Sinusoidal Position Encoding<br/>+ Joint Semantic Embedding"]
    G --> H["Gaussian Heatmap<br/>Densification"]
    H --> I["3D Geometric Feature Map"]

    E --> J["Depth-Weighted<br/>3D Softmax Blending"]
    J --> K["Motion + Geometric Features<br/>Concatenation + Causal 3D Conv"]
    I --> K
    K --> L["Concatenate with Noisy Latent"]
    L --> M["LoRA DiT Blocks"]
    M --> N["Generate Video Frames"]

Key Designs¶

1. Occlusion-Removed Context Aggregation (OCA): Extracting joint visual features from the source frame free of occlusion pollution

In egocentric hand-object interactions, severe hand self-occlusion often occurs in the source reference frame. If VAE features are directly sampled from the projected joint positions, an occluded finger might "steal" the texture of a foreground finger. This leads to hallucinations in subsequent generated frames when the occluded finger becomes visible again (e.g., texture of the thumb appearing on the generated index finger). The intuition to address this is highly natural: feature aggregation should focus only on "visible" joints, while suppressing signals from "occluded" ones.

Specifically, this is accomplished in two steps. First, each 3D joint \(\mathbf{J}_i\) is projected using camera intrinsics onto 2D coordinate \(\mathbf{u}_i\) with depth (disparity \(d_i\)), and a Gaussian heatmap \(\mathbf{M}_i\) centered at \(\mathbf{u}_i\) serves as the weight for spatial context aggregation. Second, a pairwise occlusion penalty \(P_{i\leftarrow j}\) is defined as the product of two factors: spatial overlap (the Gaussian distance between the projected positions of two joints) and depth ordering (evaluated via a sigmoid determining if joint \(j\) is closer to the camera than joint \(i\)). If \(P_{i\leftarrow j}\) is close to 1, it means joint \(j\) is almost certainly occluding joint \(i\). Finally, the contextual feature \(\mathbf{f}_i\) of the \(i\)-th joint is weighted as \(\mathbf{f}_i = (1 - \max_{j\neq i} P_{i\leftarrow j}) \cdot \text{GaussianPooling}(\mathcal{Z}_0(\mathbf{u}_i))\). When the visibility coefficient \((1-\max P)\) of a joint approaches 0, its feature contribution is completely suppressed, preventing erroneous texture propagation.

2. Occlusion-Aware Feature Propagation (OP): Resolving trajectory cross-conflict in target frames with 3D depth weighting

After obtaining the clean features \(\mathbf{f}_i\) from the source frame, they need to be propagated along the 3D joint trajectories to each target frame to provide motion guidance. The crucial challenge here is target occlusion: when fingers cross and overlap during motion, the 2D projections of different joints overlap directly on the image plane. A simple summation would blend foreground and background features.

The proposed solution is a 3D weighting mechanism similar to a differentiable Z-buffer. For each target frame \(t\) and joint \(i\), a heatmap \(\mathbf{M}_{i,t}\) is generated at the 2D projected position \(\mathbf{u}_{i,t}\), coupled with the depth \(d_{i,t}\) of the joint in that frame. Then, a softmax weighting with depth bias is computed across all \(N\) joints for each pixel \(\mathbf{x}\):

\[\mathbf{A}_{i,t}(\mathbf{x}) = \text{softmax}_i\left(\log(\mathbf{M}_{i,t}(\mathbf{x})+\epsilon) + \lambda \cdot d_{i,t}\right)\]

where \(\lambda\) is a learnable depth-sharpness parameter. Spatial proximity is determined by the log-heatmap, while the depth term grants higher logit values to joints closer to the camera. When multiple joint projections overlap, the softmax allocates the vast majority of the weight to the joint closest to the camera (i.e., the foreground finger), thereby "penetrating" background joints to correctly preserve foreground features. The final motion condition feature map is a convex combination of all joint features weighted by the softmax values, multiplied by a global opacity mask (demarcating regions where joints reside) for precise spatial localization.

3. 3D Geometric Embedding (3DGE): Explicitly preserving 3D structure and semantic identity in the latent space

Relying solely on propagated visual texture features remains fundamentally 2D: the 3D-to-2D projection inevitably compresses geometric information and obscures the semantic identity of different fingers (e.g., "which finger is making what action"). To address this, the second control stream is introduced to directly inject explicit 3D information into the latent space.

Specifically, for the 3D coordinate of joint \(i\) in frame \(t\) (consisting of 2D position \(\mathbf{u}_{i,t}\) and disparity \(d_{i,t}\)), a sinusoidal position encoding \(\gamma(\cdot)\) maps it to a continuous frequency domain. Meanwhile, a learnable semantic identity embedding \(\mathbf{E}_{id}[i]\) is associated with each joint (telling the model, e.g., "this is the 1st joint, the fingertip of the thumb"). These two elements are concatenated and projected via a shallow MLP \(\phi\) to form the joint embedding \(\mathbf{z}_{i,t}\), which is then "splatted" onto the entire spatial grid using the previously defined Gaussian heatmap \(\mathbf{M}_{i,t}\), yielding a dense geometric feature map \(\mathcal{F}_{geo}\). The motion feature map \(\mathcal{F}_{motion}\) and the geometric feature map \(\mathcal{F}_{geo}\) are concatenated and processed by a causal 3D convolutional layer. Causal convolutions employ asymmetric padding along the temporal dimension, ensuring that geometric guidance for the current frame aggregates only historical information and does not leak future frames. The entire control module contains only ~20k parameters, which is significantly lighter than ControlNet-like alternatives and introduces minimal disturbance to the pre-trained latent space.

Loss & Training¶

The model adopts the continuous-time conditional flow matching of WAN 2.1 as its objective function. LoRA (rank=64) is utilized to finetune the DiT backbone, while the control module parameters are trained from scratch. Training data comprises ~1 million high-quality video clips generated by an automated annotation pipeline on Ego4D, trained on 16 GH200 GPUs for approximately 48 hours. During training, a random joint masking strategy is introduced: 5% of the input joint coordinates are zeroed out, forcing the model to learn to plausibly complete missing joint motions under occlusions or when users perform sparse editing.

Key Experimental Results¶

Main Results¶

The proposed method is compared with mask-based (Mask2IV), 2D skeleton-based (WAN-Fun), and 2D trajectory-based (MotionStream, WAN-Move) approaches on the Ego4D and EgoDex datasets. Both visual quality and 3D hand precision metrics are reported.

Dataset	Metric	Ours	Second Best	Gain
Ego4D	FVD ↓	259.99	303.54 (WAN-Fun)	14%
Ego4D	MPJPE ↓	1.42	9.11 (WAN-Move)	84%
Ego4D	MPVPE ↓	1.70	3.28 (MotionStream)	48%
EgoDex	FVD ↓	174.73	178.67 (WAN-Fun)	2.2%
EgoDex	MPJPE ↓	1.80	5.21 (WAN-Move)	65%
EgoDex	MPVPE ↓	1.82	2.04 (WAN-Move*)	11%

Note: MPJPE is the Mean Per-Joint Position Error (mm) after Procrustes alignment, and MPVPE is the Mean Per-Vertex Position Error. Ours achieves the best or runner-up on all metrics, with a decisive margin particularly on hand precision metrics.

Ablation Study¶

Configuration	Ego4D FVD ↓	Ego4D MPJPE ↓	Notes
OCA only (Occlusion-removed aggregation)	305.49	2.75	Already outperforms WAN-Move*
OCA + OP (+ propagation occlusion-awareness)	300.99	2.10	OP contribution: MPJPE decreases by 23%
OCA + OP + 3DGE (Full model)	259.99	1.42	3DGE contribution: FVD further decreases by 11%, MPJPE by 32%

The three components are progressive: OCA resolves texture pollution caused by source-frame occlusion, OP resolves trajectory cross-conflict during target-frame motion, and 3DGE explicitly encodes 3D structural information; all three are indispensable. Among them, 3DGE contributes most significantly to both visual quality (FVD) and control precision (MPJPE).

Key Findings¶

Compared with training the strong baseline WAN-Move on equivalent data (WAN-Move*), Ours reduces the MPJPE on the EgoDex dataset from 5.21 to 1.80 (a 68% reduction), demonstrating that 3D control signals are vital for complex hand-object interactions.
User studies (2AFC, 30 participants × 30 videos) show that Ours significantly outperforms all baselines in both perceived video quality and motion accuracy.
The method transfers to robotic hands (Unitree G1-Dex3-1, H1-Inspire) with only minor finetuning, validating the morphological invariance of 3D joints—unlike skeleton maps or implicit poses, which encode human-specific kinematic assumptions.

Highlights & Insights¶

Lightweight 3D Control Injection: The entire control module operates with only ~20k parameters (LoRA-finetuned backbone, avoiding a heavy-duty duplicate training copy like in ControlNet), decoupled seamlessly from WAN 2.1. This represents a major practical advantage, allowing the method to be easily integrated into existing generation pipelines.
Elegant and Effective Differentiable Z-buffer Concept: Using a 3D depth-weighted softmax for feature propagation is essentially a Z-buffer depth test from the rendering pipeline. Formulating it as a differentiable module and transplanting it into video generation neatly adapts a classic computer graphics technique to generative modeling.
Interactive Single-Joint Editing: Since the control signal consists of sparse 3D joints, users can drag a single joint (e.g., moving only the thumb) while keeping other joints static. This is infeasible with 2D trajectory methods (where dragging a 2D point affects all motion in the overlapping projection area), providing a brand-new granularity of micro-manipulation for video editing.

Limitations & Future Work¶

Data dependency: the 1 million clips are entirely sourced from Ego4D. The diversity of scenes is limited by the source dataset (primarily daily indoor activities), and generalization ability to outdoor or rapid-motion extreme scenarios remains unknown.
3D joints require extraction by an external upstream model (WiLoR). Cascaded errors in the pipeline (hand detection \(\rightarrow\) MANO reconstruction \(\rightarrow\) tracking \(\rightarrow\) quality filtering) might accumulate in heavily occluded scenarios. Although the paper introduces random mask robustness training, the actual deployment remains heavily dependent on upstream tracking quality.
The current scheme focuses on image-to-video (I2V) generation. When scaling to pure video-to-video (where the input is entirely video with no clean reference frame) or longer-horizon generation, the temporal consistency of occlusion-aware propagation might encounter challenges.

vs WAN-Move / MotionStream (2D Trajectory Methods): These methods represent motion as 2D point trajectories, which works well for general object motions without rigid structures, but loses 3D geometric information in scenarios that involve high degrees-of-freedom hand joints and frequent self-occlusions. The core difference here lies in substituting 2D points with 3D joints and targeted design of the occlusion-handling trio.
vs WAN-Fun / Mask2IV (2D Pose/Mask Methods): These methods project hand poses into 2D skeleton maps or mask maps as control signals, suffering from topological ambiguity (unable to distinguish layers when fingers cross) and loss of depth information. Utilizing 3D joints directly evades these information compression losses.
vs EgoControl (Implicit Pose Encoding): EgoControl encodes pose into compressed latent codes whose low-frequency nature flattens fine joint details. This work preserves high-frequency spatial accuracy through explicit encoding of 3D coordinates coupled with semantic embeddings.

Rating¶

Novelty: ⭐⭐⭐⭐ Using sparse 3D hand joints to control video generation is not proposed for the first time, but the paper targets specific occlusion challenges in egocentric scenarios by designing the three-component suite (OCA + OP + 3DGE). The solution is comprehensive with a clear motivation, and the "differentiable Z-buffer" concept is elegant and inspiring.
Experimental Thoroughness: ⭐⭐⭐⭐⭐ Evaluated against 5 baselines on 2 datasets, covering visual quality, 3D hand accuracy, user studies, ablations, cross-morphology transfer, and interactive editing. The overall evaluation design is remarkably thorough.
Writing Quality: ⭐⭐⭐⭐⭐ The problem definition is clear, the motivational chain is robust, and the explanation of methods is detailed (with figures, math equations and ablations tightly linked). It is a highly readable paper.
Value: ⭐⭐⭐⭐ Provides a plug-and-play 3D control module for egocentric video generation, and both the code and automated annotation pipeline are open-sourced. A million-scale annotated dataset is contributed to the Ego4D community, ensuring high practical utility.