LOOM: Weaving Geometry-Consistent Human-Object Interaction Videos via Progressive Curriculum Learning

Conference: ECCV 2026
Paper: ECCV Paper
Code: https://neutrinoliu.github.io/byteloom/
Area: 3D Vision / Video Generation / Human Understanding
Keywords: Human-Object Interaction (HOI), Video Generation, Relative Coordinate Maps (RCM), Curriculum Learning, Diffusion Transformer (DiT)

TL;DR

ByteLOOM tackles geometric distortion and hand mesh dependency in human-object interaction video synthesis by introducing an RCM-cache mechanism to formulate view generation as spatial texture retrieval, combined with a progressive three-stage curriculum that yields highly consistent 6-DoF interaction videos conditioned only on simplified 2D skeletons.

Background & Motivation

Synthesizing realistic human-object interaction (HOI) videos is crucial for digital humans, e-commerce product demonstrations, advertising, and robotics imitation learning. While recent diffusion video generation models excel at animating human poses or rendering static backgrounds, generating dynamic scenes involving complex physical manipulations—such as rotating, inverting, and handling everyday objects—remains a severe bottleneck. Under large-angle rotations, existing models frequently suffer from severe geometric distortion, surface texture collapse, or failure to render unseen object facets.

This failure stems from two fundamental tensions in existing frameworks. First, prior methods (e.g., AnchorCrafter, DreamActor-H1, HunyuanVideo-HOMA) condition the diffusion backbone either on a single front-facing object image or on reference views restricted to an extremely narrow camera cone (e.g., ~30°). Lacking a persistent multi-view 3D prior, diffusion models are forced to hallucinate unobserved faces. Naively feeding multiple reference images fails because the attention layers treat them as independent entities and interpolate between them inconsistently across time. Second, high-quality HOI datasets with paired whole-body motion, hand meshes, and 3D object annotations are exceedingly scarce. To resolve occlusion and contact boundaries, prior works relied heavily on expensive 3D hand mesh templates and dense depth sensors, preventing scalable data collection in the wild.

The key insight of this paper is that dense 3D geometric correspondences can be encoded into continuous spatial coordinate maps, transforming ill-posed novel view synthesis into an explicit texture lookup problem, while the data scarcity bottleneck can be bypassed by leveraging diverse, task-specific datasets along an incremental learning trajectory. Core idea: introduce an RCM-cache mechanism as a persistent 3D geometric prior for precise 6-DoF rigid manipulation, coupled with a three-stage curriculum (pose pretraining → hand-object contact → whole-body HOI fine-tuning) that eliminates the need for 3D hand meshes.

Method

Overall Architecture

ByteLOOM is built upon the Wan2.1 Diffusion Transformer (DiT) backbone. The system receives multiple inputs: a reference human image, per-frame 2D human skeletons extracted via DWPose, a persistent RCM-cache of sparse-view textured images and corresponding Relative Coordinate Maps (RCM), and target per-frame 6-DoF RCM condition maps. These conditions are fused with the noisy video latents via channel concatenation and an MLP fuser, empowering the DiT self-attention mechanisms to learn spatial correspondence, cross-view texture binding, and realistic occlusion.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Multimodal Inputs<br/>Human Ref + 2D Pose + 3D Object Mesh"] --> B["RCM-Cache Construction<br/>Pairing Sparse-View RGB with Normalized RCMs"]
    B --> C["Per-Frame 6-DoF Pose Conditioning<br/>Rasterizing Target RCMs into Latent Channels"]
    C --> D["Three-Stage Curriculum Learning<br/>Pose Pretraining → Hand-Object → Whole-Body HOI"]
    D --> E["Depth-Free Data Curation Pipeline<br/>SAM2 Masking + MoGe-2 Scale Calibration + FoundationPose"]
    E --> F["Geometry-Consistent HOI Video<br/>Smooth Motion & Faithful Multi-View Manipulation"]

Key Designs

1. RCM-cache Mechanism: Reducing Novel View Synthesis to Spatial Texture Retrieval
To overcome geometric tearing and identity collapse when rotating objects, ByteLOOM introduces Relative Coordinate Maps (RCM). Given the 3D vertices \(V_i \in \mathbb{R}^3\) of an object mesh and its axis-aligned bounding box bounded by diagonal corners \([b_{\min}, b_{\max}]\), normalized 3D coordinates are computed via component-wise scaling \(C_i^{\text{RCM}} = (V_i - b_{\min}) / (b_{\max} - b_{\min}) \in [0, 1]^3\) and quantized to RGB values \(c_i = \text{round}(255 \times C_i^{\text{RCM}})\). During rasterization, the color of any pixel inside a triangle face is linearly interpolated from its vertex coordinates using barycentric weights \((\alpha, \beta, \gamma)\): $\(c_{\text{pixel}} = \alpha c_i + \beta c_j + \gamma c_k\)$ Every pixel in the RCM explicitly corresponds to a calibrated physical location on the object surface. ByteLOOM renders 9 representative sparse viewpoints of textured RGB images paired with their RCMs (matching the \(4k+1\) requirement of the Wan-VAE) and caches them along the temporal/frame dimension of the latent sequence as a global appearance anchor. During video denoising, per-frame target RCMs rendered from the desired 6-DoF trajectory are injected as spatial control signals. Because identical surface points share identical RCM colors across all frames and viewpoints, the DiT self-attention layers seamlessly establish point-to-point correspondences between the target frame and the cached reference views, reducing novel view generation to reliable texture retrieval and eliminating geometric distortion.

2. Three-Stage Progressive Curriculum Learning: Decoupling Motion and Contact Dynamics
To tackle the scarcity of annotated full-body HOI footage without relying on 3D hand mesh templates, ByteLOOM exploits existing specialized datasets through a staged curriculum: - Curriculum I (Human Pose Pretraining): Pretraining on a large proprietary video dataset containing over 100K clips (~6.4M frames) annotated with DWPose skeletons. This stage instructs the DiT backbone in robust human anatomical kinematics, temporal coherence, and broad spatial motion priors without object distraction. - Curriculum II (Hand-Object Pretraining): Fine-tuning on rich, dedicated hand-object manipulation benchmarks (DexYCB, HO3D, ARCTIC, totaling ~1.36M frames) equipped with 3D object meshes, object poses, and accurate hand interactions. Conditioned on RCMs, the model learns fine-grained grasping configurations, physical contact surfaces, and intricate mutual occlusions between fingers and manipulated surfaces. - Curriculum III (Full Human-Object Interaction Fine-tuning): Final optimization on Mani4D-Train (45K frames), a studio-quality dataset featuring actors rotating, inverting, and demonstrating daily objects across full-body poses. This stage unifies whole-body temporal coordination with localized hand-object dexterity, enabling realistic occlusion synthesis guided solely by 2D stick-figure poses.

3. T-SSIM: Feed-Forward 3DGS Metric for Quantifying Multi-View Consistency
Evaluating geometric consistency across video frames historically relied on pairwise feature distances like MEt3R, which incurs an expensive \(\mathcal{O}(N^2)\) computational cost and cannot verify global 3D integrity. ByteLOOM introduces T-SSIM, a reference-free geometric metric grounded in 3D Gaussian Splatting (3DGS). Specifically, SAM2 segments the manipulated object from the generated video frames \(I_0, \dots, I_n\). An off-the-shelf feed-forward reconstruction model, E-Rayzer, predicts a unified 3D Gaussian field \(\mathcal{G}\), shared intrinsic matrix \(K\), and per-frame camera extrinsics \(E_i\): $\(\{\mathcal{G}, K, E_0, \dots, E_n\} = \text{E-Rayzer}(I_0, \dots, I_n)\)$ The predicted 3D Gaussians are splatted back to each frame's estimated viewpoint to render synthetic projections \(I'_i = \text{Splat}(\mathcal{G}, K, E_i)\). The temporal average of structural similarity (T-SSIM) measures geometric coherence: $\(\text{T-SSIM} = \frac{1}{n}\sum_{i=1}^{n} \text{SSIM}(I_i, I'_i)\)$ If the generated object deforms or suffers from texture drifting, the single 3D Gaussian model cannot fit the inconsistent views, causing severe reprojection degradation and lowering T-SSIM.

4. Depth-Free Data Curation Pipeline for Scalable Annotation
To annotate in-the-wild video recordings without specialized motion capture or depth hardware, ByteLOOM deploys an automated multi-stage pipeline: foreground object segments are isolated via SAM2 and converted into watertight manifold meshes using 2DGS/PGSR or ReconViaGen (with Amodal3R handling occluded frames). Metric depth and camera intrinsics are extracted via MoGe-2 and calibrated across frames with RANSAC. Finally, FoundationPose tracks the 6-DoF trajectory of the reconstructed mesh using the aligned depth and RGB sequences, providing multi-view rotation coverage spanning up to 180°.

Key Experimental Results

Main Results

Quantitative evaluations are conducted on Mani4D-Test, consisting of 15 video sequences (97 frames each at 15 FPS, featuring rotations between 90° and 180°) across 5 actors, where 6 test objects are completely unseen during training.

(From original paper Table 2: Quantitative results on Mani4D-Test)

Method	Obj-IoU ↑	Obj-CLIP ↑	Face-Cos ↑	LMD ↓	MEt3R ↓	T-SSIM ↑
Ground Truth	–	–	–	–	0.0419	0.9218
UniAnimate-DiT	0.4644	0.7655	0.7920	0.3260	0.0524	0.9101
MimicMotion	0.4647	0.7455	0.7391	0.2017	0.0638	0.9010
AnchorCrafter	0.6461	0.7355	0.5771	0.2629	0.0552	0.9079
Ours (I + II + III)	0.8288	0.9100	0.8891	0.1427	0.0454	0.9147

Ablation Study

(From original paper Table 3 and Table 2 lower section: Component and Curriculum Ablations)

Config	Obj-IoU ↑	Obj-CLIP ↑	LMD ↓	MEt3R ↓	T-SSIM ↑	Note
Full model (with RCM)	0.8288	0.9100	0.1427	0.0454	0.9147	Best balance of geometry consistency and hand contact
without RCM	0.6619	0.8639	0.1878	0.0540	0.9063	Omits RCM; relies solely on RGB multi-view images
Curriculum: I + III (skip hand-object)	0.7627	0.8829	0.2054	0.0569	0.9048	Hand landmark error (LMD) increases sharply
Curriculum: I + Obj + III	0.7689	0.8901	0.1930	0.0492	0.9105	Synthetic object-drift task offers marginal gain
Curriculum: I + II + Obj + III	0.7770	0.8946	0.1861	0.0429	0.9183	Higher T-SSIM but synthetic domain gap hurts hand quality

Key Findings

• RCM is vital for 3D geometric stability: Ablating RCM (Table 3) causes Obj-IoU to plummet from 0.8288 to 0.6619 and worsens MEt3R from 0.0454 to 0.0540. Without coordinate grounding, the DiT fails to associate disparate camera angles with the same physical object, resulting in severe shape twisting during frame synthesis.
• Curriculum II bridges the whole-body to contact gap: Skipping Stage II (I+III) degrades hand landmark accuracy (LMD surges from 0.1427 to 0.2054). Whole-body motion data alone cannot provide sufficient grasp supervision; localized hand-object interaction datasets are indispensable for teaching realistic finger-surface boundaries.
• Trade-offs of synthetic object novel view synthesis: Training on synthetic object-drift sequences (I+II+Obj+III) improves raw multi-view reconstruction (T-SSIM 0.9183 vs. 0.9147), but introduces domain discrepancy that degrades hand contact fidelity, rendering it an optional curriculum.
• Superior generalization to novel subjects and unseen objects: In zero-shot inference with novel human references (Table 4), ByteLOOM achieves an Obj-IoU of 0.8212 and Face-Cos of 0.7661, outperforming AnchorCrafter (0.2435 and 0.3700) by a wide margin.

Highlights & Insights

• Formulating 3D novel view synthesis as spatial texture retrieval: Encoding object coordinates into continuous RGB space enables standard 2D/3D Transformer self-attention layers to perform explicit surface lookups, resolving multi-view geometric consistency without designing bespoke 3D architectures.
• Progressive curriculum bridging heterogeneous domains: Decoupling full-body kinematics, micro-scale hand grasping, and complete HOI synthesis enables high-fidelity generation using merely 45K frames of annotated whole-body interaction video.
• Reference-free geometric consistency quantification via 3DGS: T-SSIM establishes an elegant feed-forward benchmarking protocol that evaluates whether an animated object maintains a valid rigid 3D manifold across time without requiring paired ground truth videos.

Limitations & Future Work

• Restricted to studio-style showcase scenarios: Experiments are primarily scoped to third-person allocentric product showcase settings. In-the-wild interactions featuring heavy self-occlusions, functional tool usage (e.g., cutting or writing), and multi-object manipulation remain unexplored.
• Assumption of rigid object topology: RCM formulation assumes invariant mesh topology; deformable, articulated, or soft objects (e.g., folding clothing or flexible containers) break static coordinate-to-texture mapping and require part-level or deformation-aware coordinate representations.
• Absence of explicit physics or contact priors: Conditioned strictly on geometry and kinematics, the framework does not enforce physical non-penetration constraints. Severe mismatches between human poses and object meshes can result in finger penetration or floating artifacts.

Related Work & Insights

• vs. AnchorCrafter: AnchorCrafter relies on complex multi-condition bundles including hand meshes, depth maps, and skeletons, yet its object references remain confined to narrow 30° cones, leading to geometric collapse under large rotations. ByteLOOM eliminates depth maps and hand meshes while handling full 180° 6-DoF rotations via RCM-cache.
• vs. DreamActor-H1 & HunyuanVideo-HOMA: DreamActor-H1 depends heavily on extensive MoCap template libraries, while HunyuanVideo-HOMA uses weak conditioning but fails to showcase diverse object viewpoints. ByteLOOM achieves superior controllability and cross-view fidelity using a lightweight curriculum and 2D skeleton guidance.

Rating

• Novelty: ⭐⭐⭐⭐⭐ [Injecting RCM as an explicit 3D prior for DiT-based video synthesis is both elegant and highly effective]
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ [Comprehensive benchmarking on Mani4D, detailed curriculum ablations, and zero-shot novel subject evaluations]
• Writing Quality: ⭐⭐⭐⭐⭐ [Clear motivation, well-articulated method, and rigorous architectural rationale]
• Value: ⭐⭐⭐⭐⭐ [Provides a practical, highly scalable blueprint for digital humans, virtual e-commerce, and robotics imitation data generation]

Related Papers

• [ECCV 2026] ObjectForesight: Predicting 3D Object Trajectories from Human Videos
• [CVPR 2026] CrossHOI: Learning Cross-View Representations for Monocular 3D Human-Object Interaction Reconstruction
• [ECCV 2026] Interaction-Aware 4D Gaussian Splatting for Dynamic Hand-Object Interaction Reconstruction
• [CVPR 2026] ReGenHOI: Unifying Reconstruction and Generation for 3D Human-Object Interaction Understanding
• [ECCV 2026] Reconstructing Humans and Objects in Interaction using Large Reconstruction Models