HO-Flow: Generalizable Hand-Object Interaction Generation with Latent Flow Matching

Conference: ECCV2026
Paper: ECCV Paper
Project: HO-Flow
Area: Human Understanding
Keywords: hand-object interaction, motion generation, interaction-aware latents, autoregressive generation, flow matching

TL;DR

HO-Flow compresses hand motion, object trajectories, and local contact geometry into temporal latents, then generates continuous motions with context-aware autoregressive flow matching, increasing GRAB Phy from LatentHOI's 96.16% to 98.25% and sample diversity SD from 0.13 to 0.31.

Background & Motivation

A plausible grasp requires hand-object contact at one instant; a complete handover or controller manipulation requires that relationship to remain coherent throughout motion. Individually plausible finger rotations, global hand translations, and object trajectories can still combine into floating hands, interpenetration, or sudden jumps. The central task is therefore temporal coordination between human and object motion, rather than generic 3D shape generation.

Existing approaches address parts of this problem. DiffH2O generates explicit pose sequences but must jointly handle position and rotation variables with different structures. LatentHOI compresses local hand poses into a latent space, yet its primarily frame-wise representation does not fully unify global motion. Another approach uses discrete motion tokens for autoregressive prediction, making historical context accessible but potentially losing continuous details needed for precise contact. HO-Flow aims to preserve those details while making subsequent motion depend on the generated sequence.

Core Idea: learn local contact geometry and short-term motion jointly with an interaction-aware VAE, use a context Transformer to condition continuous flow matching, and exploit synthetic pretraining through object motion expressed relative to the initial frame.

Method

Overall Architecture

At inference, the inputs are a task description and an object point cloud in canonical coordinates. The outputs are hand poses, hand translations, and object motion sequences, not RGB videos. Training first learns Inter-VAE, then freezes it to train the latent generator. Ground-truth motion is encoded only for training supervision; inference generates latents from noise and reconstructs motion through the frozen decoders.

Inter-VAE combines relative motion representations, spatial interaction features, and short-term temporal compression. The generator uses text and object shape as conditions, with generated motion supplying context. The encoding branch below supplies training targets; inference enters the generation branch through text, object shape, and noise.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    Input["Training motions and object point cloud"] --> Relative["Relative motion representation"]
    Relative --> Geometry["Joint-view interaction encoding"]
    Geometry --> Temporal["Separate temporal latent branches"]
    Temporal -->|Frozen encoder supplies training targets| Flow["Context-aware autoregressive flow matching"]
    Conditions["Text, object shape, and noise"] --> Flow
    Flow --> Output["Frozen decoders reconstruct hand-object motion"]

Key Designs

1. Relative motion representation: reduce coordinate-convention effects on transfer

Datasets can differ in their origins and orientations. A model that memorizes absolute object trajectories may therefore learn acquisition conventions instead of motion structure. HO-Flow represents object motion relative to the initial frame and supervises hand translation relative to object translation. This makes how the hand follows the object less dependent on absolute scene coordinates. It is a representation choice that reduces cross-dataset coordinate differences, not a guarantee of complete physical invariance.

This choice supports the use of GraspXL, whose more than 5 million synthetic right-hand trajectories cover roughly 500,000 objects. Because these trajectories primarily concern relocation, pretraining supplies geometric and interaction priors, while target-dataset fine-tuning remains necessary for complex motions. Single-hand representation learning focuses on the right hand and mirrors available left-hand data into that form. This data-sharing strategy does not restrict the final generator to right-hand motion.

2. Joint-view interaction encoding: expose object geometry relative to individual fingers

Inter-VAE receives 1024 3D object points, axis-angle poses for 16 MANO joints per frame, and object rigid transformations. Object poses first place the canonical point cloud in world coordinates. The MANO kinematic tree then provides each joint's transformation by accumulating its own and its ancestors' transformations. Applying the inverse of each joint transformation expresses the object from that joint's local viewpoint: a surface approaching a fingertip or moving across a finger is directly reflected in these coordinates.

Concatenating the 16 joint views along the feature dimension produces hand-relative geometry of size \(1024\times(16\times3)\) per frame. A PointNet++ spatial encoder processes object points in canonical, world, and joint-local coordinates, sharing weights across frames and using 3 set-abstraction layers to produce 768-dimensional frame features. Two MLP layers fuse pose information into separate 256-dimensional hand and object spatial features. Rather than predicting an additional contact label, this representation makes contact-related geometry more accessible to the encoder.

3. Separate temporal latent branches: compress short motions instead of individual grasps

Spatial features alone cannot distinguish continuous sliding from frame-wise jitter. Separate hand and object temporal encoders aggregate neighboring frames with 1D convolutions, using two stride-2 downsampling stages for an overall temporal stride of 4. An input of \(N\) frames yields \(N/4\) latents of dimension 32 in each branch. Dedicated decoders upsample to the original sequence length and reconstruct hand and object motion. Separate branches do not discard interaction information because their input spatial features already describe relative geometry.

For bimanual generation, each temporal position concatenates the right-hand object latent, right-hand latent, mirrored-left object latent, and left-hand latent into a 128-dimensional motion token. These are continuous tokens carrying short-term interactions, not 128 discrete action classes. Original Table 1 also supports combining the representation with pretraining: object translation reconstruction error falls from 3.70 mm for the complete design without pretraining to 2.33 mm with pretraining. This is a VAE reconstruction metric, not a substitute for generation quality.

4. Context-aware autoregressive flow matching: separate sequence reasoning from continuous sampling

Frozen CLIP ViT-B/32 produces a 512-dimensional text embedding, while Basis Point Set (BPS) encoding represents object shape with 4096 distance features. Separate linear projections are summed into a 1024-dimensional condition. A masked autoregressive Transformer (MAR), with sinusoidal positional encodings and condition-modulated AdaLN, extracts context from partially visible motion tokens. The implementation uses a single layer with 16 attention heads. Autoregression here means dependence on generated motion, rather than independent prediction of every sequence position.

A lightweight SiT-style MLP flow-matching head handles continuous generation. It receives interpolated noise, flow time, and the contextual feature for a position, and predicts a velocity toward its target motion token. To distinguish the two time axes, \(t\) denotes token position and \(\tau\) denotes flow time below. Mathematical extraction is damaged in the cache; the following is an equivalent reconstruction from the prose in Section 3.3, not a verbatim equation transcription:

\[ x_\tau=(1-\tau)x_0+\tau z_t,\qquad x_0\sim\mathcal{N}(0,I),\qquad \tau\in[0,1]. \]

The target velocity is \(z_t-x_0\). Instead of outputting complete poses for every continuous sampling update, the Transformer supplies conditions describing how the current motion should continue. ODE integration generates continuous latents while the sequence is progressively completed. The main paper does not fully specify the unmasking order or name the ODE solver, so this should not be silently rewritten as a strictly frame-by-frame left-to-right algorithm.

A Worked Example

Consider handing an object from the left hand to the right. A sequence at GRAB's maximum length of 160 frames becomes 40 bimanual tokens of dimension 128 after VAE compression. The canonical object point cloud and text provide conditions; MAR uses generated handover context, the flow-matching head progressively produces the remaining latents, and the decoders reconstruct full hand-object trajectories.

This illustrates why two plausible static grasps are insufficient: release by the left hand, contact by the right hand, and object movement must agree over time. The numbers 160, 40, and 128 follow the paper's length and dimensional settings; the example does not imply that a separate success rate was reported for this prompt.

Loss & Training

The VAE applies L1 reconstruction supervision to hand 6D joint rotations, hand translation relative to the object, MANO mesh vertices, object 6D rotation, and object 3D translation. KL regularization with weight \(\beta=10^{-4}\) encourages hand and object latents toward a standard Gaussian. Mesh supervision constrains reconstructed hand shape, while pose and translation losses constrain motion parameters; these losses do not establish hard collision avoidance.

The generator samples token positions \(\mathcal{M}\) using a cosine mask-ratio schedule. Among selected positions, 80% receive a learnable mask token, 10% receive Gaussian noise, and 10% remain unchanged. Flow regression is evaluated at the selected positions. Its objective can be expressed equivalently from the textual description as:

\[ \mathcal{L}_{\mathrm{AR}}=\mathbb{E}_{\mathcal{M},\tau,x_0}\left[\frac{1}{|\mathcal{M}|}\sum_{t\in\mathcal{M}}\left\|v_\phi(x_\tau,\tau;h_t)-(z_t-x_0)\right\|_2^2\right]. \]

Training uses AdamW, linear warmup, and cosine learning-rate decay. VAE pretraining runs for 100k iterations on 4 NVIDIA H100 GPUs with batch size 32, decaying the learning rate from \(2\times10^{-4}\) to \(2\times10^{-5}\). Fine-tuning uses a base learning rate of \(10^{-4}\), with random global 3D rotations applied to the entire sequence.

The generator is also pretrained on GraspXL and then fine-tuned on the target dataset. The explicitly reported fine-tuning setup is 300k iterations with batch size 32 on a single H100. Text conditioning is dropped with 20% probability for classifier-free guidance, and EMA uses decay 0.9999. Inference uses 18 steps and guidance weight 1.5; those 18 steps should not be interpreted as only 18 network evaluations for an entire motion sequence.

Key Experimental Results

Main Results

GRAB uses an object-based split with 47 training objects, 4 unseen test objects, and 17 text-object test pairs. Training sequences start at first contact and have a maximum length of 160 frames. OakInk supplies 212 evaluation pairs from 100 novel objects across 20 categories, using the GRAB-trained model without training on motion data from this split. DexYCB reserves 4 of 20 objects for testing, starts sequences at the first frame, and uses a maximum length of 96 frames.

IV is mean hand-object intersection volume over frames with nonzero interpenetration, in cm³. ID is the frame-averaged maximum penetration depth under the same nonzero-interpenetration condition, in cm. Phy is the percentage of frames in which the object moves while maintaining hand-object contact, not a robotic task success rate. SD is mean pairwise L2 distance between multiple generations for the same input; the main paper does not specify its numerical unit, so none is added here.

Dataset / Original Table	Method	IV (cm³, lower better)	ID (cm, lower better)	Phy (%, higher better)	SD (higher better)
GRAB / Table 3	LatentHOI	6.38	0.77	96.16	0.13
GRAB / Table 3	HO-Flow, no pretraining	5.48	0.63	97.94	0.30
GRAB / Table 3	HO-Flow, pretrained	5.31	0.61	98.25	0.31
OakInk / Table 4	LatentHOI	7.22	1.10	71.24	0.22
OakInk / Table 4	HO-Flow, no pretraining	5.82	0.66	83.62	0.31
OakInk / Table 4	HO-Flow, pretrained	4.10	0.45	89.76	0.33
DexYCB / Table 5	LatentHOI	7.70	2.01	88.52	0.13
DexYCB / Table 5	HO-Flow, no pretraining	6.84	1.82	90.77	0.20
DexYCB / Table 5	HO-Flow, pretrained	6.37	1.20	95.41	0.20

For GRAB and OakInk, only right-hand IV and ID are shown here; DexYCB uses single-hand metrics. Pretrained means pretrained on GraspXL, so these rows do not isolate architecture under an identical training-data budget.

Ablation Study

The following rows come from original Table 2 and all use GRAB. R1-R4 have no pretraining; R5 adds GraspXL.

Config	Representation and generation	Right-hand IV (cm³)	Right-hand ID (cm)	Phy (%)	SD
R1	Explicit pose sequence, non-autoregressive	10.35	1.14	84.89	0.24
R2	Frame-wise latents, non-autoregressive	8.13	0.98	88.76	0.12
R3	Inter-VAE temporal latents, non-autoregressive	6.99	0.82	92.69	0.26
R4	Inter-VAE + autoregressive generation	5.48	0.63	97.94	0.30
R5	R4 + synthetic pretraining	5.31	0.61	98.25	0.31

Key Findings

• R3 to R4 increases Phy by 5.25 percentage points and reduces right-hand IV from 6.99 to 5.48 cm³ without pretraining, supporting an independent benefit from temporal context.
• Pretraining increases Phy from 97.94% to 98.25% on GRAB, but from 83.62% to 89.76% on OakInk. Greater unseen-object variation makes the benefit of synthetic priors more apparent.
• Not every contact metric improves: in Table 3, LatentHOI has right-hand CR of 11.94%, above pretrained HO-Flow's 11.38%. A larger proportion of contacting vertices does not automatically imply a more plausible overall motion.

Highlights & Insights

• Joint-local coordinates provide a direct interaction representation: viewing the same object from different fingers exposes contact-related geometry. Transferring this idea calls for coordinate frames attached to relevant contact parts, rather than merely adding world-coordinate features.
• Continuous latents are compatible with autoregressive context. MAR determines how motion should continue, while flow matching retains fine-grained continuous variation instead of assigning all control precision to a discrete codebook.
• Table 2 separates representation, temporal generation, and pretraining contributions. It provides local comparisons for evaluating whether improvements arise from additional data or a more suitable model.

Limitations & Future Work

• The authors explicitly note that GraspXL mainly contains relocation tasks and lacks motion diversity. Large-scale synthetic data therefore does not resolve complex task-semantic coverage, and target-data fine-tuning remains necessary.
• Phy uses contact preservation during object motion as a physical-plausibility proxy. Nonzero IV and ID also show that penetration remains; the paper does not establish force closure, frictional stability, or real-robot execution success through these metrics.
• Evaluation sequences are capped at 160 / 96 frames, and GRAB sequences start at first contact. These results should not be extrapolated to indefinitely long motions or the complete approach-and-initial-grasp process.
• The user study has only 10 evaluators and 30 test samples per benchmark. The authors caution that repeated judgments from the same evaluator are dependent, so preference ratios should be interpreted as evaluator-level evidence.
• Mathematical extraction in the cached main paper is damaged, and some metric and sampling details are deferred to unavailable supplementary material. The equivalent equations above explain the mechanism, but exact reproduction requires checking the complete paper and implementation. Explicit collision constraints, mechanical evaluation, and longer interaction tests are useful next steps.

Related Work & Insights

• vs LatentHOI: LatentHOI primarily encodes local hand poses frame by frame. HO-Flow jointly encodes spatial hand-object relationships and short-term global motion, then adds autoregressive context. Ablation R2 versus R3 is more informative about this representation difference than the final leaderboard alone.
• vs DiffH2O: Explicit pose generation avoids a latent-learning stage but asks the generator to handle complex continuous pose variables directly. HO-Flow first learns reconstruction with a VAE, then generates in a compact space.
• vs HOI-GPT: Both exploit autoregressive dependencies, but HO-Flow uses conditional flow matching for continuous tokens instead of quantizing motion into a finite discrete codebook.
• Relationship to GraspXL: GraspXL supplies interaction priors generated through physics simulation; it is not a new dataset introduced by HO-Flow. Results depend on both representation design and external pretraining resources.

Rating

• Novelty: 4/5. Combines kinematic relative geometry, temporal latents, and continuous autoregressive generation for fine-grained hand-object interaction.
• Experimental Thoroughness: 4/5. Covers three main benchmarks and component ablations, while real physical execution and very long sequences remain unverified.
• Writing Quality: 4/5. Designs map clearly to ablations, but sampling-schedule details in the main paper are insufficient for independent reproduction.
• Value: 4/5. Useful for animation and data generation requiring diverse, coherent interactions, but not directly established as an executable controller.