EgoMAN: Interaction-Structured Reasoning for Egocentric 3D Hand Trajectory Prediction

Conference: ECCV 2026
Paper: ECCV Official
Code: https://egoman-project.github.io/
Area: Multimodal VLM / LLM Reasoning
Keywords: 3D Hand Trajectory Prediction, Interaction Reasoning, Trajectory-Token Interface, Flow Matching, 6DoF Motion Generation

TL;DR

EgoMAN introduces an interaction-structured reasoning framework that decomposes egocentric 3D hand motion into stage-aware phases, bridging high-level multimodal reasoning and low-level Flow Matching motion experts via a compact 4-token Trajectory-Token Interface.

Background & Motivation

Predicting future 3D hand motion from an egocentric perspective is a foundational capability for proactive assistive systems, augmented reality, and robotic imitation learning. Systems must infer future continuous spatial trajectories across extended horizons from a single RGB frame, past motion history, and human intent under significant camera motion, depth ambiguity, and severe hand-object occlusions. However, real-world daily egocentric interactions naturally contain substantial transitional, idle, and exploratory hand movements that bear little semantic correlation to the actual task goal.

Existing methodologies struggle to address this challenge. On one hand, continuous end-to-end regression frameworks (such as USST, MMTwin, and CVAE-based HandsOnVLM) typically treat motion as homogeneous continuous time series. Lacking explicit interaction stage grounding, these models struggle to distinguish purposeful interaction from incidental egocentric motion noise, accumulating severe drift and failing to generalize across novel scenes. On the other hand, affordance-based methods heavily rely on external object bounding-box detectors and predefined heuristic contact estimators. This cascading pipeline is computationally prohibitive, operates at very low frame rates, and remains brittle to cumulative perception errors and open-vocabulary intent.

In contrast, human manual actions naturally unfold through structured, stage-aware phases—such as reaching toward an object, engaging in contact manipulation, and completing the interaction. The key insight of this paper is to decouple high-level spatiotemporal intent reasoning from high-frequency continuous physical dynamics by establishing sparse, physically interpretable interaction stage anchors. The core idea is to develop EgoMAN, an interaction-structured reasoning framework that connects a vision-language reasoning module to a continuous Flow Matching motion expert through a compact 4-token Trajectory-Token Interface, harmonized via a progressive three-stage training strategy for intent-consistent 6DoF trajectory generation.

Method

Overall Architecture

The input to EgoMAN comprises a single first-person RGB observation \(V_t\), past bi-hand wrist trajectories \(\{L_\tau, R_\tau\}_{\tau=t-H}^t\) (where \(L_\tau \in \mathbb{R}^6\) represents 3D positions and \(R_\tau \in \mathbb{R}^{12}\) denotes 6D continuous rotations), and an intent description \(I\) in natural language. The system aims to predict the future 6DoF trajectories of both wrists \(\{\tilde{L}_\tau, \tilde{R}_\tau\}_{\tau=t+1}^{t+T}\) across the approaching and manipulation stages.

The overall pipeline consists of three core components: (a) an Interaction-Structured Reasoning Module built on Qwen2.5-VL that processes visual features, past motion embeddings, and intent queries to generate action semantics and stage-aware waypoints; (b) the Trajectory-Token Interface that bridges reasoning and physical dynamics by decoding four discrete structured tokens; and (c) a Motion Expert based on Flow Matching that conditions on past motion, DINOv3 visual tokens, semantic embeddings, and decoded 6DoF stage waypoints to generate smooth, physically plausible continuous 6DoF trajectories.

%%{init: {'flowchart': {'rankSpacing': 24, 'nodeSpacing': 28, 'padding': 6, 'wrappingWidth': 400}}}%%
flowchart TD
    A["Egocentric Input<br/>RGB frame + past wrist motion + intent query"] --> B["Interaction-Structured Reasoning<br/>Qwen2.5-VL spatiotemporal stage reasoning"]
    B --> C["Trajectory-Token Interface<br/>ACT semantic token + START/CONTACT/END waypoints"]
    C --> D["Flow Matching Motion Expert<br/>Conditional velocity field generation guided by waypoints"]
    D --> E["Output 6DoF Bi-Hand Trajectories<br/>Future continuous 3D position + 6D rotation"]

Key Designs

1. Interaction-Structured Reasoning and 4-Token Parameterization: Anchoring continuous motion to interaction stages

To overcome the ambiguity of treating hand trajectories as uniform continuous signals, this design decomposes hand-object interactions into distinct, causally ordered phases. The reasoning module extends a vision-language backbone to output either free-form text or four dedicated tokens upon receiving a trigger instruction: one action semantic token <ACT>, and three stage waypoint tokens <START> (approach onset), <CONTACT> (manipulation onset / approach completion), and <END> (manipulation completion). Each waypoint token is equipped with a lightweight prediction head that regresses its normalized timestamp, 3D metric coordinates, and continuous 6D rotation representation. This anchors high-dimensional motion generation onto a minimal set of interpretable topological events.

2. Adaptive Contrastive Semantics and Weighted Waypoint Supervision: Stabilizing multi-task pretraining

Pretraining the reasoning module across 1M question-answer pairs involves a dynamic mix of text QA and numerical trajectory regression queries within each batch. Standard contrastive learning collapses when the number of valid samples \(K\) fluctuates drastically. To ensure numerical stability, an adaptive action-semantic loss switches between cosine similarity and InfoNCE based on a sample threshold \(\kappa\):

\[ \mathcal{L}_{\text{act}} = \begin{cases} 1 - \frac{1}{K}\sum_{i=1}^K \text{sim}(z_i, z_i^+), & K < \kappa \\ -\frac{1}{K}\sum_{i=1}^K \log \frac{\exp(\text{sim}(z_i, z_i^+)/\tau)}{\sum_{j=1}^K \exp(\text{sim}(z_i, z_j^+)/\tau)}, & K \ge \kappa \end{cases} \]

Concurrently, the predicted waypoints are supervised with Huber losses weighted by Gaussian temporal windows, alongside a 2D reprojection loss and a continuous geodesic rotation loss, restricted strictly to visible waypoints to prevent corrupting gradients from occluded keypoints.

3. Flow Matching Motion Expert and Progressive Alignment: Bridging the reasoning-to-dynamics gap

End-to-end joint training of multimodal reasoning backbones and continuous generative motion models typically destabilizes optimization due to gradient scale mismatches. EgoMAN addresses this through a progressive three-stage pipeline: First, the reasoning module is pretrained on QA data. Second, the Flow Matching motion expert is independently pretrained on ground-truth waypoints and phrase semantics, mastering the continuous velocity field \(\hat{v}(x_t, t)\) over DINOv3 visual context. Third, joint fine-tuning aligns the two modules across the Trajectory-Token Interface. The motion expert learns to accommodate imperfect, model-predicted waypoints, using them as soft structural priors rather than brittle hard constraints, enabling fast and stable ODE sampling.

Loss & Training

The reasoning pretraining optimizes \(\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{text}} + \lambda_{\text{act}} \mathcal{L}_{\text{act}} + \lambda_{\text{wp}} \mathcal{L}_{\text{wp}}\). The Motion Expert is optimized via the conditional Flow Matching objective:

\[ \mathcal{L}_{\text{FM}} = \mathbb{E}_{t, x_0, x_1} \left\| \hat{v}(x_t, t) - (x_1 - x_0) \right\|_2^2 \]

During inference, trajectories are sampled from a standard Gaussian prior \(x_0\) and integrated over \(N\) Euler steps: \(x_{k+1} = x_k + \Delta t \cdot \hat{v}(x_k, t_k)\) with step size \(\Delta t = 1/N\), generating full 5-second 6DoF hand trajectories.

Key Experimental Results

Main Results

Evaluation is conducted on EgoMAN-Bench under two setups: in-domain held-out scenes (EgoMAN-Unseen) and out-of-distribution environments (HOT3D-OOD). Metrics are reported as Best-of-10, encompassing Average Displacement Error (ADE), Final Displacement Error (FDE), Dynamic Time Warping (DTW) in meters, and Angular Rotation Error (Rot) in degrees.

Table 1: 3D Hand Trajectory Forecasting Performance on EgoMAN-Bench (Original Paper Table 1)

Dataset	Method	ADE (m) ↓	FDE (m) ↓	DTW (m) ↓	Rot (°) ↓
EgoMAN-Unseen	USST* (ICCV 2023)	0.233	0.394	0.220	46.98
	MMTwin*	0.206	0.256	0.204	48.98
	HandsOnVLM* (TMLR 2025)	0.171	0.228	0.161	35.22
	FM-base (Motion Expert only)	0.160	0.229	0.144	37.00
	EgoMAN-ACT (w/o stage reasoning)	0.141	0.204	0.127	35.03
	EgoMAN (Ours)	0.124	0.179	0.111	32.75
HOT3D-OOD	USST* (ICCV 2023)	0.245	0.409	0.226	55.80
	MMTwin*	0.209	0.259	0.207	44.37
	HandsOnVLM* (TMLR 2025)	0.194	0.262	0.186	38.13
	FM-base (Motion Expert only)	0.161	0.237	0.147	39.47
	EgoMAN-ACT (w/o stage reasoning)	0.153	0.228	0.141	38.42
	EgoMAN (Ours)	0.141	0.217	0.130	35.09

Ablation Study

The ablation investigates the impact of Interaction-Structured Reasoning (R), Motion Expert pretraining (M), and the Trajectory-Token Interface (WP) on EgoMAN-Unseen under single-sample inference (\(K=1\)).

Table 2: Ablation Study on Core Components (Original Paper Table 2)

Config #	Reasoning (R)	Motion Pretrain (M)	Interface (WP)	ADE (m) ↓	FDE (m) ↓	DTW (m) ↓	Rot (°) ↓
1	✗	✗	✗	0.273	0.308	0.260	51.79
2	✓	✗	6DoF	0.215	0.255	0.198	43.03
3	✗	✓	✗	0.162	0.225	0.148	36.24
4	✓	✓	✗	0.161	0.224	0.147	35.90
5	✓	✓	Emb	0.150	0.210	0.138	34.02
6 (Full model)	✓	✓	6DoF WP	0.151	0.206	0.137	33.88

Key Findings

• Substantial Gains Over Baselines: On EgoMAN-Unseen, EgoMAN reduces ADE by 27.5% compared to the strongest baseline HandsOnVLM* (0.124m vs. 0.171m). On the challenging out-of-distribution HOT3D-OOD split, it achieves a 27.3% reduction in ADE (0.141m vs. 0.194m) and lowers rotation error to 35.09°, demonstrating robust domain transfer.
• Critical Necessity of the Trajectory-Token Interface: In Table 2, comparing Config 4 (no WP interface) with Config 6 reveals that without the structured interface, reasoning alone yields almost no improvement over the base motion expert (ADE 0.161m vs. 0.162m). Explicit 6DoF waypoint conditioning is mandatory to channel high-level spatiotemporal reasoning into low-level trajectory execution.
• Superior Waypoint Precision and Inference Speed: In waypoint evaluation (Original Paper Table 3), EgoMAN achieves a contact error of 0.192m (a 33.8% improvement over VidBot's 0.290m) and a trajectory error of 0.127m (a 52.8% improvement). Furthermore, EgoMAN operates at 3.45 FPS, over \(70\times\) faster than affordance baselines (VRB* at 0.03 FPS, VidBot at 0.04 FPS).
• Strong Semantic Grounding: In the motion-to-verb retrieval task (Original Paper Table 4), EgoMAN attains a Top-3 recall of 43.9% and an FID of 0.04, significantly outperforming HandsOnVLM (27.9%, 0.10) and MMTwin (22.9%, 0.86), confirming that predicted motions are semantically distinct rather than collapsed into mean trajectory modes.

Highlights & Insights

• Bridging Semantics and Continuous Dynamics via Discrete Stage Anchors: Condensing continuous interaction into four structured tokens bypasses both the accumulation error of autoregressive float generation and the brittleness of heuristic detection pipelines.
• Soft Prior Guidance Over Hard Segmentation: Injecting waypoints as conditioning tokens rather than hard boundary cuts gives the Flow Matching expert continuous interpolation freedom, ensuring high tolerance to slight spatial or temporal prediction errors from the VLM.
• Decoupled-then-Aligned Training Paradigm: Demonstrating that high-level reasoning and physical trajectory generation suffer from gradient competition, the three-stage progressive training framework offers a practical blueprint for complex vision-language-action architectures.

Limitations & Future Work

• Author-Acknowledged Limitations: EgoMAN currently models wrist-level 6DoF motion and does not capture articulated finger postures, multi-finger dexterous manipulation, or fine contact forces. Furthermore, interaction stages are limited to approach and manipulation.
• Critical Observations: Waypoint annotation accuracy relies on monocular 3D depth and pose estimation, which exhibits residual noise in cases of severe self-occlusion or dynamic motion blur. Additionally, generated trajectories do not enforce explicit collision avoidance against dense 3D scene meshes.
• Future Directions: Integrating full-hand articulation models (e.g., MANO/SMPL-X), incorporating fine-grained contact mechanics, and directly deploying predicted trajectories to closed-loop impedance controllers on real robotic arms or AR devices.

Related Work & Insights

• vs. HandsOnVLM: HandsOnVLM utilizes a CVAE to decode 50 implicit hand tokens from vision-language representations, lacking explicit physical or temporal stage grounding. EgoMAN employs only 4 stage-aware tokens, achieving superior spatial accuracy and smoother trajectory execution.
• vs. Affordance Baselines (VRB, VidBot): Conventional affordance approaches rely on external object detectors and heavy post-processing pipelines, leading to slow inference (<0.05 FPS) and failure under occlusion. EgoMAN directly decodes metric waypoints within the VLM backbone at 3.45 FPS.
• Takeaway: For embodied VLA architectures, models should avoid directly regressing high-frequency continuous control signals from language backbones. Providing discrete semantic waypoints as structural guidance for continuous diffusion/flow modules offers the most balanced paradigm.

Rating

• Novelty: ⭐⭐⭐⭐⭐ Elegant Trajectory-Token Interface grounding continuous hand motion into discrete topological stages.
• Experimental Thoroughness: ⭐⭐⭐⭐⭐ Comprehensive 219K-trajectory dataset, rigorous OOD zero-shot evaluations, complete ablations, and semantic retrieval tests.
• Writing Quality: ⭐⭐⭐⭐⭐ Clear logical progression, well-structured methodology, and insightful empirical analysis.
• Value: ⭐⭐⭐⭐⭐ Provides a foundational methodology and high-value benchmark for egocentric motion forecasting and robotic imitation learning.

Related Papers

• [ECCV 2026] EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoning
• [ECCV 2026] 3D-Layout-R1: Structured Reasoning for Language-Instructed Spatial Editing
• [ECCV 2026] CORE-V: Chain-Of-thought REasoning for Image Editing with Visual Interaction
• [CVPR 2026] EgoProx: Evaluating MLLMs on Egocentric 3D Proximity Reasoning Across a Cognitive Hierarchy
• [ECCV 2026] Spatial Amsan: A Benchmark for Perception-Grounded Spatial Reasoning and Action Evaluation in Egocentric Manipulation